19,106 Hits in 3.5 sec

Survey of web-based crowdsourcing frameworks for subjective quality assessment

Tobias Hobfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, Christian Timmerer
2014 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP)  
Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR  ...  The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE).  ...  Subjective quality assessment or QoE assessment of multimedia is another task suitable for crowdsourcing.  ... 
doi:10.1109/mmsp.2014.6958831 dblp:conf/mmsp/HossfeldHKHGKT14 fatcat:mwa6uh5k3baafnh3qnzkygq65m

A web based subjective evaluation platform

Benjamin Rainer, Markus Waltl, Christian Timmerer
2013 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX)  
The presented framework can be used for both crowdsourced and laboratory experiments. It should ease the task of designing a subjective quality assessment by providing a flexible framework.  ...  Preparing and conducting subjective quality assessments is a time consuming and expensive task.  ...  INTRODUCTION Planning, preparation, and conducting subjective quality assessments in the area of Quality of Experience (QoE) is time consuming and in many cases an expensive task.  ... 
doi:10.1109/qomex.2013.6603196 dblp:conf/qomex/RainerWT13 fatcat:ppjd5tdhjvbwxfjyzph7wc5dzi

Crowdsourcing versus the laboratory: Towards crowd-based linguistic text quality assessment of query-based extractive summarization

Neslihan Iskender, Tim Polzehl, Sebastian Möller
2020 Conference on Digital Curation Technologies  
By conducting experiments both in crowdsourcing and laboratory environments, and comparing the results of linguistic quality judgments, we found out that microtask crowdsourcing shows high applicability  ...  We measure the linguistic quality of crowd-based forum summarizations, which is usually conducted in a traditional laboratory environment with the help of experts, via comparative crowdsourcing and laboratory  ...  with respect to OQ and the five linguistic quality scores applying one-way ANOVA or in case of non-normal distribution Kruskal-Wallis tests.  ... 
dblp:conf/qurator/IskenderP020 fatcat:f34j62wayngbbdxwxu6k2egj5a

Crowdsourcing Quality of Experience Experiments [chapter]

Sebastian Egger-Lampl, Judith Redi, Tobias Hoßfeld, Matthias Hirth, Sebastian Möller, Babak Naderi, Christian Keimel, Dietmar Saupe
2017 Lecture Notes in Computer Science  
Qualinet members that participated in the creation of Best Practices and Recommendations for Crowdsourced QoE -Lessons learned from the Qualinet Task Force [38] .  ...  The authors want to thank Schloss Dagstuhl Leibniz-Zentrum für Informatik, the participants of Dagstuhl Seminar 15481 Evaluation in the Crowd: Crowdsourcing and Human-Centred Experiments as well as the  ...  Buffering the video can help in lifting this limitation, but buffering will extend the time needed per test case, limiting in turn the number of test cases that can be assessed per crowdsourcing task and  ... 
doi:10.1007/978-3-319-66435-4_7 fatcat:mkhhxbmtpzhqzcawfep7m7v5oi

Application of Just-Noticeable Difference in Quality as Environment Suitability Test for Crowdsourcing Speech Quality Assessment Task [article]

Babak Naderi, Sebastian Möller
2020 arXiv   pre-print
Thus, the combination of listening device and background noise needs to be screened in a crowdsourcing speech quality test.  ...  In the case of speech quality assessment, preliminary work showed that environmental noise at the listener's side and the listening device (loudspeaker or headphone) significantly affect perceived quality  ...  The results of these tests are analyzed in Section III, by checking the impact of the JND test in relation to the crowdsourcing speech quality assessment outcome.  ... 
arXiv:2004.05502v1 fatcat:csnkddfgebdh7biawvym6fybry

Detecting Linked Data quality issues via crowdsourcing: A DBpedia study

Maribel Acosta, Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Fabian Flöck, Jens Lehmann, Marta Sabou, Lora Aroyo, Kalina Bontcheva, Alessandro Bozzon
2018 Semantic Web Journal  
We then propose and study different crowdsourcing approaches to identify these Linked Data quality issues, employing the DBpedia dataset as our use case: (i) a contest targeting the Linked Data expert  ...  In this paper we examine the use of crowdsourcing as a means to master Linked Data quality problems that are difficult to solve automatically.  ...  ., for specifying rules [18] or test cases [26] , or fail due to the context-specific nature of quality assessment, which does not lend itself well to general workflows and rules that could be executed  ... 
doi:10.3233/sw-160239 fatcat:dus7b5hplzbexg47xbcw57akhq

Do Scale-Design and Training Matter for Video QoE Assessments through Crowdsourcing?

Bruno Gardlo, Sebastian Egger, Tobias Hossfeld
2015 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia - CrowdMM '15  
Crowdsourcing (CS) has evolved into a mature assessment methodology for subjective experiments in diverse scientific fields and in particular for QoE assessment.  ...  Additionally, training of subjects is well known to enhance result quality for laboratory ACR evaluations.  ...  BACKGROUND AND MOTIVATION For assessing QoE of a large variety of different test cases, crowdsourcing has recently gained considerable attention [1, 3, 9, 18, 26] as a fast and economical alternative  ... 
doi:10.1145/2810188.2810193 dblp:conf/mm/GardloEH15 fatcat:gw2wvknfzzd5ffrtamgnruu6wq

On subjective quality assessment of adaptive video streaming via crowdsourcing and laboratory based experiments

Jacob Søgaard, Muhammad Shahid, Jeevan Pokhrel, Kjell Brunnström
2016 Multimedia tools and applications  
On subjective quality assessment of adaptive video streaming via crowdsourcing and laboratory based experiments.  ...  Besides laboratory based subjective experiments, crowdsourcing based subjective assessment of video quality is gaining popularity as an alternative  ...  and the perceptual preference of video quality.  ... 
doi:10.1007/s11042-016-3948-3 fatcat:nyw5fwzx6zdwzo64ure46vdrky

The TUM high definition video datasets

Christian Keimel, Arne Redl, Klaus Diepold
2012 2012 Fourth International Workshop on Quality of Multimedia Experience  
Recent contributions have also shown that crowd-based video quality assessment can deliver results comparable to traditional testing in some cases.  ...  A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet.  ...  CROWD-BASED VIDEO QUALITY ASSESSMENT In crowd-based video quality assessment we utilize these crowdsourcing platforms to perform subjective testing with a global worker pool, usually with a web-based application  ... 
doi:10.1109/qomex.2012.6263865 dblp:conf/qomex/KeimelRD12 fatcat:zfgszmggqbd43pji4oc3rrj3py

Crowdsourced Testing for Enterprises: Experiences

Prahlad Rao, Alpana Dubey, Gurdeep Virdi
2015 Asia-Pacific Software Engineering Conference  
It brings cost and quality advantage with faster delivery of software products. Among all the software development activities, testing is one of the primary activities considered for crowdsourcing.  ...  Crowdsourced testing is an emerging phenomenon in software testing which utilizes benefits of crowdsourcing for software testing.  ...  Also crowdsourced testing need to be assessed from the perspective of confidentiality and security. TABLE I .  ... 
dblp:conf/apsec/RaoDV15 fatcat:ps42d2v34jcpbbmnoohkc27m4q

Towards a Reliable and Robust Methodology for Crowd-Based Subjective Quality Assessment of Query-Based Extractive Text Summarization

Neslihan Iskender, Tim Polzehl, Sebastian Möller
2020 International Conference on Language Resources and Evaluation  
Further, we investigate the effect of the number of repetitions of assessments on the robustness of mean opinion score of crowd ratings, measured against the increase of correlation coefficients between  ...  To investigate the appropriateness of crowdsourcing for this task, we conduct intensive comparative crowdsourcing and laboratory experiments, evaluating nine extrinsic and intrinsic quality measures on  ...  To test the normality of these crowdsourcing and laboratory ratings, we carried out Anderson Darling tests.  ... 
dblp:conf/lrec/IskenderPM20 fatcat:2ih34bsdpre35gfbjom6p3ymxy

Speech Quality Assessment in Crowdsourcing: Comparison Category Rating Method [article]

Babak Naderi, Sebastian Möller, Ross Cutler
2021 arXiv   pre-print
This paper evaluates an adaptation of the CCR test procedure for assessing speech quality in the crowdsourcing set-up. The CCR method was introduced in the ITU-T Rec.  ...  Traditionally, Quality of Experience (QoE) for a communication system is evaluated through a subjective test.  ...  In this paper we compare the results of multiple CCR crowdsourcing test to assess its reproducibility.  ... 
arXiv:2104.04371v1 fatcat:dndhrwgo6ngfzm5wmmhlimnope

A crowdsourced implementation of ITU-T P.910 [article]

Babak Naderi, Ross Cutler
2022 arXiv   pre-print
We provide an open-source extension of P.910 based on crowdsourcing principles which address the speed, usage cost, and barrier to usage issues.  ...  The gold standard for measuring video quality is the subjective test, and the most prevalent is the ITU-T P.910, a lab-based subjective standard in use for the past two decades.  ...  RELATED WORK A recent review of subjective quality assessment tools is given in [12] . P.910 [1] provides a general subjective video quality assessment standard for multimedia applications.  ... 
arXiv:2204.06784v1 fatcat:22vrmsgos5hbhfodaxadlzieyq

Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

Chen-Chi Wu, Kuan-Ta Chen, Yu-Chun Chang, Chin-Laung Lei
2013 IEEE transactions on multimedia  
In this paper, we propose a crowdsourceable framework to quantify the QoE of multimedia content. To overcome the aforementioned quality problem, we employ a paired comparison method in our framework.  ...  Moreover, the results of four case studies support our assertion that the framework can provide reliable QoE evaluation at a lower cost.  ...  In other words, our framework makes crowdsourced QoE assessment studies immune to dishonest participants because the latter will neither affect the quality of QoE assessment nor the economic cost.  ... 
doi:10.1109/tmm.2013.2241043 fatcat:o6a5kekhnjarvi4b2gjn6zei3q

Crowdsourcing in Computer Science Education

Nea Pirttinen
2021 Proceedings of the 17th ACM Conference on International Computing Education Research  
Crowdsourcing is a method of collecting services, ideas, materials or other artefacts from a relatively large and open group of people.  ...  In my research, I focus on investigating the effects of computing education systems that use crowdsoucing on students' learning, and the types of quality assurance methods required to use the artefacts  ...  RQ2 can be investigated by looking at the quality assurance methods in use at the moment, such as peer reviews and automated assessment of the crowdsourced content, and evaluating what methods are the  ... 
doi:10.1145/3446871.3469781 fatcat:jqzuqedlwjf2ffpv6fzecmxv64
« Previous Showing results 1 — 15 out of 19,106 results