5,612 Hits in 5.5 sec

Survey of web-based crowdsourcing frameworks for subjective quality assessment

Tobias Hobfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, Christian Timmerer
2014 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP)  
In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers.  ...  The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE).  ...  The authors alone are responsible for the content.  ... 
doi:10.1109/mmsp.2014.6958831 dblp:conf/mmsp/HossfeldHKHGKT14 fatcat:mwa6uh5k3baafnh3qnzkygq65m


Malek Hammou, Cise Midoglu, Steven A. Hicks, Andrea Storås, Saeed Shafiee Sabet, Inga Strümke, Michael A. Riegler, Pål Halvorsen
2022 Proceedings of the 13th ACM Multimedia Systems Conference  
Our proposed framework can be used as an online survey tool by researchers working on different topics such as Machine Learning (ML), audio, image, and video quality assessment, Quality of Experience (  ...  The web-based framework is built in a modular and configurable fashion to allow for the easy adjustment of the user interface (UI) and the multimedia content, while providing integrations with reliable  ...  crowdMOS [26] , an open source project designed for video and image quality assessment, QualityCrowd [21] , a web-based framework designed for video quality assessment, and Crowdgaming [29] , a JavaScript-based  ... 
doi:10.1145/3524273.3532887 fatcat:5m4u4a2hjfbmxcantnkoodlv7q

Perspectives on crowdsourcing annotations for natural language processing

Aobo Wang, Cong Duy Vu Hoang, Min-Yen Kan
2012 Language Resources and Evaluation  
Crowdsourcing has emerged as a new method for obtaining annotations for training models for machine learning.  ...  While many variants of this process exist, they largely differ in their method of motivating subjects to contribute and the scale of their applications.  ...  Acknowledgements We would like to thank many of our colleagues who have taken time off their tight schedules to help review and improve to this paper, including Yee Fan Tan, Jesse Prabawa Gozali, Jun-Ping  ... 
doi:10.1007/s10579-012-9176-1 fatcat:dl6tvexzn5edlpdjhsn3uczfv4

A framework for syntactic and semantic quality evaluation of ontologies [article]

Vivek Iyer, Lalit Mohan Sanagavarapu, Raghu Reddy
2021 arXiv   pre-print
SemValidator uses Twitter-based expertise of validators for semantic evaluation. The efficacy and validity of the framework is shown empirically on multiple ontologies.  ...  This paper proposes an ontology quality evaluation framework consisting of: (a) SynEvaluator and (b) SemValidator for evaluating syntactic and semantic aspects of ontologies respectively.  ...  .: Triplecheckmate: A tool for crowdsourcing the quality assessment of linked data. In: International Conference on Knowledge Engineering and the Semantic Web. pp. 265–272.  ... 
arXiv:2112.08543v1 fatcat:5l4h42ymura57ittm5me74ho44

Crowdsourcing Quality of Experience Experiments [chapter]

Sebastian Egger-Lampl, Judith Redi, Tobias Hoßfeld, Matthias Hirth, Sebastian Möller, Babak Naderi, Christian Keimel, Dietmar Saupe
2017 Lecture Notes in Computer Science  
Qualinet members that participated in the creation of Best Practices and Recommendations for Crowdsourced QoE -Lessons learned from the Qualinet Task Force [38] .  ...  The authors want to thank Schloss Dagstuhl Leibniz-Zentrum für Informatik, the participants of Dagstuhl Seminar 15481 Evaluation in the Crowd: Crowdsourcing and Human-Centred Experiments as well as the  ...  Web-based crowdsourcing frameworks for multimedia quality assessment represent a conceptual approach with programming tools to develop subjective tests that can be executed in a web browser.  ... 
doi:10.1007/978-3-319-66435-4_7 fatcat:mkhhxbmtpzhqzcawfep7m7v5oi

CMS Model Subjective Quality Evaluation VIA Crowdsourcing

Sikander Rahu
2018 International Journal for Research in Applied Science and Engineering Technology  
Crowdsourcing for subjective tests have been done by hiring faculty members of SAU who shall be the potential users of this CMS.  ...  and quality score through crowdsourcing.  ...  The authors made a CMS web based system for conducting subjective quality evaluations using crowdsourcing platform.  ... 
doi:10.22214/ijraset.2018.1045 fatcat:abrrscqprvhujhikb6uxoin55u

Crowdsourcing without Data Bias: Building a Quality Assurance System for Air Pollution Symptom Mapping

Marta Samulowska, Szymon Chmielewski, Edwin Raczko, Michał Lupa, Dorota Myszkowska, Bogdan Zagajewski
2021 ISPRS International Journal of Geo-Information  
In this study, we address the problem of detecting and removing crowdsourced data bias as a prerequisite for better-quality open-data output.  ...  The method includes the geospatial web (GeoWeb) platform as well as a QAs based on conditional statements.  ...  As crowdsensing (or, more generally, crowdsourcing) methods for health symptom mapping are subject to data bias [11] , we developed and tested a quality assurance mechanism (QAm) framework (Section 2.2  ... 
doi:10.3390/ijgi10020046 fatcat:yia6w3oy4fdc3f7kuktum3sqdy

Teaching Sensor Technology and Crowdsourcing with Reusable Learning Objects

E. A. Mikhailova, R. Y. Stiglitz, C. J. Post, R. P. Pargas, T. M. Campbell, K. S. Payne, J. A. Cooper
2018 Natural Sciences Education  
This study also used the web-based survey tool Qualtrics before and after the laboratory-based activity to systematically measure various constructs including familiarity with sensors, crowdsourcing, and  ...  The quality of each RLO is assessed by a learning object review instrument (LORI) framework rubric. Online quizzes at the end of each RLO are used to examine the learning outcomes.  ...  sensor application; (2) assess the quality of each RLO using the LORI framework; (3) use the web-based survey tool Qualtrics to systematically measure various constructs (e.g., familiarity with sensors  ... 
doi:10.4195/nse2018.08.0015 fatcat:7ybnxva6zbcj5kkxmp73tpqdra

A Survey of Volunteered Open Geo-Knowledge Bases in the Semantic Web [chapter]

Andrea Ballatore, David C. Wilson, Michela Bertolotto
2013 Intelligent Systems Reference Library  
Particular attention is devoted to the crucial issue of the quality of geo-knowledge bases, as well as of crowdsourced data.  ...  In this article, we survey these open knowledge bases, focusing on their geospatial dimension.  ...  Section 5 surveyed the existing strategies to assess the quality of geo-knowledge bases, with particular emphasis on the quality of crowdsourced data sources.  ... 
doi:10.1007/978-3-642-37688-7_5 fatcat:i72rmxk36fbnrmnrdpkmccrbfu

User-driven quality evaluation of DBpedia

Amrapali Zaveri, Dimitris Kontokostas, Mohamed A. Sherif, Lorenz Bühmann, Mohamed Morsey, Sören Auer, Jens Lehmann
2013 Proceedings of the 9th International Conference on Semantic Systems - I-SEMANTICS '13  
We present a methodology for assessing the quality of linked data resources, which comprises of a manual and a semi-automatic process.  ...  However, these datasets are of varying quality ranging from extensively curated datasets to crowdsourced and even extracted data of relatively low quality.  ...  RELATED WORK Web data quality assessment frameworks.  ... 
doi:10.1145/2506182.2506195 dblp:conf/i-semantics/ZaveriKSBMAL13 fatcat:loseqbfq3zezbb4fysujsi5kl4

Quality Assessment of Crowdsourced Data against Custom Recorded Map Data

Sukhjit Singh Sehra, Hardeep Singh Rai, Jaiteg Singh
2015 Indian Journal of Science and Technology  
Web 2.0 has impelled crowdsourcing among Internet users.  ...  The datasets analysed based on specific elements of data quality defined by ISO/TC211, such as on coverage, accuracy and completeness.  ...  It deals with the structural integrity of a given data set based on formal framework for modeling of spatial data and relationships among objects.  ... 
doi:10.17485/ijst/2015/v8i33/79884 fatcat:f4baiiamhzauhfl3lbgsrve274

Open Community-Based Crowdsourcing Geoportal for Earth Observation Products: A Model Design and Prototype Implementation

Mohammad H. Vahidnia, Hossein Vahidi
2021 ISPRS International Journal of Geo-Information  
The key contribution lies in the conceptualization of the frameworks for automated publishing of standard map services such as the Web Map Service (WMS) and the Web Coverage Service (WCS) from heterogeneous  ...  EO products prepared by volunteers as well as the communication portion to request voluntary publication of the map services and giving feedback for quality assessment and assurance.  ...  Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/ijgi10010024 fatcat:a23dnb7bgvhjxjd54etw4ogtgy

Crowdsourcing Linked Data Quality Assessment [chapter]

Maribel Acosta, Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Sören Auer, Jens Lehmann
2013 Lecture Notes in Computer Science  
Based on this analysis, we implemented a quality assessment methodology for Linked Data that leverages the wisdom of the crowds in different ways: (i) a contest targeting an expert crowd of researchers  ...  The results show that the two styles of crowdsourcing are complementary and that crowdsourcing-enabled quality assessment is a promising and affordable way to enhance the quality of Linked Data.  ...  This work was supported by grants from the European Union's 7th Framework Programme provided for the projects EUCLID (GA no. 296229), Geo-Know (GA no. 318159) and LOD2 (GA no. 257943).  ... 
doi:10.1007/978-3-642-41338-4_17 fatcat:oquvmpsssvdkpmh4lpyzlnpb3u

A crowdsourceable QoE evaluation framework for multimedia content

Kuan-Ta Chen, Chen-Chi Wu, Yu-Chun Chang, Chin-Laung Lei
2009 Proceedings of the seventeen ACM international conference on Multimedia - MM '09  
In this paper, we propose a crowdsourceable framework to quantify the QoE of multimedia content.  ...  The advantages of our framework over traditional MOS ratings are: 1) it enables crowdsourcing because it supports systematic verification of participants' inputs; 2) the rating procedure is simpler than  ...  Acknowledgements The authors would like to thank Wei Tsang Ooi and the anonymous reviewers for their constructive comments.  ... 
doi:10.1145/1631272.1631339 dblp:conf/mm/ChenWCL09 fatcat:byqiddgf5jfpbfzeq2yad4fzam

Quality Control in Crowdsourcing Systems: Issues and Directions

M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H. R. Motahari-Nezhad, E. Bertino, S. Dustdar
2013 IEEE Internet Computing  
This article proposes a framework for characterizing various dimensions of quality control in crowdsourcing systems, a critical issue.  ...  As a new distributed computing model, crowdsourcing lets people leverage the crowd's intelligence and wisdom toward solving problems.  ...  The requester should choose contributions that reach his or her accepted level of quality for the outcome. Quality is a subjective issue in general.  ... 
doi:10.1109/mic.2013.20 fatcat:p3xpvckwcrfplcmoh5xq4e26pm
« Previous Showing results 1 — 15 out of 5,612 results