Filters








112,759 Hits in 5.2 sec

Evaluating the performance of Automated Text Scoring systems

Helen Yannakoudakis, Ronan Cummins
2015 Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications  
Various measures have been used to evaluate the effectiveness of automated text scoring (ATS) systems with respect to a human gold standard.  ...  Finally, we identify the most appropriate measures of agreement and present general recommendations for best evaluation practices.  ...  Acknowledgments We would like to thank Ted Briscoe for his valuable comments and suggestions, Cambridge English Language Assessment for supporting this research, and the anonymous reviewers for their useful  ... 
doi:10.3115/v1/w15-0625 dblp:conf/bea/YannakoudakisC15 fatcat:ktj7nwcnkjfvtogm3clgntalhm

Automated MT evaluation metrics and their limitations

Bogdan Babych
2014 Tradumàtica tecnologies de la traducció  
Automated evaluation of MT characterizes performance of MT systems on specific text or a corpus.  ...  Automated scores are expected to correlate with certain parameters of MT quality scored by human evaluators, such as adequacy of fluency of translation.  ...  The main objective of methods and tools for automated evaluation of MT is to compute numerical scores, which characterize the 'quality', or the level of performance of specific Machine Translation systems  ... 
doi:10.5565/rev/tradumatica.70 fatcat:lkr27o633vc3vla76gk7mwtpy4

Comparative Study on Feature-Based Scoring Using Vector Space Modelling System

Tarandeep Singh Walia, Tarek Frikha, Omar Cheikhrouhou, Habib Hamam, Dr. Dilbag Singh
2021 Mathematical Problems in Engineering  
Considering the potential of the automated scoring system, there is further a need to refine and develop the existing system. The paper goes through the state of the art.  ...  This paper shows the importance of automated scoring (AS) and that it is better than human graders in terms of degree of reproducibility.  ...  Omar Cheikhrouhou thanks Taif University for its support under the project Taif University Researchers Supporting Project (no. TURSP-2020/55), Taif University, Taif, Saudi Arabia.  ... 
doi:10.1155/2021/9946573 fatcat:g634sxeylfa35g57c5w6nk5p7y

CaseSummarizer: A System for Automated Summarization of Legal Texts

Seth Polsley, Pooja Jhunjhunwala, Ruihong Huang
2016 International Conference on Computational Linguistics  
It is evaluated using ROUGE and human scoring against several other summarization systems, including summary text and feedback provided by domain experts.  ...  Attorneys, judges, and others in the justice system are constantly surrounded by large amounts of legal text, which can be difficult to manage across many cases.  ...  Acknowledgments The authors would like to thank each of the domain experts who provided high-quality summaries and ratings on the systems, as well as all those who gave feedback on CaseSummarizer's various  ... 
dblp:conf/coling/PolsleyJH16 fatcat:y3dcjxu5dvg5djhph5lndbd76e

Open Problems in Indonesian Automatic Essay Scoring System

Faisal Rahutomo, Trisna Ari Roshinta, Erfan Rohadi, Indrazno Siradjuddin, Rudy Ariyanto, Awan Setiawan, Supriatna Adhisuwignjo
2018 International Journal of Engineering & Technology  
This paper presents open problems in Indonesian Scoring System. The previous study exposes the comparison of several similarity metrics on automated essay scoring in Indonesian.  ...  The study also evaluates the stemming approach for the system performance. The difference between all methods between using stemming or not is around 4-9%.  ...  The authors would like to thanks to the institution, especially UPT P2M for the full support and funding.  ... 
doi:10.14419/ijet.v7i4.44.26974 fatcat:k43k464lgncfnn2okuz36fr474

Automated Evaluation ofTelugu Text Essays Using Latent Semantic Analysis

M Varaprasad Rao, Et. al.
2021 Turkish Journal of Computer and Mathematics Education  
Automated systems are increasingly required to help students write essays. In the field of educational assessment technology, automated test evaluation is becoming more and more common.  ...  Language skills; the structure of the essay and the contents that fit the subject are the principal requirements for evaluating the essays.  ...  We can only describe AEE as a means of automated computer evaluation of the written prose [10] . Evaluation means that the computer system will do the job of scored or assigned a number to an essay.  ... 
doi:10.17762/turcomat.v12i5.2267 fatcat:h3dly3tmbrgv5i4e7llqdbruee

SURVEY PAPER ON SMART ESSAY GRADER

Prof.Nidhi Sharma, Aman Sharma, Aman Sharma, Adhoksh Sonawane
2021 International Journal of Engineering Applied Sciences and Technology  
In order to optimize Human-Machine agreement for automatic evaluation of textual summaries or essays, automated essay grading has been a research field.  ...  , in order to rate the essay with the highest level of accuracy.  ...  The results show that the system outperforms a strong baseline in automated essay scoring and achieves state-of-the-art success.  ... 
doi:10.33564/ijeast.2021.v05i11.017 fatcat:g4hyhad7izerdfgldj4o4fveyy

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models

Wangchunshu Zhou, Ke Xu
2020 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Training with the proposed metric yields better performance in human evaluation, which further demonstrates the effectiveness of the proposed model.  ...  We also propose to evaluate the model-level quality of NLG models with sample-level comparison results with skill rating system.  ...  evaluator with n human-annotated scores, 3) score-based human evaluation facilitates the evaluation of correlation scores, and 4) as all other metrics do not perform pairwise comparison, using pairwise  ... 
doi:10.1609/aaai.v34i05.6521 fatcat:xbkgmohbsvayzoiecslendjley

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models [article]

Wangchunshu Zhou, Ke Xu
2020 arXiv   pre-print
Training with the proposed metric yields better performance in human evaluation, which further demonstrates the effectiveness of the proposed model.  ...  We also propose to evaluate the model-level quality of NLG models with sample-level comparison results with skill rating system.  ...  evaluator with n human-annotated scores, 3) score-based human evaluation facilitates the evaluation of correlation scores, and 4) as all other metrics do not perform pairwise comparison, using pairwise  ... 
arXiv:2002.05058v1 fatcat:s5ykq6n6vncxdgsshunnpfefza

DOMAIN SPECIFIC AUTOMATED ESSAY SCORING USING CLOUD BASED NLP API

George Pashev, Silvia Gaftandzhieva, Yuri Hopteriev
2021 International journal of computer science and mobile computing  
These topic graphs are either to be created by the teacher or automatically generated by scanning a certain amount of sample texts in the subject area.  ...  The paper presents a methodology and an application framework (PUAnalyzeThis) that makes use of MeaningCloud API to automatically extract entities, concepts, relations, etc. and calculate scores and grades  ...  The performance of the system-generated scores is compared with the human scores using Pearson correlation. The results show that system and human scores correlatewith each other. Dong et al.  ... 
doi:10.47760/ijcsmc.2021.v10i10.006 fatcat:czlijofpcnhtjj67bqxj3efidu

Coherence Modeling for the Automated Assessment of Spontaneous Spoken Responses

Xinhao Wang, Keelan Evanini, Klaus Zechner
2013 North American Chapter of the Association for Computational Linguistics  
An analysis on the annotated corpus shows that the prediction accuracy for human holistic scores of an automated speech scoring system can be improved by around 10% relative after the addition of the coherence  ...  Further experiments indicate that a weighted F-Measure of 73% can be achieved for the automated prediction of the coherence scores.  ...  Acknowledgments The authors wish to express our thanks to the discourse annotators Melissa Lopez and Matt Mulholland for their dedicated work and our colleagues Jill Burstein and Slava Andreyev for their  ... 
dblp:conf/naacl/WangEZ13 fatcat:yj66iwkjobexratb4jzp2o3eha

Speech- and Text-driven Features for Automated Scoring of English Speaking Tasks

Anastassia Loukina, Nitin Madnani, Aoife Cahill
2017 Proceedings of the Workshop on Speech-Centric Natural Language Processing  
Although adding any single category of acoustic features to the text-only system on its own does not significantly improve performance, adding all acoustic features together does yield a small but significant  ...  We combine features from a text-only content scoring system originally designed for written responses with several categories of acoustic features.  ...  We also thank Matt Mulholland for help with processing the data.  ... 
doi:10.18653/v1/w17-4609 dblp:conf/emnlp/LoukinaMC17 fatcat:nzdlk3bji5hevppwa7lsi36tha

Bring It on! Challenges Encountered While Building a Comprehensive Tutoring System Using ReaderBench [chapter]

Marilena Panaite, Mihai Dascalu, Amy Johnson, Renu Balyan, Jianmin Dai, Danielle S. McNamara, Stefan Trausan-Matu
2018 Lecture Notes in Computer Science  
This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system.  ...  Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students' practice activities.  ...  This research was partially supported by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education  ... 
doi:10.1007/978-3-319-93843-1_30 fatcat:uxsua3sscnb53ererrgahmseue

Contactless Academia – The Case for Automated Essay Scoring (AES) System in COVID 19 Pandemic

Kennedy A. Osakwe, Kunle Ola, Pete Omotosho
2021 Current Journal of Applied Science and Technology  
An invaluable measure worth considering is the inclusion of 'Automated Essay Scoring' (AES) system in the mitigation toolkits for higher institutions of learning.  ...  The outcome of reviewed literatures varied on suitability of AES in scoring essay task in Higher Institution of Learning.  ...  Giles investigated the impact of an automated grading system (SAGrader) on student's performance.  ... 
doi:10.9734/cjast/2021/v40i431292 fatcat:hu2tfmidrzczrodg5bta7jsaiu

Effectiveness of an Online Automated Evaluation and Feedback System in an Introductory Computer Literacy Course

Ilhan Varank,
2014 Eurasia Journal of Mathematics, Science and Technology Education  
The purpose of this study was to investigate the effectiveness of an online automated evaluation and feedback system that assessed students' word processing assignments prepared with Microsoft Office Word  ...  in which the experimental group students used the online automated evaluation and feedback system to get feedback on their assignments, and the control group students didn't receive any feedback.  ...  Majority of the students found the automated evaluation system excellent and they believed it perfectly evaluated their codes and corrected the errors.  ... 
doi:10.12973/eurasia.2014.1062a fatcat:h4gwagjjqnfvrm4kms6cid4qw4
« Previous Showing results 1 — 15 out of 112,759 results