Filters








449 Hits in 5.0 sec

BLEU is Not Suitable for the Evaluation of Text Simplification [article]

Elior Sulem, Omri Abend, Ari Rappoport
2018 arXiv   pre-print
In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation.  ...  BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects.  ...  Acknowledgments We would like to thank the annotators for participating in our generation and evaluation experiments. We also thank the anonymous reviewers for their helpful advices.  ... 
arXiv:1810.05995v1 fatcat:wlhjmytl7neyplex2z3voiu3qm

BLEU is Not Suitable for the Evaluation of Text Simplification

Elior Sulem, Omri Abend, Ari Rappoport
2018 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing  
In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation.  ...  BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects.  ...  Acknowledgments We would like to thank the annotators for participating in our generation and evaluation experiments. We also thank the anonymous reviewers for their helpful advices.  ... 
doi:10.18653/v1/d18-1081 dblp:conf/emnlp/SulemAR18 fatcat:2bavglf6e5bfrdroblf6tse4ru

Controllable Text Simplification with Lexical Constraint Loss

Daiki Nishihara, Tomoyuki Kajiwara, Yuki Arase
2019 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop  
We propose a method to control the level of a sentence in a text simplification task.  ...  Text simplification is a monolingual translation task translating a complex sentence into a simpler and easier to understand the alternative.  ...  These studies do not consider the level of each sentence. Controllable Text Simplification In addition to W-SW, Newsela (Xu et al., 2015) is a famous dataset available for text simplification.  ... 
doi:10.18653/v1/p19-2036 dblp:conf/acl/NishiharaKA19 fatcat:cxr64nhkgvcqdp3u7uoxt7x2xm

A Deeper Exploration of the Standard PB-SMT Approach to Text Simplification and its Evaluation

Sanja Štajner, Hannah Bechara, Horacio Saggion
2015 Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)  
Additionally, we point out several important differences between cross-lingual MT and monolingual MT used in text simplification, and show that BLEU is not a good measure of system performance in text  ...  Motivated by those results, we investigate the influence of quality vs quantity of the training data on the effectiveness of such a MT approach to text simplification.  ...  Acknowledgements The research described in this paper was partially funded by the project SKATER  ... 
doi:10.3115/v1/p15-2135 dblp:conf/acl/StajnerBS15 fatcat:jaqemxc3cbenxextrfw34upsma

Rethinking Automatic Evaluation in Sentence Simplification [article]

Thomas Scialom, Louis Martin, Jacopo Staiano, Éric Villemonte de la Clergerie, Benoît Sagot
2021 arXiv   pre-print
To investigate this phenomenon further, we release a new corpus of evaluated simplifications, this time not generated by systems but instead, written by humans.  ...  In the context of Sentence Simplification, this is particularly challenging: the task requires by nature to replace complex words with simpler ones that shares the same meaning.  ...  BLEU (Papineni et al., 2002) measures the overlap of n-grams between a reference text and the evaluated one.  ... 
arXiv:2104.07560v2 fatcat:qtfesadq7nhutp32dpskoidzzm

Optimizing Statistical Machine Translation for Text Simplification

Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, Chris Callison-Burch
2016 Transactions of the Association for Computational Linguistics  
Our work is the first to design automatic metrics that are effective for tuning and evaluating simplification systems, which will facilitate iterative development for this task.  ...  In this paper, we conduct an indepth adaptation of statistical machine translation to perform text simplification, taking advantage of large-scale paraphrases learned from bilingual texts and a small amount  ...  The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of the NSF or the U.S. Government.  ... 
doi:10.1162/tacl_a_00107 fatcat:mdqn6uqfcjc5homhuci4m77ubq

ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations [article]

Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, Lucia Specia
2020 arXiv   pre-print
Furthermore, we motivate the need for developing better methods for automatic evaluation using ASSET, since we show that current popular metrics may not be suitable when multiple simplification transformations  ...  Despite these varied range of possible text alterations, current models for automatic sentence simplification are evaluated using datasets that are focused on a single transformation, such as lexical paraphrasing  ...  Acknowledgements This work was partly supported by Benoît Sagot's chair in the PRAIRIE institute, funded by the French national agency ANR as part of the "Investissements d'avenir" programme under the  ... 
arXiv:2005.00481v1 fatcat:nsiagoekprewhetbg32zagmx7a

The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification

Fernando Alva-Manchego, Carolina Scarton, Lucia Specia
2021 Computational Linguistics  
Second, we conduct the first meta-evaluation of automatic metrics in Text Simplification, using our new dataset (and other existing data) to analyse the variation of the correlation between metrics' scores  ...  For that, we first collect a new and more reliable dataset for evaluating the correlation of metrics and human judgements of overall simplicity.  ...  This allows exploiting the alignment between TurkCorpus, HSplit (Sulem, Abend, and Rappoport 2018a) and ASSET (Alva-Manchego et al. 2020) to investigate  ... 
doi:10.1162/coli_a_00418 fatcat:53a5hn2jxfgw5oepweoi65sbwm

Data-Driven Sentence Simplification: Survey and Benchmark

Fernando Alva-Manchego, Carolina Scarton, Lucia Specia
2020 Computational Linguistics  
In this article, we survey research on SS, focusing on approaches that attempt to learn how to simplify using corpora of aligned original-simplified sentence pairs in English, which is the dominant paradigm  ...  We expect that this survey will serve as a starting point for researchers interested in the task and help spark new ideas for future developments.  ...  As such, BLEU should not be used as the only metric for evaluation and comparison of SS models.  ... 
doi:10.1162/coli_a_00370 fatcat:k7mlggplrreudk5pgq62x2fmva

Automatic Lexical Simplification for Turkish [article]

Ahmet Yavuz Uluslu
2022 arXiv   pre-print
In this paper, we present the first automatic lexical simplification system for the Turkish language.  ...  Being a low-resource language in terms of available resources and industrial-strength tools, it makes the text simplification task harder to approach.  ...  BLEU score for the evaluation of text simplification was recently disputed (Sulem et al., 2018) . However, our method is out of scope for the major shortcomings mentioned such as sentence splitting.  ... 
arXiv:2201.05878v2 fatcat:oafusm7bbna4pey4cwzzyeldoe

Learning How to Simplify From Explicit Labeling of Complex-Simplified Text Pairs

Fernando Alva-Manchego, Joachim Bingel, Gustavo Henrique Paetzold, Carolina Scarton, Lucia Specia
2017 Zenodo  
Current research in text simplification has been hampered by two central problems: (i) the small amount of high-quality parallel simplification data available, and (ii) the lack of explicit annotations  ...  End-to-end models also make it hard to interpret what is actually learned from data. We propose a method that decomposes the task of TS into its sub-problems.  ...  Acknowledgements This work was partly supported by the EC project SIMPATICO (H2020-EURO-6-2015, grant number 692819).  ... 
doi:10.5281/zenodo.1042505 fatcat:vcmaka3d7fgxdiclvdx4qxo4f4

Break it Down for Me: A Study in Automated Lyric Annotation

Lucas Sterckx, Jason Naradowsky, Bill Byrne, Thomas Demeester, Chris Develder
2017 Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing  
We introduce the task of automated lyric annotation (ALA). Like text simplification, a goal of ALA is to rephrase the original text in a more easily understandable manner.  ...  This motivates the need for systems that can understand the ambiguity and jargon found in such creative texts, and provide commentary to aid readers in reaching the correct interpretation.  ...  Acknowledgments This work was supported the Research Foundation -Flanders (FWO) and the U.K. Engineering and Physical Sciences Research Council (EPSRC grant EP/L027623/1).  ... 
doi:10.18653/v1/d17-1220 dblp:conf/emnlp/SterckxNBDD17 fatcat:34jlw35k3fcgjb7q6bzp6nyv2i

Towards more patient friendly clinical notes through language models and ontologies [article]

Francesco Moramarco, Damir Juric, Aleksandar Savkov, Jack Flann, Maria Lehl, Kristian Boda, Tessa Grafen, Vitalii Zhelezniak, Sunir Gohil, Alex Papadopoulos Korfiatis, Nils Hammerla
2021 arXiv   pre-print
Also, we define a novel text simplification metric and evaluation framework, which we use to conduct a large-scale human evaluation of our method against the state of the art.  ...  We present a novel approach to automated simplification of medical text based on word frequencies and language modelling, grounded on medical ontologies enriched with layman terms.  ...  Traditional evaluation metrics There are three general evaluation approaches for simplification that have been tried in the past: • BLEU score 34 is one of the standard metrics of success in machine  ... 
arXiv:2112.12672v1 fatcat:njwnkr25vbcx5fvm4zuupla27i

Document-Level Text Simplification: Dataset, Criteria and Baseline [article]

Renliang Sun, Hanqi Jin, Xiaojun Wan
2021 arXiv   pre-print
Then, we propose a new automatic evaluation metric called D-SARI that is more suitable for the document-level simplification task.  ...  Text simplification is a valuable technique. However, current research is limited to sentence simplification.  ...  Xiaojun Wan is the corresponding author.  ... 
arXiv:2110.05071v1 fatcat:ncr3de3kbzdrljrd5rwkeqp4qq

Learning to Simplify Children Stories with Limited Data [chapter]

Tu Thanh Vu, Giang Binh Tran, Son Bao Pham
2014 Lecture Notes in Computer Science  
In this paper, we examine children stories and propose a text simplification system to automatically generate simpler versions of the stories and, therefore, make them easier to understand for children  ...  Our system learns simplifications from limited data built from a small repository of short English stories for children and can perform important simplification operations, namely splitting, dropping,  ...  Acknowledgments We would like to thank The Vietnam National Foundation for Science and Technoloogy Development (NAFOSTED) for financial support.  ... 
doi:10.1007/978-3-319-05476-6_4 fatcat:nu7kkdbitzhszawf7new4z3vbu
« Previous Showing results 1 — 15 out of 449 results