23,267 Hits in 4.0 sec

Task-oriented dependency parsing evaluation methodology

Alexander Volokh, Gunter Neumann
2012 2012 IEEE 13th International Conference on Information Reuse & Integration (IRI)  
First, because it is being done for a domain which is almost always different from the domain of the application and second because many of the tested dependencies are irrelevant for the application.  ...  We propose a method which combines the strengths of attachment scores and extrinsic evaluation and avoids their weaknesses.  ...  For many NLP tasks, such as MT being the extreme example, their evaluation is at least as complex as the task itself. For dependency parsing for a long time it was not the case.  ... 
doi:10.1109/iri.2012.6303001 dblp:conf/iri/VolokhN12 fatcat:rnxfukntkfcy7enuhfggtrwpke

Using Spatial Semantic and Pragmatic Fields to Interpret Natural Language Pick-and-Place Instructions for a Mobile Service Robot [chapter]

Juan Fasola, Maja J. Matarić
2013 Lecture Notes in Computer Science  
The design and implementation details of our methodology are also presented, including the grammar utilized and our procedure for pruning multiple candidate parses based on context.  ...  We present a methodology for enabling mobile service robots to follow natural language instructions for object pick-and-place tasks from nonexpert users, with and without user-specified constraints, and  ...  In contrast, the phrase "Put the cup on the bookcase in the kitchen" has three possible valid parses, listed below: In evaluating each candidate parse, our methodology first attempts to ground the NPs  ... 
doi:10.1007/978-3-319-02675-6_50 fatcat:i2y3r2jypbdanb6tiwzpquxquy

Task-oriented Evaluation of Syntactic Parsers and Their Representations

Yusuke Miyao, Rune Sætre, Kenji Sagae, Takuya Matsuzaki, Jun'ichi Tsujii
2008 Annual Meeting of the Association for Computational Linguistics  
We evaluate eight parsers (based on dependency parsing, phrase structure parsing, or deep parsing) using five different parse representations.  ...  This paper presents a comparative evaluation of several state-of-the-art English parsers based on different frameworks.  ...  A major drawback of our methodology is that the evaluation is indirect and the results depend on a selected task and its settings.  ... 
dblp:conf/acl/MiyaoSSMT08 fatcat:2ronjncgprbijibkk3n2dtywh4

How important is syntactic parsing accuracy? An empirical evaluation on rule-based sentiment analysis

Carlos Gómez-Rodríguez, Iago Alonso-Alonso, David Vilares
2017 Artificial Intelligence Review  
In this article, we perform an empirical, task-oriented evaluation to determine how parsing accuracy influences the performance of a state-of-the-art rule-based sentiment analysis system that determines  ...  In particular, we evaluate the system using four well-known dependency parsers, including both current models with state-of-the-art accuracy and more innacurate models which, however, require less computational  ...  Experimental methodology The aim of our experiments is to show how parsing accuracy influences polarity classification, following a task-oriented evaluation.  ... 
doi:10.1007/s10462-017-9584-0 fatcat:7qjl25sidrb43bzjggxv63hztm

Aspects Based Opinion Mining for Teacher and Course Evaluation

2019 Sukkur IBA Journal of Computing and Mathematical Sciences  
Teacher and course evaluation by students at the end of each term is an important task in almost every academic institution worldwide.  ...  The data collected from evaluation comprises of two parts, Likert Scale and open-ended feedback.  ...  Once the parse tree for desired input is generated using parse annotator, the Grammatical Structure stores dependency relations between the nodes in a tree.  ... 
doi:10.30537/sjcms.v3i1.375 fatcat:3fm5up2webc4di4acpyt3z73m4

Generalizing Natural Language Analysis through Span-relation Representations [article]

Zhengbao Jiang, Wei Xu, Jun Araki, Graham Neubig
2020 arXiv   pre-print
We perform extensive experiments to test this insight on 10 disparate tasks spanning dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based  ...  Finally, we convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis.  ...  Dependency parsing (Kübler et al., 2009) is also a relation-oriented task that aims to relate a word (single-word span) to its syntactic parent word with the corresponding dependency type.  ... 
arXiv:1911.03822v2 fatcat:7uopl3lytvczrgyh2yxlzvhvtu


Shaidah Jusoh, Fangju Wang, Simon X. Yang
2005 IFAC Proceedings Volumes  
The methodology uses a fuzzy approach and history knowledge. The history knowledge is an innovation in this work.  ...  This paper presents a new methodology for creating an intelligent NLI processor for a mobile service robot that uses a behaviour-based control.  ...  Thus, a new methodology for creating an intelligent NLI processor that is able to convert an ambiguous user-task behaviour into an unambiguous task-oriented behaviour is presented in this paper.  ... 
doi:10.3182/20050703-6-cz-1902.01417 fatcat:v54xg3kdn5fsjmqyv52qi7ic7i

On Evaluation of Natural Language Processing Tasks - Is Gold Standard Evaluation Methodology a Good Solution?

Vojtěch Kovář, Miloš Jakubíček, Aleš Horák
2016 Proceedings of the 8th International Conference on Agents and Artificial Intelligence  
We discuss problems and validity of this type of evaluation, for various tasks, and illustrate the problems on examples. Then we propose using application-driven evaluations, wherever it is possible.  ...  Usually, some form of gold standard data is used for evaluation of various NLP tasks, ranging from morphological annotation to semantic analysis.  ...  The tasks where high IAA cannot On Evaluation of Natural Language Processing Tasks -Is Gold Standard Evaluation Methodology a Good Solution?  ... 
doi:10.5220/0005824805400545 dblp:conf/icaart/KovarJH16 fatcat:g4bjvhxfybdqlfjv2opefth34y

State of the Art Language Technologies for Italian: The EVALITA 2014 Perspective

Giuseppe Attardi, Valerio Basile, Cristina Bosco, Tommaso Caselli, Felice Dell'Orletta, Simonetta Montemagni, Viviana Patti, Maria Simi, Rachele Sprugnoli, Salvatore Ruggieri
2015 Intelligenza Artificiale  
Shared task evaluation campaigns represent a well established form of competitive evaluation, an important opportunity to propose and tackle new challenges for a specific research area and a way to foster  ...  The motivations underlying individual shared tasks, the level of knowledge and development achieved within each of them, the impact on applications, society and economy at large as well as directions for  ...  Its central role in a wide range of applications makes dependency parsing a still interesting task to be explored in evaluation campaigns.  ... 
doi:10.3233/ia-150076 fatcat:x6nfr34bubflzpang7e6otw2pa

Page 1390 of Linguistics and Language Behavior Abstracts: LLBA Vol. 24, Issue 3 [page]

1990 Linguistics and Language Behavior Abstracts: LLBA  
use; eye movement/ cognitive processes relationship; 9005673 recall, prereading rhetorically oriented framework effects, structurally different expository texts; 9005061 recoding/ presentation format  ...  application validity; 9004967 dyslexia research, skill automatization/conscious compensation hy- pothesis evaluated; dyslexic children aged 12-13; 9007309 English as a second language instruction, vocabulary  ... 

A framework for structured knowledge extraction and representation from natural language via deep sentence analysis

Stefania Costantini, Niva Florio, Alessio Paolucci
2011 Italian Conference on Computational Logic  
We present a framework that we are currently developing, that allows one to extract knowledge from natural language sentences using a deep analysis technique based on linguistic dependencies.  ...  The task based evaluation seems to be the best one [16 , ?]: we must choose whether to use a parser rather than another simply basing on our needs.  ...  The ontology-oriented integration is at a very early stage, and there is room for substantial improvements, including a better usage of the current reference ontologies, and the evaluation study about  ... 
dblp:conf/cilc/CostantiniFP11 fatcat:jruqcntp6vfttbr3mdqswfmr5m

Scheduled Multi-Task Learning: From Syntax to Translation [article]

Eliyahu Kiperwasser, Miguel Ballesteros
2018 arXiv   pre-print
We apply this methodology to jointly learn the multiple tasks, however many of the tasks are not of sequential nature (such as dependency parsing in which the output should be a well-formed dependency  ...  We evaluate the dependency parsing scores and the part-of-speech tagging accuracy of the models tuned to perform translation on the held-out development set.  ... 
arXiv:1804.08915v1 fatcat:atldrozgujdqjfazojiowug24a

Concurrent Lexicalized Dependency Parsing: The ParseTalk Model

Norbert Broeker, Udo Hahn, Susanne Schacht (Computational Linguisitcs Research Group, Freiburg University, Germany)
1994 arXiv   pre-print
A grammar model for concurrent, object-oriented natural language parsing is introduced.  ...  Complete lexical distribution of grammatical knowledge is achieved building upon the head-oriented notions of valency and dependency, while inheritance mechanisms are used to capture lexical generalizations  ...  , object-oriented parsing (cf.  ... 
arXiv:cmp-lg/9410017v1 fatcat:3pd6tmi27fg7vgafgyuiijwome

Cross-TOP: Zero-Shot Cross-Schema Task-Oriented Parsing [article]

Melanie Rubino, Nicolas Guenon des Mesnards, Uday Shah, Nanjiang Jiang, Weiqi Sun, Konstantine Arkoudas
2022 arXiv   pre-print
Deep learning methods have enabled task-oriented semantic parsing of increasingly complex utterances.  ...  In this paper we describe Cross-TOP (Cross-Schema Task-Oriented Parsing), a zero-shot method for complex semantic parsing in a given vertical.  ...  To evaluate our methodology, we focus on tasks in the food-ordering domain, where each task contains examples from a restaurant with the schema generated from its menu.  ... 
arXiv:2206.05352v1 fatcat:ltbhyuifnvcyzkpn6urh7hdn2i

Assessing Data Efficiency in Task-Oriented Semantic Parsing [article]

Shrey Desai, Akshat Shrivastava, Justin Rill, Brian Moran, Safiyyah Saleem, Alexander Zotov, Ahmed Aly
2021 arXiv   pre-print
Data efficiency, despite being an attractive characteristic, is often challenging to measure and optimize for in task-oriented semantic parsing; unlike exact match, it can require both model- and domain-specific  ...  We apply our protocol in two real-world case studies -- model generalizability and intent complexity -- illustrating its flexibility and applicability to practitioners in task-oriented semantic parsing  ...  We experiment with a range of task-oriented semantic parsing models when conducting case studies.  ... 
arXiv:2107.04736v1 fatcat:laaghpg5wnhepocst77kybf5ce
« Previous Showing results 1 — 15 out of 23,267 results