Open- vs. Restricted-Domain QA Systems in the Biomedical Field

María-Dolores Olvera-Lobo, Juncal Gutiérrez-Artacho
2011 Journal of information science  
Question answering systems (QA systems) stand as a new alternative for information retrieval systems. We conducted a study to evaluate the efficiency of QA systems as terminological sources for physicians, specialized translators and users in general. To this end we analysed the performance of two open-domain and two restricted-domain QA systems. The research entailed a collection of 150 definitional questions from WebMed. We studied the sources that QA systems used to retrieve the answers, and
more » ... later applied a range of evaluation measures to mark the quality of answers. Through analysing the results obtained by asking the 150 questions in the QA systems MedQA, START, QuALiM and HONqa, it was possible to evaluate the systems' operation through applying specific metrics (MRR, FHS, TRR, Precision, Recall). Despite the limitations demonstrated by these systems, it has been confirmed that these four QA systems are valid and useful for obtaining definitional medical information in that they offer coherent and precise answers. Keywords biomedical information; evaluation; open-domain question answering systems; question answering systems; restricted-domain question answering systems Journal of Information Science 37(2) 152-162
doi:10.1177/0165551511398575 fatcat:7au34kspejhl3pr3n2vqmyxvwu