The influence of semantic roles in QAa comparative analysis

Résumé

Question sets normally used to evaluate QA systems are mainly based on questions whose answers are named entities, therefore most of these systems rely on NERs to extract possible answers. Nowadays, semantic role labeling and its contribution to question answering has recently become an interesting issue. Nevertheless, NE-based systems will always work better than SR-based ones extracting answers for questions with NE-based answers. The aim of this paper is to evaluate both of approaches for location questions under the same conditions and using not only NE-based questions but also common noun-based ones. In order to achieve this goal we present three different proposals of an answer extraction module embedded into a QA system: one based on named entities and two based on semantic roles. Results show that while NE-based approach performs better with NE-based questions (MRR +49.57%), SR-based approaches show the best results in common noun-based ones (MRR +223.48%) and obtain a higher precision in both types of questions.