Browse Prior Art Database

Automatic error analysis advisor for question answering systems

IP.com Disclosure Number: IPCOM000247245D
Publication Date: 2016-Aug-17
Document File: 5 page(s) / 137K

Publishing Venue

The IP.com Prior Art Database

Abstract

A system and method for automatic error analysis for a question answering systems is disclosed.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 28% of the total text.

Page 01 of 5

Automatic error analysis advisor for question answering systems

Disclosed is a system and method for automatic error analysis for a question answering (QA) systems. In contrast with the current state-of-the-art practices, which rely either

on statistical machine learning algorithms to improve performance or on intensive manual diagnosis, the disclosed method automates the diagnostic, solution exploration, and testing processes.

There are many reasons why a QA system can fail to produce a correct answer to a question in any given case. The more complex and comprehensive the QA system, the greater the number of possible failure modes. That is one reason why a statistical machine learning approach that focuses mainly on the scoring module of the QA system is an attractive solution that has been used effectively in certain domains. However, one problem with such approaches is that they require an enormous amount of training data. In most real-world domains and applications such quantities of data

are simply not available. Additionally, statistical systems are somewhat limited in the kinds of "corrections" they can make to a QA system. Typically the corrections are adjustments to weights in some function used to score candidate answers on the basis of a fixed set of features. There are simply some categories of error that such an approach cannot correct, e.g., the absence of information in the corpus that contains a correct answer to the question at hand.

For these reasons, a manual diagnosis of problems is often the only practical way in

which a QA system's performance can be improved. This process is often referred to as "Error Analysis" and it involves attempting to figure out what went wrong in a particular QA episode by determining which system algorithms and features played a significant role in putting forth incorrect answers. The manual diagnosis continues by determining why the system algorithms and features that might have come up with the correct answer did not perform as well as they might have. By analyzing failures in this

way. one can hope to arrive at hypotheses and to identify measures that can be taken to improve the performance of the system, not only on a single QA episode, but on a more general set of similar episodes.

The disclosed method seeks to remedy the situation described above. As a first step a

system is built on top of the underlying QA architecture in such a way that the type of information required for diagnosing the problem in any given QA episode can be automatically identified and generated in a form that is conducive to forming reasonable hypotheses for system improvement. As a second step the disclosed method has the

capability of automatically suggesting and testing actions that could be taken to correct the problem. Finally, as a third step, the disclosed method has the capability to perform experimental tests with these suggested fixes to determine what the impact of their implementation would...