InnovationQ will be updated on Sunday, May 20, from 10am - noon ET. You may experience brief service interruptions during that time.
Browse Prior Art Database

System and Method to identify increases and decreases in accuracy after ingestion

IP.com Disclosure Number: IPCOM000235693D
Publication Date: 2014-Mar-20
Document File: 2 page(s) / 174K

Publishing Venue

The IP.com Prior Art Database


This article explores how accuracy can be improved in a question and answering system such as Watson when incrementally ingesting data. Not all data which is incrementally ingested will require a complete regression test of the prior questions. The article explores a way in which documents which have been updated are tracked and the subsequent questions which were updated will only need to be used for regression testing. Additionally, the article explores how the incremental updates to documents can be tracked to identify which ones cause regression in previously asked questions.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 51% of the total text.

Page 01 of 2

System and Method to identify increases and decreases in accuracy after ingestion

If one can imagine a scenario where IBM Watson* is performing incremental ingestion of data. Large amounts of this data may not necessarily be new, but rather updates to existing data. An important aspect of this process is quickly identifying data which may be inexact or deviate widely from the norm and potentially cause inaccuracies for users when asking questions of the newly updated system . The inaccuracies may be reflected in lower confidence values for the answers or lower passage weighting for relevancy .

Historically, one way to measure accuracy is to have a set of test questions which are asked of the system and the overall accuracy performance is measured. This can be time consuming and may not be necessary when only a small subset of documents have been updated in the corpus through incremental ingestion. In addition, this set of test questions may not "touch" the updated documents so there is no way to know if a user has caused an accuracy regression.

This invention provides for the automated testing and tracking of what documents have changed and the questions which were impacted by the change to uncover increases or decreases in accuracy.

Watson begins by storing all the questions that have been asked of the system in addition to artifact information such as the confidence value , passage relevancy score, and documents which were accessed to answer the question. As enabling art, this is done today with what is referred to as a file based answer cache.

When a document or documents are updated through incremental ingestion , the system refers back to the list of questions which accessed the changed passage(s) within the source document. These questions are then used to automatically form the set of questions which are asked of the system during regression testing prior to putting a new corpus into production . Decreases in the accuracy may indicate that parts of the newly ingested data are inaccurate, the question/answer sets need updating (stored answer no longer correct) or the need for retraining on the new corpus.

1) After answering a question, the system maintains text of the question asked, the answers, and information on the ids of the supporting passages, including the confidence and passage...