How to Build an Open-Domain Question Answering System?

Lilian Weng published a post reviewing common approaches for building an open-domain question answering (ODQA) system. An updated note dated 2020-11-12 adds an example of closed-book factual QA using the OpenAI API (beta). ODQA is defined as producing answers to factoid questions in natural language, where the "open-domain" part refers to the lack of relevant context for an arbitrarily asked factual question; when both question and context are provided, the task is reading comprehension.

The post states that an ODQA model may work with or without access to an external knowledge source such as Wikipedia, referred to as open-book or closed-book question answering. It cites Lewis et al., 2020 for a three-level classification of open-domain questions in increasing difficulty, and for the finding that 58-71% of test-time answers also appear somewhere in training sets, while 28-34% of test-set questions have a near-duplicate paraphrase in their corresponding training sets.

The open-book retriever-reader framework, first proposed in DrQA (Chen et al., 2017), decomposes answering into finding related context in an external knowledge repository and processing the retrieved context to extract an answer. Retrievers use classic IR (TF-IDF-based) or neural IR, with dense representations encoded by a language model and ranked by dot-product score. The post describes ORQA, REALM, and DPR using such a scoring function, and DenSPI encoding phrases offline for nearest-neighbor search. Reader models covered include DrQA's 3-layer bidirectional LSTM and BERT-based readers such as BERTserini and Multi-passage BERT. End-to-end joint training covers R^3, ORQA, REALM, and DPR.