FINDING DIRECTIONLarge language models can produce inaccurate information in language that makes it appear credible.
Hallucinations pose a significant challenge for large language models. These models generate responses by finding patterns across large amounts of data and predicting what belongs in the response. The result may read naturally and fit the surrounding context while containing information that is inaccurate, internally inconsistent, or unsupported.
NIST uses the term confabulation for this behavior. Whatever term we use, the risk comes from the combination of an incorrect answer and a presentation that gives someone reason to trust it.
Hallucinations are a risk to trust and reliability, especially in applications involving medical information, legal guidance, or financial decisions. An incorrect response can harm a user, damage a company’s reputation, or create legal liability.
An application can ground responses in supplied sources, retrieve current information, constrain the expected output, test common scenarios, and require human review for consequential decisions. These measures reduce the opportunities for an unsupported answer to reach a user, although they cannot guarantee that every response will be accurate.[1]
Air Canada experienced the consequence after its chatbot gave a customer incorrect information about the company’s bereavement-fare policy. In February 2024, the British Columbia Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation and ordered it to compensate the customer.
The decision did not establish that the chatbot used a large language model. It established something broader that applies to any automated response: the company was responsible for information delivered through its website. Companies using LLMs need to verify consequential answers and give people a clear way to challenge them.