Measuring Hallucination Risk Under NIST AI 600-1

Every enterprise deploying a large language model eventually has the same conversation. Someone asks the chatbot a question, gets a confident, well-formatted, completely wrong answer, and asks legal whether the company is exposed. NIST gave that problem a name in July 2024: confabulation. Not a bug report. A named risk category, with a definition, a place in a formal risk management framework, and an expectation that organizations measure and document it the way they'd document a quality deviation or a security finding.

That's the shift I want to walk through here. NIST AI 600-1, the Generative AI Profile that NIST published as a companion to the AI Risk Management Framework, treats confabulation as a controllable risk rather than an inherent quirk of the technology. If you're building an AI governance program and you're still describing hallucination as "a known limitation of the model," you're behind where the standard already is.

What NIST AI 600-1 Actually Says About Confabulation

NIST AI 600-1, formally titled Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, is the generative AI-specific companion to NIST AI 100-1, the AI Risk Management Framework 1.0 that NIST released in January 2023. The Generative AI Profile doesn't replace the original framework's four functions — Govern, Map, Measure, Manage — it applies them specifically to the risks generative AI introduces or makes worse.

Confabulation is one of twelve risk categories NIST identifies in the profile as novel to, or exacerbated by, generative AI. The document defines it directly: confabulation is the production of confidently stated but erroneous or false content, produced by generative AI systems, that may deceive or mislead users. NIST is explicit that this is the same phenomenon commonly called "hallucination" or "fabrication," and I think that direct equivalence matters. It tells you NIST isn't inventing a new problem — it's giving an existing, poorly-defined problem the precision a risk register needs.

The other eleven categories sit alongside confabulation in the profile: CBRN information, dangerous or violent content, data privacy, environmental impact, harmful bias and homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene or abusive content, and value chain and component integration risks. Confabulation touches nearly all of them. A hallucinated citation is an information integrity problem. A fabricated safety claim in a regulated product context edges toward CBRN or dangerous-content territory. A model that invents a customer's account history is a data integrity and privacy problem at once. That overlap is exactly why NIST didn't leave confabulation as a footnote — it built out specific control expectations for it.

Why NIST Chose "Confabulation" Over "Hallucination"

The word choice isn't cosmetic. "Hallucination" implies a sensory malfunction — a system perceiving something that isn't there. "Confabulation" is a term borrowed from clinical psychology, describing a person who fills a memory gap with a fabricated but sincerely believed account. That's a closer description of what a language model actually does: it doesn't perceive falsely, it completes a pattern with statistically plausible tokens regardless of whether those tokens correspond to anything true.

I've found this distinction useful in board-level conversations. "Hallucination" invites the response "well, all AI hallucinates sometimes, it's a known tradeoff." "Confabulation" invites a harder question: what's our rate, and how do we know it? NIST's terminology does some of your governance selling for you. Use it.

The Four-Function Control Structure

NIST AI 600-1 organizes its suggested actions using the same four functions as the base framework, and confabulation controls show up in all four — not just in testing, which is where most organizations instinctively put "AI accuracy" work.

The mistake I see most often is organizations doing Measure without doing Govern first. They run an evaluation suite, get a number, and have no policy that says what number is acceptable for which use case.

A 3% confabulation rate might be fine for a marketing copy assistant and unacceptable for a system generating dosing guidance. Without a governed threshold, the measurement is just a number sitting in a slide deck.

Building a Measurement Program

Measurement is where most teams get stuck, because "measure hallucination" sounds simple until you try to define what counts. NIST AI 600-1 doesn't hand you a single metric — it calls for assessing AI system performance against defined conditions similar to the deployment setting, which puts the burden on you to define what "similar to deployment" means for your system. In practice, a defensible measurement program combines several approaches, because no single one catches everything.

Groundedness scoring

For retrieval-augmented systems, this checks whether every factual claim in an output traces back to a passage in the retrieved source material. A claim with no supporting passage is a candidate confabulation. This is the cheapest control to automate and the first one I recommend building.

Closed-domain benchmark testing

Run the system against a fixed set of questions where you already know the correct answer — internal policy questions, product specifications, regulatory text. This gives you a repeatable, comparable confabulation rate over time, which matters more than any single snapshot number. A rate that's stable at 2% is a different governance conversation than a rate that jumped from 2% to 9% after a model update.

Human adjudication sampling

Automated groundedness checks miss confabulations that are technically consistent with retrieved text but still misleading — a subtly wrong inference, an overstated confidence level, a fabricated number dressed up next to a real one. A trained reviewer sampling a defined percentage of production outputs on a fixed cadence catches what automated checks miss, and gives you the documentation trail an auditor will actually want to see.

Adversarial and edge-case testing

Deliberately probing the system with questions designed to induce confabulation — asking about events after the training cutoff, asking for citations to sources that don't exist, asking about narrow topics where training data is thin. This is closer to red-teaming than routine QA, and it belongs on a recurring schedule, not a one-time pre-launch check.

None of these substitute for the others. A system can pass groundedness checks and still fail human review because grounded-but-wrong is a real failure mode — the model cites a real passage but draws a conclusion the passage doesn't support.

What Documentation Actually Needs to Show

This is the part organizations underbuild. Measuring confabulation and never writing it down is functionally the same, from an audit standpoint, as never measuring it.

NIST AI 600-1's whole structure assumes a paper trail connecting governance decisions to measurement results to management action. When a customer, regulator, or acquiring company asks to see your AI risk documentation, here's the minimum I'd expect to be able to produce:

1. A written confabulation risk tolerance by use case, tied to consequence severity — not a single company-wide number.
2. Evaluation methodology documentation: what benchmark, what sample size, what pass/fail threshold, and who approved the threshold.
3. Dated evaluation results showing the confabulation rate over time, not just at launch.
4. A record of every production incident where confabulation caused a downstream problem, with root cause and remediation.
5. Evidence that the measurement cadence is actually being followed — not just a policy stating it should happen.

If you're operating under ISO/IEC 42001:2023 alongside NIST AI 600-1, this documentation does double duty. Clause 6.1.2 of ISO 42001 establishes the requirement for a formal AI risk assessment, performed at clause 8.2, and clause 6.1.4 establishes the requirement for an AI system impact assessment, performed at clause 8.4 before deployment — both of which need exactly this kind of evidence to be more than a checkbox exercise. I generally advise clients running both frameworks to build one documentation set that satisfies both rather than maintaining parallel files, because the underlying evidence — the eval results, the incident logs, the tolerance thresholds — is identical either way.

How This Compares to Other Frameworks

NIST AI 600-1 isn't the only place hallucination risk shows up in AI governance. It's worth knowing how the frameworks differ, because most mid-size organizations will end up complying with more than one.

The practical takeaway: NIST gives you the vocabulary and the measurement logic, ISO 42001 gives you the certifiable management system to prove you're doing it consistently, and the EU AI Act gives you the legal floor if you operate in or sell into the EU. None of them tell you the actual number to hit. That threshold-setting is a judgment call NIST deliberately leaves to the organization, because acceptable confabulation risk in a customer service chatbot is not the same as acceptable confabulation risk in a system drafting regulatory submissions.

A Practical Implementation Sequence

Organizations often ask where to start, especially mid-size companies without a dedicated AI governance team. I've written before about sequencing a full NIST AI RMF rollout for a 200-person company, and the same logic applies at smaller scale to confabulation controls specifically:

1. Classify your generative AI use cases by consequence severity before you build any measurement tooling. This is a half-day workshop, not a research project.
2. Stand up groundedness scoring for any retrieval-augmented system first — it's the highest-value, lowest-cost control.
3. Set a documented tolerance threshold per use case, signed off by whoever owns the business risk, not just IT.
4. Build the human sampling review cadence before you need it, not after a customer complains.
5. Log every confabulation incident from day one, even minor ones. A thin incident log looks worse in an audit than an honest one showing you catch and fix problems.

Common Pitfalls

The most common failure I see is treating confabulation testing as a one-time pre-launch gate. Models get updated, retrieval indexes get refreshed, prompts get tweaked by well-meaning product teams — and confabulation rates drift. A control that only runs once tells you about the system that existed at launch, not the one running today.

The second is conflating confidence with correctness in the output itself. Models are, by construction, fluent. A confabulated answer reads exactly like a correct one — same tone, same structure, same apparent certainty. That's precisely why NIST's definition centers on the word "confidently." Any evaluation program that relies on a human reviewer's gut sense of whether an answer "sounds right" will systematically miss confabulations, because sounding right is the one thing confabulated text does reliably.

The third is measuring at the model level and forgetting the system level. A foundation model's benchmark confabulation rate tells you almost nothing about your deployed system's rate once you've added retrieval, prompt engineering, and guardrails on top of it. NIST AI 600-1's emphasis on testing "under conditions similar to deployment" exists specifically to close that gap. Test the system your users actually touch.

FAQ

What is confabulation under NIST AI 600-1?

Confabulation is NIST's formal term, defined in AI 600-1, for the production of confidently stated but erroneous or false content by a generative AI system — the same phenomenon commonly called "hallucination." NIST treats it as one of twelve risk categories unique to or worsened by generative AI.

Is NIST AI 600-1 mandatory?

No. NIST AI 600-1 and the underlying AI Risk Management Framework are voluntary, non-regulatory guidance. However, the framework increasingly appears as a reference point in federal procurement requirements and in vendor risk assessments from enterprise customers, which makes adoption a practical competitive necessity even without a legal mandate.

What's the difference between confabulation and hallucination?

They describe the same underlying failure mode — NIST AI 600-1 says so directly. What matters is why the label choice is operationally useful: "hallucination" tends to get waved off in a board conversation as an unavoidable tradeoff, while "confabulation" invites a harder, measurable question — what's our rate, and how do we know it. Standardize on NIST's term in your own documentation for that reason, not because the underlying mechanism is a new discovery.

How often should we measure confabulation rates?

There's no single NIST-mandated cadence, because the right frequency depends on how often your model, prompts, or retrieval sources change and how severe the consequences of an error would be. A high-consequence use case with frequent updates warrants continuous or near-continuous sampling; a low-consequence, stable system can reasonably run on a quarterly review cycle. What matters for documentation purposes is that the cadence is defined in writing and that you can show it's actually being followed.

Does ISO 42001 certification cover confabulation risk automatically?

Not automatically. ISO/IEC 42001:2023 requires an AI risk assessment (established at clause 6.1.2, performed at 8.2) and an AI system impact assessment (established at clause 6.1.4, performed at 8.4), and confabulation would need to be scoped into those assessments explicitly for the certification to actually cover it. A generic risk assessment that never names confabulation as a category leaves a gap an auditor is likely to flag.

Where This Leaves You

Confabulation controls aren't a research problem anymore. NIST gave the industry a definition, a place in a formal risk taxonomy, and a four-function structure for building controls around it. The organizations that will struggle in the next round of vendor security reviews and AI governance audits aren't the ones with a nonzero confabulation rate — every generative AI system has one. They're the ones who can't produce a number, a threshold, and a log showing they're watching it.

If you're building this program from scratch, start with the use-case classification and the groundedness scoring, get a documented threshold signed off, and build the incident log before you need it. That sequence alone puts you ahead of most organizations still calling this "a known limitation" instead of a managed risk. For a structured look at where your organization stands against this and the rest of the AI RMF, our AI readiness assessment walks through exactly this gap analysis, and if you're specifically deploying generative AI in production, our generative AI consulting work is built around closing it.

Last updated: 2026-09-08