Anthropic published research as part of their AI Fluency Index that contains a finding I have not been able to stop thinking about. The research found that users are significantly less likely to verify AI-generated content when that content appears polished and well-formatted. The better the output looks, the more confident the tone, the cleaner the structure, the more authoritative the presentation, the less scrutiny it receives.
This is not a surprising finding if you think about it from a cognitive science perspective. We use surface quality as a proxy for content quality all the time. A well-formatted document signals effort and care. A confident tone signals expertise. These heuristics work reasonably well when applied to human-generated content, because the effort required to produce polished output usually correlates with the effort required to get the content right. AI breaks that correlation entirely. Generating polished, confident, well-structured output costs a model nothing extra. The formatting is free. The confidence is free. The authoritative tone is free. None of it is evidence of accuracy.
The practical consequences of this finding are significant for any organization deploying AI in contexts where accuracy matters.
Consider a scenario that is already common in many organizations: a staff member uses an AI tool to draft a report, a policy brief, or a client communication. The output looks good, well-organized, clearly written, appropriately confident. The staff member reviews it, makes a few edits, and sends it. The problem is that the review process was calibrated to the surface quality of the output, not to the accuracy of its claims. Errors that would have been caught in a rough draft, because rough drafts signal the need for scrutiny, slip through in a polished one.
This is not a hypothetical risk. It is a pattern I have observed in organizations that have deployed AI writing tools without thinking carefully about how those tools change the review process. The tools make output faster and more polished. They do not make it more accurate. And the polish actively undermines the scrutiny that would catch inaccuracies.
What does this mean for organizations deploying AI? A few things.
First, your AI governance framework needs to address the review process, not just the generation process. It is not enough to have a policy that says humans must review AI output. You need to think about how that review is structured, what it is designed to catch, and whether the reviewers have the context and the incentive to actually scrutinize the content rather than just approve the format.
Second, training matters. Employees who use AI tools need to understand this dynamic explicitly, that polished output is not a signal of accurate output, and that their review process needs to be calibrated accordingly. This is a specific skill that needs to be taught, not assumed.
Third, for high-stakes outputs, anything that will be sent to clients, submitted to regulators, or used to make significant decisions, consider building in structural friction. Require a second reviewer. Require the primary reviewer to document what they checked and how. Make the verification process visible and accountable.
The AI tools are not going away, and they should not. The productivity gains are real. But the governance frameworks need to catch up with the specific ways these tools change human behavior, including the ways they make us less careful precisely when we should be more careful.