Plausible Nonsense: Only 4 of 60 AI Models Cleared a Reasoning Benchmark

A paper in the Proceedings of the National Academy of Sciences (PNAS) on 9 September 2026 asks a narrower question than whether AI is smart. It asks whether the reasons a language model gives line up with the choices it makes in the way they do among people who have deliberated together. Francesco Veri and Gustavo Kreia Umbelino tested 60 large language models (LLMs) against human deliberation across nine policy scenarios, using a measure called the Deliberative Reason Index (DRI). Only four models consistently exceeded the study’s permutation-based null benchmark. The rest did not, even though their outputs can still read as coherent and persuasive, which is the paper’s point: fluency and reasoning structure are different things.

What did the PNAS study actually test?

The setting is democratic governance. The authors note that LLMs are entering democratic contexts as instruments of governance, where problems are ill-structured: marked by ambiguity and contestation rather than a single correct answer. Such problems, they argue, demand more than factual precision. They call for intersubjective reasoning, meaning context-sensitive judgments that other people can understand and publicly accept. So the test is not a quiz with a key. Sixty models were compared with human deliberation across nine policy scenarios, using the DRI.

What is the Deliberative Reason Index?

The DRI measures intersubjective consistency. A later arXiv paper by Maurice Flechtner (2608.10186, 10 August 2026) describes it in terms of two correlations for pairs of participants, one across the considerations they weigh and one across the preferences they hold, and defines consistency as the similarity of the two. High consistency means that when two participants share or diverge in their reasoning about considerations, this is reflected proportionally in their shared or divergent preferences. In plain words, people who reason alike should end up choosing alike, and people who reason differently should choose differently. A reasoner whose reasons and choices do not track each other that way is not deliberating coherently, however good each sentence sounds.

What does “plausible nonsense” mean here?

The phrase names a gap the abstract states directly: outputs can appear reasonable even when the alignment with human reason-giving is absent, and the gap between surface plausibility and deliberative coherence is the reason for caution. That answers a question the title invites. The paper does not test whether a model can spot nonsense in other people’s text. It tests whether the model’s own pattern of reasons and preferences behaves like a human deliberator’s. Because fluent prose is what these systems are built to produce, fluency cannot be the test.

What did the same authors find before?

This is not the first benchmark from the team. A separate analysis by Kreia Umbelino and Veri (2025), as described in Flechtner’s paper, evaluated 54 off-the-shelf LLMs against 526 post-deliberation human DRI responses across 24 cases spanning 19 topics. Humans scored higher on average, 0.34 against 0.18 for the models (p < 0.0001), and outperformed the models in 19 of the 24 cases. At model level, humans consistently outperformed 23 of the 54 models, while the remaining majority performed on par with humans on average. The models were tested as released, without fine-tuning or access to deliberation transcripts, so Flechtner treats the result as a baseline rather than a ceiling. The PNAS paper uses 60 models and nine scenarios, so the two sets of numbers are different analyses and should not be merged.

How is this different from an IQ-style benchmark?

An IQ test scores right and wrong answers against a norm group and reports where the taker falls, in percentile terms. This benchmark scores something else: whether reasons and preferences cohere across many decisions. A model can do well on answer-based tests and still fall short on structure, and the reverse. That is one reason we argue that an AI’s score on an IQ-style test is not an IQ, and why AI benchmark scores disagree when the same models are measured in different ways.

Four questions to ask of any AI reasoning headline

• What is the baseline? A chance-level benchmark, a human benchmark and a previous model are three different bars.
• What is being scored? Answers, or the structure behind the answers.
• How many settings? Consistent results across nine scenarios are a stronger claim than one result.
• Which versions and dates? Models change every few months, so a result belongs to a snapshot.

What does this mean for reading AI “IQ” claims?

It is a reminder that one number rarely captures reasoning. Human IQ is a percentile on a norm group, with a stated mean, standard deviation and margin of error; the percentile calculator shows how that works. AI results are often reported as a single leaderboard score without any of that, as in our look at what the intelligence index says. If you want a human reference point of your own, take the IQ test and read the result as a range.

The takeaway is narrow and useful. In this study, most of 60 models did not consistently clear a bar for human-like reason-giving, and the authors want that checked before models are deployed in governance. It says little about models that were not tested and nothing about a model’s general intelligence.

Common questions