GPQA Diamond Score: What It Measures and How Far AI Has Come

GPQA Diamond is a 198-question multiple-choice science benchmark, and on 6 October 2026 the highest score Artificial Analysis displayed for it was 96.3%. That sounds like a model near perfection, and it is, but the number sits on a smaller and stranger foundation than its reputation suggests. The human scores that every explainer quotes belong to a different set of questions, the test is too short to rank models that are one or two points apart, and an independent audit has asked how many of the answer keys can be trusted. This guide sets out what GPQA is, what its human baselines really are, where AI scores stand, and why it is not an IQ test.

What is GPQA?

GPQA stands for "Graduate-Level Google-Proof Q&A". The paper, by David Rein and colleagues at New York University (arXiv 2311.12022, submitted 20 November 2023), describes it as "a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry". One author lists Cohere and another, Samuel Bowman, lists Anthropic as an affiliation, and NYU's archive record lists the paper under the Proceedings of COLM. The questions were written and validated by 61 contractors hired through Upwork, each required to hold or be working towards a PhD in their field, and paid about $95 an hour on average.

Each question went through a pipeline. A writer drafted it, two expert validators answered it (one before and one after the writer's revision), and three non-expert validators then tried it. In all the team collected 564 questions and held 18 out, leaving an extended set of 546. The main set of 448 keeps questions where at least one of two experts agreed with the key and no more than two of three non-experts were right. The Diamond set of 198 keeps those where both experts were right and a majority of non-experts were wrong.

What does "Google-proof" mean?

The paper defines it by a test, not a slogan. The questions had to be hard for "highly skilled and incentivized non-experts": people with or pursuing PhDs in other fields, allowed any internet resource they could find except AI assistants. The abstract says they spent "over 30 minutes" on each question; the body gives a mean of 37 minutes and a median of 30. On the extended set they reached 34.1%. Experts in the question's own field reached 65%, or 74% when the experts' own retrospective corrections for clear mistakes are discounted. The best GPT-4 baseline of the time reached 39%. Because every question has four options, a random guesser scores 25%, a floor Epoch AI states on its benchmark page.

Are the 65% and 34% figures the Diamond scores?

No, and this is the most common slip. Table 2 of the paper separates the three sets. The extended set (546 questions) has expert accuracy of 64.8% and non-expert accuracy of 34.1%. The main set (448) shows 71.9% and 30.4%, and Diamond (198) shows 81.3% and 22.1%. The paper marks the main-set and Diamond figures with an asterisk and says validator accuracies there are "skewed by selection effects": a question only enters Diamond if both experts got it right and most non-experts got it wrong, so the human scores on Diamond are partly produced by the selection rule. Artificial Analysis's page for Diamond nonetheless describes the set as one "where PhD experts achieve 65% accuracy but skilled non-experts only reach 34%", which attaches the extended-set numbers to the smaller subset.

A separate human figure comes from OpenAI. Epoch AI's benchmark page reports that OpenAI recruited PhD-level experts to answer GPQA Diamond for its o1 announcement and that they scored 69.7%. We could not open OpenAI's own page to confirm it, so treat that as Epoch's report of it. Any of these numbers, 65%, 69.7% or 81.3%, can be called "the expert score", and they are not the same measurement.

What are the highest GPQA Diamond scores?

Artificial Analysis's evaluation page, read on 6 October 2026, said the highest score was GPT-6 Astra (Xhigh) at 96.3%, followed by GPT-6 Astra (Max) at 96.1% and Gemini 3.8 Flash (High) at 95.3%. The chart is drawn by script and covers 20 of 615 tracked models, so we could read only the top three. For comparison, the paper's Table 5 gives GPT-4's Diamond score as 38.8%. From 38.8% to 96.3% is 57.5 points in about three years (our subtraction). Every one of those top scores is far above the human figures quoted above, whichever you choose.

Scores in this range are not comparable between sites without checking the setup. Epoch AI notes that its grader gives no points if an answer is not in the form "ANSWER: LETTER", so a model that rambles can be marked below the guessing floor. Prompts, number of runs and reasoning settings (the labels "Xhigh" and "Max" above are such settings) all change a result. For why leaderboards disagree, read why AI benchmark scores disagree, and for the index that includes GPQA-type tasks, the Artificial Analysis Intelligence Index explained.

How much can 198 questions tell you?

Less than the second decimal place suggests. The paper's own limitations section says GPQA "is small at only 448 examples" and gives limited statistical power for comparing accuracies. At 198 questions each item is worth 0.505 points. Our binomial arithmetic at 80% accuracy gives a standard error of 2.84 points and a 95% confidence interval of about plus or minus 5.6 points (73.9% to 85.0% by the Wilson method). At the top of the board the picture is tighter, but the logic holds: the 1.0-point gap between 96.3% and 95.3% is about two questions, and 0.2 points between first and second is less than one.

The audit question compounds this. In May 2025 Epoch AI noted that top models were clustered around 83%, which "led one of the creators of the benchmark to speculate that there's something wrong with the other 17%". Epoch looked at where models disagreed with the key: the best 14 models, those scoring at least 70%, collectively got over 50% right on 158 of the 198 questions, with their wrong answers clustered on the other 40, and organic chemistry made up 70% of those 40 against 36% of the whole set. Of six questions it examined closely, about 2.25 were invalid, which it extrapolated, as "a stretch", to 15 of 198, or 8%. Its conclusion was that it was "likely that at least 90% of the benchmark is valid".

Since that post the best scores have moved from about 83% to 96.3%. That is hard to square with an 8% invalid rate unless the rate is lower than Epoch's rough extrapolation or models are learning to match answers the key gets wrong. The sources we read do not say which, so this is our inference, not a finding.

Is GPQA an IQ test?

No. GPQA tests specialised science knowledge and reasoning on four-option questions, scored as the percentage correct, with a 25% floor. The human comparisons are accuracy figures for experts in their own field, not scores on a population norm, so nothing here lets you place a model on the IQ bell curve. An IQ test samples reasoning on items that are new to the person and anchors the result to a norm group; for that distinction applied to AI, see why an AI score on an IQ-style test is not an IQ and the ARC-AGI-3 results.

GPQA also shows why newer exams exist. Humanity's Last Exam was built after benchmarks such as MMLU and GPQA were reached or passed by frontier models, with questions chosen because models had failed them; its top scores and a dispute over its answer key are covered in our guide to the Humanity's Last Exam leaderboards. For the broader questions of whether AI has a general factor and whether it is smarter than people, see is there a g factor for AI models, ChatGPT's IQ score and whether AI is smarter than humans.

What should a reader take from a GPQA score?

• Check which set the number is for: extended (546), main (448) or Diamond (198)
• Do not quote 65% or 34% as Diamond scores; Diamond has its own selection-skewed human figures
• Treat gaps of a point or two near the top as a few questions, not a ranking
• Ask about the setup: reasoning level, number of runs, grader and answer format
• Remember it is a science-knowledge test with a 25% guessing floor, not a measure of intelligence

Our IQ Metrics test is built for people, and you can place any score you receive with the IQ percentile calculator. For the most-cited ranking of AI models, read which AI model the index calls most intelligent.

Common questions