IQ Metrics
IIF Certified Assessment Start IQ Test
IIF Certified Assessment
AI & Machine IntelligenceEmerging evidence

What Is the MMLU Benchmark? The AI Exam Everyone Quotes, and Its Errors

MMLU is a 15,908-question multiple-choice exam in 57 subjects that AI models take. A 2024 audit estimated that 6.49% of its questions contain errors, and 57% in virology. Here is what a score means and what replaced it.

What Is the MMLU Benchmark? The AI Exam Everyone Quotes, and Its Errors
Illustration generated for IQ Metrics. No photograph is used. IQ Metrics

What you need to know

  • The MMLU benchmark is a multiple-choice exam for AI models: 15,908 questions in 57 subjects (Hendrycks et al., ICLR 2021). A score is the share answered correctly, and 25% is a random guess.
  • An audit by Gema and colleagues estimated that 6.49% of MMLU questions contain errors, or about 1 in 15, and 57% of the sampled virology questions did (arXiv 2406.04127, January 2025). It built MMLU-Redux from 5,700 re-checked questions.
  • In the original paper GPT-3 scored 43.9%, non-specialist humans 34.5%, and the authors estimated expert-level accuracy at about 89.8%.
  • MMLU-Pro (ten options, accuracy down 16% to 33% against MMLU) and harder exams replaced it, and an MMLU score is not an IQ.

The MMLU benchmark is a multiple-choice exam for AI models: 15,908 questions in 57 subjects, from elementary mathematics to law. A model's MMLU score is the share it answers correctly, so 25% is a random guess. A 2024 audit estimated that 6.49% of the questions contain errors, and 57% of the virology questions it sampled did.

Below: what MMLU is, what a score means, how many answers are wrong, what has replaced it and whether it has anything to do with IQ. Each figure was re-read at its source on 13 November 2026.

What is MMLU?

MMLU stands for Massive Multitask Language Understanding. Dan Hendrycks and colleagues proposed it in 2020 as a test of a text model's multitask accuracy, and it was published at ICLR 2021. It covers 57 tasks, including elementary mathematics, US history, computer science and law. So, what is MMLU in practice? It is a very long school-and-professional exam, taken by a model instead of a student.

The authors collected 15,908 questions. They split them into a small few-shot development set, a validation set of 1,540 and a test set of 14,079, with at least 100 test questions in every subject. Some questions come from exams, such as the US Medical Licensing Examination questions behind the Professional Medicine task. Others come from textbooks, such as the human virology textbook behind the Virology task, according to the later audit.

What kinds of questions are in the MMLU benchmark?

They are four-option multiple-choice questions at four levels: elementary, high school, college and professional. Graduate and undergraduate students collected them from sources available online, according to the paper. These include practice questions for the Graduate Record Examination and the US Medical Licensing Examination, questions for undergraduate courses, and questions written for readers of Oxford University Press books.

An elementary mathematics item, for example, asks how to find the number of teams when 30 players are split into teams of 5, and the right option is to divide 30 by 5 to get 6. A professional law or medicine item is far harder. The same score averages both kinds, so one number hides a wide spread.

MMLU is meant to measure what a model learned during pretraining, so models take it without fine-tuning on it, in zero-shot or few-shot settings. Zero-shot means no worked examples in the prompt, and few-shot means a handful. The test is text only.

What does an MMLU score mean?

An MMLU score is the percentage of questions a model answers correctly, and 25% is the chance level. The original paper reports that few-shot models up to 13 billion parameters scored near 25%, and the 175-billion-parameter GPT-3 reached 43.9%. Non-specialist humans on Amazon Mechanical Turk scored 34.5%.

The authors estimated expert-level accuracy at about 89.8%. They took the accuracy of 95th-percentile test-takers on the source exams, and made an educated guess where no figure existed. Treat 89.8% as an estimate, not a measured human record.

By 2024 the test was crowded at the top. Wang and colleagues wrote that performance on benchmarks like MMLU had begun to plateau, which makes models hard to tell apart. On the harder MMLU-Pro, Artificial Analysis lists 89.8%, 89.5% and 89.5% as its top three scores, and files the test under legacy evaluations. We did not read a current MMLU leaderboard, so we print no current MMLU score. Our guide to the Artificial Analysis Intelligence Index notes that MMLU-Pro left that index in January 2026.

How many MMLU answers are wrong?

About 1 in 15, by one audit. Gema and colleagues estimated that 6.49% of MMLU questions contain errors (arXiv, January 2025). They asked 14 human experts to re-annotate 5,700 questions, 100 sampled from each of the 57 subjects, and called the result MMLU-Redux. The 6.49% comes from weighting that sample back up to the whole set.

  • Whole benchmark: an estimated 6.49% of questions contain errors
  • Virology: 57% of the sampled questions had errors, including wrong answer keys (33%), more than one correct answer (4%) and unclear questions (14%), per the paper's Figure 3
  • Logical Fallacies and College Chemistry: more than 20% of sampled examples were wrong
  • Errors range from scraping mistakes to questions that assume US law or US surveys without saying so

One virology item offered sending the American army to West Africa to stop Ebola outbreaks as an answer. In College Chemistry, some questions that ran over several lines were parsed so that part of the question became option A and option D vanished. These are the kinds of errors the authors found.

Why does that matter for an MMLU score? When the authors re-scored leading models on MMLU-Redux, the metrics changed and the ranking changed. MMLU errors blur small gaps between models. The paper is an audit by one team, so treat 6.49% as an estimate.

About 1 in 15 MMLU questions contains an error, and in virology it was 57% of those sampled.

How should you read an MMLU score in a headline?

Check three things: which test, which setting and how big the gap is. MMLU, MMLU-Pro and MMLU-Redux are different tests, and zero-shot and few-shot scores are not the same. Then look at the size of the gap. One percentage point of the 14,079-question test set is about 141 questions, which is well inside the error rate the audit estimated.

So a lead of one or two points between two models is a weak signal on its own. That is a judgement from the numbers above, not a finding of the audit.

What replaced MMLU?

MMLU-Pro replaced it in many reports, and harder exams now sit above it. Wang and colleagues built MMLU-Pro with ten answer options instead of four and removed trivial and noisy questions. Accuracy fell by 16% to 33% compared with MMLU, and scores were more stable under different prompts, according to their abstract. Artificial Analysis describes it as 12,000 questions in 14 subject areas.

For frontier models the field moved to harder tests. Read how GPQA Diamond works and what the Humanity's Last Exam leaderboards show. Our guide to why AI benchmark scores disagree explains why one model can lead on one list and trail on another. For the wider question of how AI is measured, see how smart is AI. Searches for MMLU Redux point to the audit set above.

Is an MMLU score an IQ?

No. An IQ is a score compared with a norm group of people of the same age, on a scale with a mean of 100 and a standard deviation of 15. An MMLU score is accuracy against an answer key. It has no age norms and no standard deviation, and the only human reference in the paper is crowd workers at 34.5% and an estimated expert level.

That is why no one can convert an MMLU score into an IQ. Our page on whether AI is smarter than humans covers the claim, and our IQ percentile calculator shows how a real IQ score maps to a percentile of people.

Method, 13 November 2026: we re-opened the abstract pages of arXiv 2009.03300, 2406.04127 and 2406.01574, the full text of the first two, and the Artificial Analysis MMLU-Pro page. The Redux paper's running text gives 30% and 15% for two virology error types where its Figure 3 caption gives 33% and 14%, so we print the caption and no sub-figures beyond it. We did not read a current MMLU leaderboard, so no current score is printed. Counts of 1 in 15 are our arithmetic from 6.49%.
Your own number

Where would your own score land?

MMLU compares a model with an answer key, not with people. For a score on the standard IQ scale with its percentile, take the IQ Metrics test.

Find your IQ score now! →
Secure & encryptedInstant results10–20 minutes

To see where your own reasoning sits against other adults, take the IQ Metrics test.

Common questions

What is MMLU?

MMLU, Massive Multitask Language Understanding, is a multiple-choice test of AI models with 15,908 questions across 57 subjects, proposed by Hendrycks and colleagues in 2020 and published at ICLR 2021. A model's score is the percentage it answers correctly.

What is a good MMLU score?

The chance level is 25%. In the original paper GPT-3 scored 43.9%, non-specialist humans 34.5% and the authors estimated expert-level accuracy at about 89.8%. Because about 1 in 15 questions has an error, small gaps between models should be read with care.

How many MMLU questions are wrong?

An audit estimated that 6.49% contain errors, from 5,700 questions re-checked by 14 human experts. In virology 57% of the sampled questions had errors. The figure is one team's estimate.

What is MMLU-Redux?

MMLU-Redux is a subset of 5,700 MMLU questions, 100 per subject, that 14 experts re-annotated to find errors. Re-scoring leading models on it changed their metrics and their ranking.

Sources for this story

  1. Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, Steinhardt J. Measuring Massive Multitask Language Understanding. ICLR 2021; arXiv 2009.03300 v3, 12 January 2021; full text read — arXiv / ICLR
  2. Gema AP, Leang JOJ, Hong G, et al. Are We Done with MMLU? arXiv 2406.04127 v3, 10 January 2025; abstract and full text read — arXiv
  3. Wang Y, Ma X, Zhang G, et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv 2406.01574 v6, 6 November 2024; abstract read — arXiv
  4. MMLU-Pro benchmark page, read 13 November 2026 (12,000 questions, ten options, listed under Legacy) — Artificial Analysis
  5. IQ Metrics guide to the Artificial Analysis Intelligence Index (MMLU-Pro removed from version 4.0, January 2026) — IQ Metrics
  6. IQ Metrics computation, 13 November 2026: 1 in 15 from 6.49% — IQ Metrics

Corrections: spotted an error? Email corrections@iqmetrics.org and we will update this story and note the change here.

Share this story

Know someone who keeps seeing this number quoted without the scale it was measured on? Send it to them — it takes one tap.

Filed under#ai benchmarks#study quality#standardized exams#scores and scales

Read the research.
Then find your own number.

Our IIF-certified assessment reports your score with its scale, percentile and confidence range — and a breakdown of the cognitive domains behind it.

Start IQ Test →
Secure & encryptedInstant results10–20 minutes