As large language models (LLMs) rapidly advance, benchmarking their capabilities has become essential for assessing progress and guiding future research. A diverse array of benchmarks has emerged, each designed to evaluate specific facets of language understanding and generation, spanning domains such as commonsense reasoning, mathematical problem-solving, code generation, and question-answering.
By 2026, a useful evaluation stack combines MMLU-Pro, GPQA Diamond, and Humanity’s Last Exam for knowledge and reasoning; LiveCodeBench and SWE-bench for coding; MMMU for multimodal understanding; SimpleQA for factuality; BrowseComp for web research; and task-specific agent evaluations. No single score is sufficient: check contamination, grading quality, latency, cost, tool access, and human performance together. Robustness sits outside this stack and is measured by adversarial and security evaluations — red teaming, prompt injection tests, and safety classifiers.
We analyzed the most popular open- and closed-source LLMs to devise a comprehensive list of the most widely used benchmarks for evaluating state-of-the-art LLMs.
For current human-preference rankings, use the live Arena text leaderboard. Arena is based on blind pairwise comparisons, and its rankings change as models, votes, categories, and ranking methods change. A frozen model list quickly becomes misleading.
August 2026 model snapshot: officially released frontier families include OpenAI’s GPT‑5.6; Anthropic’s Claude Fable 5 and limited-access Mythos 5 plus Claude Opus 5; and Google’s Gemini 3.1 Pro and Gemini 3.5 Flash. These are examples of current official releases, not a claim about their exact Arena order. On the open-weight side, we compared Kimi K2, DeepSeek-R1, Qwen3, and GLM-4.5 on the same capability axes.
Now, to the main list of the benchmarks! →
Outdated vs. Current LLM Benchmarks
Older signal | Stronger current complement | Why |
|---|---|---|
MMLU alone | MMLU-Pro, GPQA Diamond, Humanity’s Last Exam | Harder questions, more reasoning, and less saturation |
HumanEval alone | LiveCodeBench and SWE-bench | Fresher coding tasks and real repository work |
Static short-answer QA | SimpleQA and BrowseComp | Separates factuality from web research and tool use |
One multimodal score | MMMU plus task-specific visual evaluations | Different image, chart, document, and spatial skills fail differently |
One leaderboard rank | A capability portfolio plus cost, latency, and human review | Preference and benchmark scores do not prove production reliability |
Classic benchmarks in the rest of this guide still matter for historical comparison and diagnostics. The change in 2026 is not to discard them, but to pair them with fresher, harder, auditable evaluations and with tests built from your own production tasks.
Commonsense Reasoning Benchmarks for LLMs
1. HellaSwag
Objective: Test commonsense reasoning by selecting a plausible continuation of an everyday scenario.
Format: Four-way multiple choice: one correct ending and three adversarially generated distractors.
2026 note: HellaSwag is now heavily saturated and vulnerable to contamination. Use it for historical comparison or regression testing, not as a standalone frontier benchmark.
Original paper: HellaSwag: Can a Machine Really Finish Your Sentence?
2. WinoGrande
Objective: Test commonsense coreference reasoning while reducing exploitable annotation artifacts.
Format: A sentence contains a blank and two noun-phrase candidates; the model chooses the candidate that makes the sentence coherent.
Scale: The original dataset contains about 44,000 problems and uses adversarial filtering.
2026 note: WinoGrande is a saturated classic. Treat it as a diagnostic or historical measure and pair it with newer reasoning evaluations.
Original paper: WinoGrande: An Adversarial Winograd Schema Challenge at Scale
3. PIQA (Physical Interaction Question Answering)
Objective: Assess physical commonsense about how to achieve everyday goals.
Format: Each item presents a goal and two possible solutions; the model selects the more physically plausible one.
2026 note: PIQA is useful for regression testing but is too mature and narrow to establish frontier reasoning by itself.
Original paper: PIQA: Reasoning about Physical Commonsense in Natural Language
Objective: Evaluate reasoning about people’s intentions, reactions, and social consequences.
Format: A short social situation is followed by a question and three answer choices.
2026 note: SocialIQA remains a useful targeted diagnostic, but its static multiple-choice format is not a strong standalone frontier signal.
Original paper: SocialIQA: Commonsense Reasoning about Social Interactions
5. OpenBookQA
Objective: Test elementary science question answering that combines a small “open book” of core facts with broader commonsense knowledge.
Format: Four-way multiple-choice science questions.
2026 note: OpenBookQA is a small, mature benchmark. Use it for diagnostics and reproducibility rather than frontier ranking.
6. ARC (AI2 Reasoning Challenge)
Objective: Evaluate grade-school science knowledge and reasoning.
Format: Multiple-choice questions divided into ARC-Easy and the more difficult ARC-Challenge set.
2026 note: ARC is widely used and increasingly saturated. Report it for continuity, but pair it with fresher, harder, and auditable tasks.
Original paper: Think You Have Solved Question Answering? Try ARC
7. CommonsenseQA
Objective: Test commonsense reasoning over concepts and relations drawn from ConceptNet.
Format: Five-way multiple-choice questions designed to require more than simple fact lookup.
2026 note: CommonsenseQA is now a mature diagnostic with limited power to separate frontier models.
LLM Reasoning Benchmarks
8. MMLU (Measuring Massive Multitask Language Understanding)
Objective: Measure broad academic and professional knowledge across 57 subjects.
Format: Four-way multiple-choice questions, commonly evaluated in zero-shot or few-shot settings.
2026 note: Original MMLU is saturated and exposed to contamination. Use it for continuity, but prefer MMLU-Pro, GPQA Diamond, Humanity’s Last Exam, and private domain evaluations for frontier comparisons.
Original paper: Measuring Massive Multitask Language Understanding
9. BIG-Bench Hard (BBH)
Objective: Evaluate 23 BIG-bench tasks on which the models studied in the original BIG-bench work had not surpassed the average human rater.
Format: A heterogeneous suite of algorithmic, symbolic, linguistic, and multi-step reasoning tasks; the BBH paper commonly evaluates them with few-shot chain-of-thought prompting.
2026 note: BBH is now substantially saturated. BIG-Bench Extra Hard (BBEH) is a stronger modern complement.
Math Benchmarks for LLMs
10. GSM-8K (Grade School Math 8K)
Objective: Test multi-step grade-school arithmetic reasoning on 8,500 human-written word problems.
Format: Primarily open-ended problems with a natural-language solution and a final numeric answer—not multiple choice.
Challenge: Problems generally require two to eight elementary arithmetic steps using addition, subtraction, multiplication, and division.
2026 note: GSM8K is heavily saturated. Keep it for regression and historical comparison, but use harder and fresher math sets for frontier evaluation.
Original paper: Training Verifiers to Solve Math Word Problems
11. MATH
Objective: Evaluate competition-style mathematical problem solving.
Format: Free-response problems with worked solutions, organized into five difficulty levels and seven subjects: prealgebra, algebra, number theory, counting and probability, geometry, intermediate algebra, and precalculus.
2026 note: Original MATH is widely trained on and increasingly saturated. Pair it with recent competition problems, private sets, and proof-quality evaluation.
Original paper: Measuring Mathematical Problem Solving With the MATH Dataset
12. MGSM (Multilingual Math Reasoning)
Objective: Measure multilingual mathematical reasoning rather than English-only performance.
Format: A 250-problem subset of GSM8K translated into ten typologically diverse languages, with open-ended numerical answers.
2026 note: MGSM is small and derived from a saturated source benchmark. Report results by language and complement it with broader native-language evaluations.
Original paper: Language Models are Multilingual Chain-of-Thought Reasoners
13. DROP (Discrete Reasoning Over the content of Paragraphs)
Objective: Test reading comprehension that requires discrete reasoning over passages, including counting, addition, sorting, and date comparison.
Format: A passage and question with answers represented as text spans, numbers, or sets of spans.
2026 note: DROP remains a useful targeted diagnostic, but its static corpus should be complemented with fresher and task-specific reasoning evaluations.
LLM Coding Benchmarks
14. HumanEval (or HumanEval-Python)
Objective: Measure functional correctness in Python code generation.
Format: 164 hand-written function-level problems provide a signature and docstring; generated solutions are executed against unit tests and commonly scored with pass@k. The complete evaluation tests are not part of the model prompt.
2026 note: HumanEval is small, contaminated, and saturated. Use it for continuity, but rely on LiveCodeBench, SWE-bench, repository-level tasks, and private tests for current capability. Benchmark scores also diverge from practical use: we tested 15 coding agents head-to-head and scored them on code, testing, tooling, docs, and polish.
Original paper: Evaluating Large Language Models Trained on Code
15. MBPP (Mostly Basic Python Programming)
Objective: Evaluate entry-level Python program synthesis.
Format: MBPP contains 974 crowd-sourced problems, each with a short description, a reference solution, and three automated test cases.
Scoring: Standard MBPP measures functional correctness against tests; it is not primarily an efficiency benchmark.
2026 note: MBPP is saturated and has limited test coverage. Use MBPP+ or stronger current coding evaluations as complements.
Original paper: Program Synthesis with Large Language Models
LLM Question Answering Benchmarks
16. Natural Questions
Objective: Evaluate question answering on real, anonymized queries issued to Google Search.
Format: An annotator receives a query and a Wikipedia page from the top search results, then marks a long answer, a short answer, or no answer.
2026 note: Natural Questions remains valuable for reading comprehension and retrieval research, but its static Wikipedia corpus does not by itself test current web research or agentic browsing.
Original paper: Natural Questions: a Benchmark for Question Answering Research
17. TriviaQA
Objective: Evaluate reading comprehension and retrieval using naturally authored trivia questions and independently collected evidence.
Format: 95,956 question-answer pairs with Wikipedia and web evidence documents—about six documents per question on average.
2026 note: TriviaQA is a classic static corpus with contamination risk; complement it with fresh factuality and retrieval evaluations.
18. MMMU (A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark)
Objective: Test expert-level vision-language understanding and reasoning across academic disciplines.
Format: 11,500 interleaved image-and-text questions from college exams, quizzes, and textbooks across 30 subjects and 183 subfields. Original MMMU does not test audio understanding.
Challenge: Questions use heterogeneous visual material such as charts, diagrams, maps, tables, music notation, and chemical structures.
2026 note: Pair MMMU with MMMU-Pro and task-specific visual evaluations to reduce text-only shortcuts and saturation effects.
19. TruthfulQA
Objective: Test whether a model resists common human misconceptions and avoids imitative falsehoods.
Format: 817 questions across 38 categories, with open-ended generation and multiple-choice evaluation variants.
Challenge: Good performance requires answers that are both truthful and informative; the benchmark is about factual truthfulness, not general ethical behavior.
2026 note: TruthfulQA is a small, static benchmark. Pair it with fresher factuality sets such as SimpleQA and domain-specific verification. Hallucination is a distinct failure mode with its own evaluation sets — we cover 13 hallucination benchmarks separately.
Original paper: TruthfulQA: Measuring How Models Mimic Human Falsehoods
The 2023 Survey on Evaluation of Large Language Models remains useful historical background, but it is not a current catalog. For maintained evaluation implementations, use the LM Evaluation Harness; for current discovery, use live leaderboard and evaluation directories.
Useful current leaderboard and evaluation resources include:
Arena text leaderboard: dynamic human-preference rankings derived from blind pairwise comparisons. Results vary by category and should not be treated as a universal capability score.
Hugging Face Leaderboards and Evaluations: a directory of benchmark datasets, reproducible evaluation results, and community leaderboards.
Stanford HELM: standardized, transparent evaluation across scenarios, metrics, and models.
FAQ
What is HellaSwag?
HellaSwag is a four-way multiple-choice benchmark for commonsense reasoning. A model selects the most plausible continuation of a short real-world scenario from adversarially constructed options. By 2026 it is heavily saturated and is best used for historical comparison or regression testing rather than frontier ranking.
What is the MMLU benchmark?
MMLU measures knowledge and problem solving across 57 academic and professional subjects using four-way multiple-choice questions. Original MMLU is now saturated and exposed to contamination. MMLU-Pro is a harder successor with more reasoning-focused questions and ten answer choices.
What are LLM benchmarks?
LLM benchmarks are standardized tasks, datasets, scoring rules, and evaluation harnesses used to compare model capabilities such as knowledge, reasoning, coding, factuality, multimodal understanding, tool use, safety, latency, and cost. A reliable evaluation program uses several benchmarks plus task-specific and human tests. The safety axis is often scored by guardian models such as Llama Guard and ShieldGemma acting as automated judges.
What is the difference between MMLU and HellaSwag?
MMLU tests broad academic and professional knowledge across many subjects. HellaSwag tests whether a model can choose a plausible continuation for an everyday scenario. MMLU is closer to an exam; HellaSwag emphasizes commonsense prediction and resistance to misleading answer options.
What are the limitations of LLM benchmarks?
Public benchmarks can become contaminated, saturated, overfit, mislabeled, or disconnected from real work. Scores also depend on prompts, tool access, sampling, grading, and compute. They rarely capture reliability, cost, latency, security, and user value at the same time. Those signals come from production monitoring instead — we cover 13 open-source LLM observability tools separately.
We post helpful lists and bite-sized explanations daily on our X (Twitter). Let’s connect!
