humans last exam llm : A Comprehensive Evaluation

Top AI models that ace traditional benchmarks are stumbling badly on a new test called "Humanity's Last Exam," scoring a mere 25%. This dramatic performance gap reveals just how far we still are from achieving expert-level artificial intelligence, despite the impressive capabilities of today's large language models.

Humanity's Last Exam  represents a new frontier in AI evaluation, an ultra-hard benchmark specifically designed to identify when machines can truly match human expertise across diverse fields. Created through a collaboration between the Center for AI Safety  and Scale AI, this benchmark emerged from a critical need: earlier tests had become too easy, failing to distinguish between increasingly powerful models. We'll explore how HLE works, examine current model performance, understand the critiques it has faced, and unpack why this benchmark matters for the future of AI development.

What HLE Is and How It Was Built

The origins of Humanity's Last Exam trace back to conversations among AI leaders who recognized a pressing problem. By 2024, leading models were achieving over 90% accuracy on established benchmarks like MMLU, rendering these tests essentially useless for measuring further progress. Elon Musk was among those who noted this saturation effect, spurring Dan Hendrycks (CAIS) and Alexandr Wang (Scale AI) to spearhead the creation of a genuinely challenging alternative.

Announced in September 2024 and finalized in spring 2025, HLE went through an unusually rigorous development process for an AI benchmark. The creators issued a global call for the toughest questions imaginable, offering a $500,000 prize pool and co-authorship opportunities to attract highly qualified contributors. The response was overwhelming: nearly 1,000 domain experts from more than 500 institutions across 50+ countries contributed to the project.

The final dataset comprises 2,500 carefully vetted questions covering more than 100 subjects. The distribution reflects a strong emphasis on technical fields, with approximately 41% focusing on mathematics and 11% on biology. The remaining questions span physics, chemistry, computer science, humanities, social sciences, and engineering.

Format diversity adds another layer of complexity. About 14% of questions are multimodal, requiring interpretation of diagrams, scientific figures, or historical inscriptions. Roughly 24% use multiple-choice format, while the remainder demand precise short-answer responses. Every question is designed to have one objectively correct answer, eliminating ambiguity in scoring.

The vetting process was exceptionally stringent. From over 70,000 initial submissions, only questions that stumped state-of-the-art LLMs at chance levels advanced to expert review. Approximately 13,000 questions passed this initial filter, undergoing two rounds of PhD-level expert evaluation. Reviewers verified domain alignment, difficulty level, and answer precision, though they weren't required to exhaustively check reasoning if it took more than five minutes.

A community bug bounty in early 2025 provided additional quality control, identifying and removing problematic questions to arrive at the final 2,500-question set. The creators also maintain a private held-out test set to detect potential overfitting and ensure long-term benchmark integrity.

How HLE Scores Models

The scoring methodology for HLE prioritizes both accuracy and confidence calibration. Models receive points only for exact-match accuracy on each question, whether multiple-choice or short-answer. This strict criterion ensures that partial credit or "close enough" answers don't inflate scores.

Beyond raw accuracy, HLE measures calibration, how well a model's confidence aligns with its actual performance. Each model reports a 0-100% confidence score with its answer, allowing researchers to calculate calibration error. This metric reveals whether models "know what they don't know" or exhibit dangerous overconfidence.

The leaderboard employs a sophisticated ranking system using "rank upper-bound" groups at 95% statistical confidence. This approach prevents minor performance differences from creating misleading rankings. The benchmark maintains separate leaderboards for text-only and full multimodal performance, acknowledging that vision capabilities significantly impact certain questions.

Anti-contamination measures are built into the design. Questions avoid easily searchable content, and periodic retesting with private held-out sets helps detect any data leakage into training corpora. This vigilance ensures that improvements reflect genuine capability gains rather than memorization.

How LLMs Are Doing Right Now

The reported mid-2025 snapshot of LLM performance on HLE paints a sobering picture. Even the most advanced models struggled to exceed 25% accuracy*. xAI's Grok 4 led with approximately 25.4%, followed closely by GPT-5 at 25.3%. Google's Gemini 2.5 Pro hovered in the low 20s, while many mainstream models remained between 10-15%.

These numbers become even more striking when contextualized against performance on older benchmarks. The same models routinely achieve over 90% on MMLU, have rendered the ARC science test trivial, and have largely cracked BigBench Hard challenges. HLE's difficulty represents a quantum leap beyond these saturated tests.

Calibration errors compound the accuracy problem. Models frequently express high confidence in incorrect answers, indicating systematic overconfidence on challenging questions. This pattern suggests that current LLMs struggle to recognize the limits of their knowledge when facing expert-level problems.

Tracking calibration experiments systematically with platforms like PromptLayer helps researchers identify when models improve in both accuracy and self-awareness

Tool augmentation provides some improvement but falls far short of human expert performance. FutureHouse analysis found that Grok 4's accuracy on text questions jumps from 26.9% without web access to approximately 44% with search and internet tools. While significant, this boost still leaves models failing more often than succeeding on expert-level questions.

Looking ahead, HLE's creators anticipate rapid progress. They expect models to surpass 50% accuracy by the end of 2025. These projections reflect both the current pace of AI advancement and the benchmark's design as a moving target for capability measurement.

Why HLE Matters vs Earlier Benchmarks

Humanity's Last Exam addresses fundamental limitations in previous AI evaluation methods. Traditional benchmarks have become victims of their own success, with rapid AI progress rendering them obsolete as meaningful tests. MMLU scores jumped from 10% in 2021 to over 90% by 2024. The challenging MATH competition problems saw similar saturation within just a few years.

HLE distinguishes itself through several key features. First, it targets frontier knowledge requiring graduate-level understanding across diverse fields. Questions demand the kind of specialized expertise typically found in advanced research settings.

The multimodal elements set HLE apart from primarily text-based benchmarks. Scientific diagrams, mathematical figures, and historical inscriptions require integrated visual-textual reasoning, a capability poorly tested by traditional language benchmarks.

The adversarial, unsearchable nature of questions prevents simple retrieval strategies from succeeding. Unlike benchmarks that can be "gamed" through massive training data, HLE rewards genuine reasoning and knowledge synthesis.

Perhaps most importantly, HLE remains model-agnostic. Success doesn't depend on specific architectural choices but rather on fundamental capabilities: vast knowledge stores combined with sophisticated reasoning. This design ensures the benchmark's relevance regardless of future technical developments in AI systems.

Critiques, Revisions, and What HLE Doesn't Measure

Despite its ambitious goals, HLE has faced significant criticism. FutureHouse researchers identified that 20-30% of biology and chemistry answers appeared questionable or incorrect when checked against published scientific evidence. One notable example involved identifying oganesson as Earth's rarest noble gas, a claim disputed by scientists who argue oganesson isn't considered a terrestrial gas at all.

The root of these issues traces back to the benchmark's core premise. The intense focus on stumping AI models incentivized the creation of increasingly convoluted questions, sometimes prioritizing difficulty over accuracy. The five-minute limit on rationale verification during review allowed some errors to slip through.

HLE's organizers acknowledged these problems, confirming approximately 18% error rate in a reviewed subset. They've instituted a rolling revision process, continuously removing problematic questions and adding validated replacements. This ongoing refinement reflects a commitment to benchmark quality over static perfection.

The benchmark's scope limitations are intentional but significant. HLE exclusively tests closed-ended academic questions, excluding open-ended reasoning, creativity, social common sense, and dynamic planning capabilities. Hazardous domains are deliberately omitted for safety reasons. The multimodal component covers only static images, missing real-world vision-language tasks involving video or interactive graphics.

Critics also note that HLE's dramatic framing, the name itself suggests finality, can skew public perception of AI progress. Media coverage often implies that passing this "last exam" equates to achieving artificial general intelligence, a claim the creators explicitly reject.

How to Use HLE Scores (Builders, Educators, Policymakers)

For AI laboratories and developers, HLE serves as a critical progress tracker. Sudden score improvements signal capability jumps requiring investigation. The gap between text-only and multimodal performance highlights specific areas needing development. Teams can use HLE performance to evaluate readiness for deploying AI in expert-support roles.

Governance bodies and policymakers find HLE valuable as a shared reference point for capability discussions. Specific thresholds, such as models exceeding 50% accuracy, can trigger oversight reviews or safety evaluations. The benchmark provides quantitative grounding for otherwise abstract debates about AI readiness and risk.

Educators face both challenges and opportunities from HLE's implications. The benchmark demonstrates that even advanced AI cannot answer truly challenging, off-the-books questions, preserving space for human expertise. However, it also prompts reflection on assessment methods. Should education focus on "cruel teacher" problems that stump both students and AI, or emphasize collaborative human-AI problem-solving?

In hiring contexts, HLE performance serves as a proxy for the hardest technical interviews. Organizations can calibrate their trust in AI assistants for expert roles based on benchmark scores. A model scoring 25% on HLE clearly isn't ready to replace domain experts, but one approaching 60-70% might meaningfully augment human specialists.

Key indicators to monitor include leaderboard updates, the performance gap between tool-augmented and base models, threshold crossings (especially 50-60% accuracy), and ongoing dataset revisions. These metrics collectively paint a picture of AI's journey toward expert-level performance.

How LLMs Are Doing Right Now

Humanity's Last Exam stands as a rigorous stress-test revealing the substantial gap between current AI capabilities and true expert-level performance. With top models achieving only 25% accuracy, HLE starkly illustrates how far we remain from artificial intelligence that can reliably match human expertise across diverse domains.

Yet HLE is more than just a humbling reality check. It provides the AI community with a clear, measurable target for progress while maintaining scientific rigor through continuous refinement. The benchmark's evolution, from 70,000 submissions to 2,500 carefully vetted questions, with ongoing revisions based on community feedback, demonstrates a commitment to accuracy over hype.

It's crucial to remember that passing HLE, even with high scores, represents a milestone rather than the finish line for AI development. The benchmark measures specific types of closed-ended expertise. As models improve and eventually surpass human performance on HLE, we'll need new benchmarks capturing different dimensions of capability, creativity, ethical reasoning, and real-world problem-solving.

For now, HLE serves its intended purpose admirably: providing a challenging, trustworthy measure of progress toward expert-level AI while keeping our expectations grounded in reality.

PromptLayer is an end-to-end prompt engineering workbench for versioning, logging, and evals. Engineers and subject-matter-experts team up on the platform to build and scale production ready AI agents.

Made in NYC 🗽

Sign up for free at www.promptlayer.com 🍰