Artificial Analysis launches CritPt v2, a revised 70-challenge physics benchmark

Artificial Analysis describes CritPt v2, a revised version of CritPt, a benchmark of 70 composite research-level physics challenges for testing LLMs. Models solve problems with a code-execution tool in an offline Linux sandbox. The benchmark authors reviewed and corrected underspecified statements, errors, and typos in the dataset. Artificial Analysis states it provides an evaluation API for independent grading, granted case by case. The page lists scores for 11 models, with GPT-6 Astra (Max) highest at 83.1%.

Key takeaways

• CritPt v2 contains 70 composite research-level physics challenges, revised from the original CritPt dataset.
• Models solve problems using a code-execution tool inside an offline Linux sandbox.
• Authors and physics collaborators corrected underspecified statements, errors, and typos in the revised dataset.
• GPT-6 Astra (Max) scores highest at 83.1%, followed by GPT-6.1 Sol (Max) at 82.0%.

Why it matters

A physics benchmark built from unpublished research-level problems, with a code sandbox, offers a harder and more research-oriented test of LLM reasoning. Independent grading by a third party adds a layer of evaluation transparency.

Read with care

Source claims not independently checked; the listed scores come from the Artificial Analysis page and have not been verified against the underlying model outputs.