FrontierCS 2: A Benchmark for Computer Science R&D

FrontierCS 2 is a benchmark of 48 open-ended computer science R&D tasks built from published papers by deleting each paper's contribution from its released code, which becomes the human reference. Agents work inside a research environment with an inherited repository and may submit patches. A judge scores patches on a development suite, while a private held-out suite grades final submissions against the reference. Tasks span 13 domains. Twelve agents were evaluated; the best, Claude Opus 5.5, passed 58% of its runs, and no run passed 16 tasks.

Key takeaways

• The best evaluated agent, Claude Opus 5.5, passed 58% of its runs on the 48 tasks.
• No agent run passed on 16 of the tasks.
• Most runs that beat the reference on combined final score still fell short on some metric or workload.
• The median task is roughly as difficult as what the strongest agent can handle, per a shared capability scale.

Why it matters

The benchmark tests agents in a realistic research setting with hidden final evaluation, rather than grading one program against a problem statement. It suggests current agents remain limited on frontier computer science R&D work.

Read with care

Source claims not independently checked; the private set of 100+ tasks is not yet released, so results cover only the 48 public tasks.