The TheoPhysEval authors report that GPT-6-Astra running with Codex produced correct final answers for 49 of 50 tasks in the realistic track and 46 of 50 in the isolated track. The 50 tasks tested bounded calculations with verifiable answers, using established techniques; the authors excluded major open research problems. Each agent completed the suite once per track.
What TheoPhysEval tested
The benchmark covered 50 unpublished problems across 11 areas of theoretical physics. It assessed agents’ final answers to research-level calculations, rather than their ability to choose research questions or develop new physical theories. Many symbolic problems were converted to numerical answers for grading, and the authors identified two prompts with underspecified conventions.
The authors evaluated five model-and-tool pairings in two tracks. The realistic track allowed web search and local tools. The isolated track disabled web search but retained local tools and calls to the model provider.
| Agent pairing | Realistic track, correct out of 50 | Isolated track, correct out of 50 |
| GPT-6-Astra with Codex | 49/50 | 46/50 |
| Opus 5.5 with Claude Code CLI | 46/50 | 46/50 |
| Gemini 3.8 Flash with Antigravity | 34/50 | 31/50 |
| DeepSeek-V4.1-Flash with Codex | 27/50 | 25/50 |
| GLM 5.3 with Claude Code CLI | 21/50 | 26/50 |
What the scores can tell you
GPT-6-Astra with Codex answered three more tasks correctly in the realistic track than in the isolated track. Opus 5.5 with Claude Code CLI scored 46/50 in both. The authors ran each agent once per track and say the differences between tracks fall within the statistical uncertainty of that single-run design, so the score gap does not establish how much web access changed performance.
The benchmark authors also say expert consultation uncovered subtle errors in several reference solutions that had initially counted against Astra. After correcting those answers, they judged Astra’s answers correct in those cases. One inverse problem, which asked agents to find a differential equation reproducing a supplied curve, was not solved by any tested model.
The problems had known, checkable answers and were unpublished when the evaluation took place. Some contributors used AI tools while working on problems, so the authors could not rule out data leakage. Those conditions make 49/50 a result on this particular suite, not a general measure of autonomous scientific research.
What the reported cost covers
The authors report a cost of $116.28 for GPT-6-Astra with Codex’s 50-task realistic run. Dividing that total by 50 gives an average of $2.3256 per task, or about $2.33.
