AI Models Found to Significantly Outperform Physics Benchmarks

A new study has found that leading AI models like GPT-5.6-Sol and Claude Fable 5 are significantly better at solving physics problems than standard tests show. The researchers discovered that the benchmarks used to measure AI performance are often "broken," incorrectly marking the models' correct answers as wrong. After manually re-grading the tests, the team concluded that the models have effectively "saturated" these benchmarks, meaning the tests are no longer difficult enough to measure their true capabilities.

According to a paper posted to the arXiv preprint server on September 14, 2026, a team of researchers led by Ali Ansari evaluated several frontier AI models, including Google’s Gemini 3.1 Pro, across six different physics benchmarks. Their central finding was that the automated evaluation systems for these tests are flawed. When human physics experts reviewed the models' answers, they found that performance was far higher than initially reported. The models were not just getting the right final number but were often providing correct reasoning that the automated graders failed to recognize. This expert re-evaluation revealed that the models' scores were so high they approached the maximum possible, a state the researchers call saturation. This suggests the benchmarks are no longer a reliable yardstick for differentiating the capabilities of top-tier AI.

The Gap Between Problems and Puzzles

Despite demonstrating near-perfect performance on established physics problems, the same AI systems struggle when confronted with unsolved questions in the field. The paper details how a specialized, agentic AI system built on GPT technology was applied to open problems in theoretical physics. This same type of system had previously shown success in helping to resolve open conjectures in mathematics. When applied to physics, however, the researchers reported that it "has not managed to fully resolve even one autonomously."

The study also noted that the AI agent made "considerably less progress" on partially solved physics problems compared to its performance on similarly difficult math problems. This finding highlights a critical distinction: while the models have become extraordinarily proficient at solving problems for which a solution is known to exist, they show limited ability to perform the novel reasoning required for genuine scientific discovery. They can solve textbook exercises but cannot yet act as research physicists.

The Training Data Debate

The paper's publication prompted a debate about why the models perform so well on these benchmarks. In a discussion on the social news website Hacker News, some users speculated that the models' high scores are a result of "data contamination," meaning the benchmarks themselves were included in the models' vast training data. One user, writing under the name "fsh," suggested that AI companies likely hire people specifically to ensure their models are trained on all publicly available benchmarks.

However, other participants in the discussion pointed to the paper's own findings as evidence against simple memorization. User "letmevoteplease" argued that if the models had merely memorized the benchmark solutions, they would have produced the exact answers the flawed automated graders were expecting. The fact that the models generated correct, alternative solutions that were initially marked as wrong suggests they are reasoning through the problems rather than recalling pre-packaged answers.

This debate was further complicated by a claim from a user named "red75prime," who stated that "Half of the benchmarks don't have published answers." According to this user, AI companies hire physicists to solve these problems specifically to create training data. Another commenter, "redwood," characterized this alleged practice as "unconventional benchmaxxing," a strategy where models are trained on correct solutions that differ from the official answer keys, thereby improving their general problem-solving ability at the expense of their official benchmark scores. These claims about data creation practices have not been independently verified.

From Benchmarks to Reality

While the models excel in the structured environment of a test, anecdotal evidence suggests they still falter when trying to understand physical situations described in natural language. One anonymous Hacker News commenter with self-professed expertise shared that frontier models "absolutely struggle with understanding a physical situation based on words." The user cited examples where GPT-5.6 Sol made "outrageous errors" on relatively simple problems, such as those involving NPT pipe threads or a complex arrangement of spheres. This points to a potential gap between manipulating formal equations and possessing a robust, intuitive grasp of physical concepts.

This skepticism reflects a broader uncertainty about the nature of AI intelligence. Another commenter drew an analogy to the historical phlogiston theory, a discredited scientific concept that persisted for a time. The user suggested that established experts may be resistant to new findings that challenge the foundations of their field, though this remains an interpretation of the community's reaction rather than a statement on the models' abilities. The core of the issue remains unresolved: the models have surpassed the tests designed to measure them, yet their capacity for genuine physical understanding and discovery is still very much in question.