How AI agents predict the effects of code changes

An Improvement Dynamics Model (IDM) predicts how a proposed change to an AI agent’s program may affect its performance on individual tasks, and can prompt one additional revision before the changed agent is evaluated. In a preprint, researchers tested this approach across four benchmark domains; the coding results favored the IDM-guided method on reported performance measures, while its full-run coding experiments used more tokens than HyperAgents. The preprint describes a way to change an agent’s code and instructions—not to retrain its underlying foundation model.

What the predictor evaluates

HyperAgents uses a task agent to solve problems and a meta agent that can modify the task agent’s program and its own. The IDM is an editable program that invokes a frozen large language model (LLM) to forecast task-level outcomes for a proposed change. Its inputs include the system’s archive, a selected parent agent, that agent’s observed outcomes, and the proposed code diff—the precise changes to the program.

Rather than predicting only one overall score, the IDM estimates outcomes for individual tasks. That distinction can help direct a revision: different failures may call for different fixes, even when they produce similar aggregate scores. The forecast guides the system’s next step; it does not determine whether the changed agent will succeed.

A forecast can prompt one more revision

The aggregate forecast can trigger at most one extra revision before evaluation. Task-level predictions can guide that revision, and the IDM forecasts the revised code diff again. The resulting child agent is evaluated even if that last forecast looks unfavorable.

The predictor itself can also change. In the reported experiments, the meta agent proposed IDM updates every five generations. A candidate was tested against the next five records while the existing IDM remained in use. The system adopted the candidate only if it lowered task-prediction loss without increasing centered mean squared error, another measure of prediction error. These rules put evaluation after the proposed change, rather than treating a forecast as a guarantee that a change is safe or successful.

Four benchmark settings, with different measures

The preprint evaluated Polyglot coding, paper review, simulated robotics reward design, and Olympiad-level mathematical grading. Each method had five complete runs of 40 generations. In coding, a task counted as resolved when a patch passed its test suite. Paper review involved predicting acceptance or rejection; the robotics work generated reward functions for quadruped velocity tracking in the Genesis simulator; and math grading assigned scores from 0, 1, 6, or 7.

Two coding results capture different things. Median area under the budget curve (AUBC) tracks performance across the 40-generation budget. The held-out resolve-rate result instead measures how selected agents performed on 166 separate coding tasks, relative to the common initial agent. Its median gain was 13.25 percentage points for Selective IDM and 5.42 points for HyperAgents; the initial agent’s resolve rate was 16.27%.

For paper review and robotics, Selective IDM’s median AUBC was 5.9% higher on paper review and 3.9% higher for the equally weighted joint objective than HyperAgents. A predictor evolved on those two tasks was also transferred to mathematical grading: its median AUBC was 3.4% higher than Selective IDM under a generation budget and 2.3% higher under a matched-token budget. On held-out math tasks, its median accuracy was 4 percentage points above Selective IDM and 8 points above HyperAgents.

The coding gains came with higher token use

In the full-run Polyglot coding comparison, Selective IDM had a higher median AUBC than HyperAgents. Its mean token use was also higher:

At a matched coding budget of 44.21 million tokens, the preprint reports a 10.9% higher median AUBC for Selective IDM. That comparison indicates better reported performance at the specified token budget; the full-run totals show that the methods did not use the same number of tokens in their ordinary runs. The paper’s token accounting includes agent evolution, task evaluation, forecasting, IDM updates, and validation, but token totals alone do not measure overall compute, energy use, or monetary cost.

Across the four domains, the final deployed IDM had 8.7%–24.4% lower task-prediction loss than the initial IDM. In the coding experiments, candidate IDMs had lower Brier loss in 19 of 28 comparisons, including candidates that were adopted and rejected. The results describe the preprint’s benchmark experiments, including simulated robotics; they do not establish how the method would perform on deployed software projects.