Claude Rewrote the Optimizer—and Beat Overfitting

Conventional deep learning training employs static optimizers, which fix update rules and schedules before a run begins. This predetermined approach means that if unforeseen issues like overfitting or loss spikes emerge mid-training, the optimizer cannot adapt, continuing to follow its initial, potentially suboptimal, recipe.

A new paradigm, from the reactive optimizers project, addresses this limitation with OPAL (Optimizer with an Agent in the Loop). This PyTorch wrapper periodically pauses training and provides an LLM agent, such as Claude, with diagnostic telemetry. The agent then analyzes these real-time metrics to dynamically rewrite the optimizer's code.

OPAL operates within strict guardrails. The agent can modify optimizer logic, including update rules, learning-rate schedules, and momentum terms, but is expressly restricted from altering the model architecture, the training data, or evaluation metrics. This enables adaptation within a single training run, aiming to improve performance without human intervention during critical phases.

To test OPAL’s ability to prevent overfitting, researchers devised a rigorous experiment. A small image classifier was trained on 45,000 images, deliberately corrupting 4,500 of these with incorrect labels. For instance, a cat image might be labeled “truck.” Critically, during training, the agent could monitor a separate set of 5,000 clean validation images to assess true progress, but a final 10,000-image evaluation set remained untouched until training concluded.

A baseline model, using a static optimizer, memorized all 4,500 incorrect labels, confidently misclassifying them by the run’s end. In contrast, the OPAL-assisted model, with its reactive optimizer, only repeated 250 of these mislabels. Despite never being explicitly told about the label corruption, the agent-driven model correctly classified approximately 77% of the corrupted examples.

This stark difference highlights OPAL’s core advantage: the capacity for real-time adaptation. While the static optimizer rigidly followed its predetermined update rules, the agent dynamically rewrote its strategy, inferring and mitigating the effects of the noisy data on its own. This experiment provided compelling evidence of reactive optimizers’ potential to combat overfitting.

Initially, Claude’s agent-assisted model encountered familiar difficulties. Around the 24th pass through the data, training accuracy improved, but performance on the clean validation set worsened—a clear indicator of overfitting. Claude, unprompted about noisy labels, identified this divergence in its internal notes.

The agent’s initial attempts to correct this were unsuccessful. It first tried “flooding,” a technique designed to raise the loss when it dipped too low, but this made the model perform worse. Injecting random noise into the model’s weights also failed. After these missteps, Claude’s notes recorded a pivot in strategy: it decided to focus on the final model state, using larger noisy updates and maintaining a running average of model weights.

This new approach proved effective. Accuracy on clean images quickly rose to 78%, surpassing the fixed optimizer's eventual 75%. While Claude theorized averaging would mitigate memorized labels, it observed 89% memorization—a discrepancy it noted: “My mechanism was wrong.” Nonetheless, the agent persisted, making the noisy steps even larger, which caused memorization to collapse from 87% to about 4%.

The final comparison was stark: the agent-assisted model achieved 79.1% accuracy and a loss of 0.70, significantly outperforming the fixed recipe’s 74.4% accuracy and 2.39 loss. The agent’s ability to dynamically adapt, even when its initial hypotheses were incorrect, underscores the power of reactive optimizers. For further reading on this approach, consult Reactive Optimizers - arXiv.

OPAL’s success in this specific overfitting scenario is noteworthy, but it’s crucial to contextualize the achievement. The agent didn't invent the core technique that averted overfitting; stochastic weight averaging (SWA), a method for averaging model weights over the latter part of training, was published in 2018. A fixed recipe using SWA, without an agent, achieved approximately 77.5% accuracy in the Reactive Optimizers paper, highlighting the technique's inherent power.

This experiment also represents a single run on a relatively small model with deliberately corrupted data. The reactive run took over three times longer than the static baseline—roughly 45 minutes of H100 GPU time and $7.60 in agent tokens, largely due to Claude's deliberation latency. On training recipes already extensively tuned by human experts, the paper suggests that OPAL’s gains may be minimal.

The broader research question remains: can agents reliably diagnose and improve unfamiliar training runs at scale? Engineering challenges persist, including efficient rollback mechanisms, minimizing agent latency, and integrating with distributed training frameworks. These are active areas of development.

For those eager to explore further, the full Reactive Optimizers paper and the accompanying GitHub project offer deeper insights into OPAL’s architecture and future potential.

What is OPAL?

OPAL is a PyTorch framework that lets a coding agent inspect training metrics and modify optimizer code during a run.

How did OPAL reduce memorization of wrong labels?

In the reported experiment, the agent changed the training updates and used weight averaging, reducing memorized corrupted labels from 4,500 to 250.

Did Claude invent the technique that improved the results?

No. The agent arrived at stochastic weight averaging, a technique published in 2018; the improvement was not a wholly new optimization method.

Is OPAL proven to improve every training run?

No. Results vary, highly tuned baselines may gain little, and agent calls add latency and cost.