How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

A paper from Apple Machine Learning Research examines how much scaffolding, or harness, a strong agent needs for autonomous machine learning engineering (MLE). The authors are Kirill Brilliantov, Alejandro Hernández-Cano, and Emmanuel Abbé, with affiliations to EPFL; Brilliantov and Hernández-Cano are listed as equal contributors, and the work was done while at Apple.

The paper notes that recent autonomous MLE agents have made significant progress on public leaderboards. It says modern MLE agents are often motivated by progress stagnation over long-horizon cycles and limited Large Language Model primitives, and are deployed on increasingly elaborate machinery including multi-agent orchestrators and dedicated retrieval subagents. Meanwhile, the use of more primitive but improved coding agents, where LLMs have direct access to the execution environment through read, write, and bash primitives, has received little attention in the field.

The authors report that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline. They point to the backbone as the primary driver for performance. Through a series of large-scale systematic ablation studies, they argue that the machinery layers become redundant in the coding agent setting.

The paper concludes that effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.