Composer doesn't generalize

TL;DR#

Neural nets might not be pure black boxes to us anymore, but frontier AI labs certainly are.

As frontier labs become ever more secretive, someone else needs to produce public knowledge on critical questions like how generalization works in industrial-scale training. Producing this knowledge increasingly calls for strategies from social science. Much of our day to day work focuses on leveraging social deduction, historical assumptions, and natural experiments to build reasonable hypotheses about how generalization works at the frontier.

A major question we need to answer is “how far does training on code generalise to other domains?” A closely-related one is “is narrow AI doomed?”.

In a huge stroke of luck, the AI lab Cursor provided us with an expensive natural experiment, by spinning off the Composer 2 series of coding-finetuned models from Moonshot’s Kimi K2.5 instruct-tuned model. They spent a lot on this spinoff:

We used this natural experiment to ask how much ‘horizontal generalization’ happens within frontier model training. When a reasonably strong base-model receives heavy post-training in a domain like coding, what is the effect on other capabilities-domains?

Comparing capabilities is, of course, tricky business: Composer 2.5 was optimized to live in Cursor whereas Kimi k2.5 was not, so the most natural comparison -- Composer 2.5 in Cursor versus raw Kimi k2.5 -- tests differences not between models but between model+scaffold combos. Comparing Composer 2.5 in Cursor to Kimi k2.5 in Cursor, on the other hand, gives a mechanically clean comparison between Composer 2.5 and Kimi k2.5 but has a worse claim to ecological validity.

Rather than choose, we ran both the “product-to-product” comparison (Composer 2.5 in Cursor versus raw Kimi k2.5) and the “model-to-model” comparison (Composer 2.5 inside Cursor versus Kimi k2.5 inside Cursor).

“Product-to-product”#

Intuitively, the main tested differences between Composer 2.5 in Cursor and Kimi 2.5 raw seem to be:

• Significant gains on visual reasoning benchmarks
• Significant gains on RPG-style games
• Significant gains on agentic coding
• Severe losses on mathematical reasoning benchmarks
• Severe losses on scientific reasoning benchmarks

But, taken aback by the size of the losses on math and science, we had to check whether these losses are artifacts of the Cursor harness or Cursor API token-politics rather than evidence of negative generalization.

"Model-to-model"#

We built a harness for using Composer 2.5 outside of Cursor through xAI’s grok-composer-2.5-fast API. Using this, we found that, while Composer 2.5’s math and science performance does improve a little outside Cursor (at the cost of agentic coding) the deficit remains huge:

Finally, we wanted to get a mechanically clean “model-to-model” comparison between Composer 2.5 in Cursor -- its natural habitat -- and Kimi 2.5 in Cursor, which after all does offer Kimi k2.5 as an option. We found that while Cursor is generally bad for Kimi k2.5 and often beneficial for Composer 2.5, Kimi k2.5 superiority in math and science reasoning remain strong. What did newly emerge is a significant advantage for Composer 2.5 on factual recall tasks:

---

How should these gains and losses be factored? Assuming that Composer 2.5's post-training really is almost-all coding, is there a clear generalization-dynamics narrative we can extract?

It's not Claudiness#

One hypothesis we considered early but which didn’t quite match up is that we’re seeing what Epoch AI called ‘Claudiness’. Epoch AI found that Claude series AIs – which are generally believed to be more heavily coding-assistant trained than other frontier models – tend to outdo non-Claude AIs on tool-heavy, agency-heavy tasks while nominally underperforming in math and science.

In our product-to-product comparison (Composer 2.5 in Cursor versus Moonshot-served Kimi 2.5), things do look a bit Claudey, with the exception of large gains in visual reasoning:

And matters get even less Claudey on a ‘model-to-model’ (Compose 2.5 in Cursor to Kimi k2.5 in Cursor) comparison:

-------------------------------------------------

As Greg Burnham acutely says in Epoch’s Claudiness post, one of the key high-level questions in the study of frontier AI is whether capability clusters are ‘deep’ or contingent. We currently favor a ‘contingentist’ interpretation of the gains side of our Composer 2.5/Kimi k2.5 comparison: We think Composer 2.5’s benchmark gains on visual reasoning and RPG-style games likely come from optimization on adjacent subdomains and on subdomain-useful skills, such as front-end coding (adjacent to visual reasoning and to RPG-style games) and software use (useful for RPG-style games).

We find it harder to explain Composer 2.5’s severe deterioration on mathematical reasoning. One factor may be that Composer 2.5 is not only coding-optimized but optimized to function as a “budget conscious” model, and mathematics benchmarks are reasoning-tokens hungry. Our estimates of Composer 2.5’s in-Cursor token use on math benchmarks seem to support this, with the caveat that they are estimates rather than proper counts:

But even assuming that these estimated numbers hold, a ‘budget-consciousness optimization’ factor gives only a partial account of the gap. When deploying Composer 2.5 outside of Cursor, the “budget-consciousness” effectively disappears (on math problems), but a large math benchmark performance gap between Composer 2.5 and Kimi k2.5 remains:

---

Needless to say, we cannot assume that coding is a generic example domain. Nor can we assume that testing out-of-domain generalization across a relatively sharp divide like coding/non-coding is a perfect proxy to more amorphous questions about whether post-training approaches ‘general intelligence’ or tends towards more local capabilities-gain.

Still, we believe that a Composer 2.5/Kimi k2.5 comparison is a valuable natural experiment for anyone seeking to understand what’s going on with generalization near the frontier. We invite everyone to experiment with our repo, and to develop and test their own hypothesis about our scoreboard:

Methodology#

(Niccolo)

Footnotes#

1. Terminal-Bench was run through the Grok Composer route using `cursor-agent-local`, whereas the official Cursor comparison used `cursor-agent`; the two runtimes reported the same visible version but were not byte-identical. The strict Grok run scored 42/89 with 17 errored trials, mostly agent timeouts. Targeted reruns raised the estimated score to ~50/89, but we treat those reruns as diagnostic rather than benchmark-comparable. ↩