Coding vs thinking

TL;DR#

Neural nets might not be pure black boxes to us anymore, but frontier AI labs certainly are.

As frontier labs become ever more secretive, someone else needs to produce public knowledge on critical questions like how generalization works in industrial-scale training. Producing this knowledge increasingly calls for strategies from social science. Much of our day to day work focuses on leveraging social deduction, historical assumptions, and natural experiments to build reasonable hypotheses about how generalization works at the frontier.

A major question we need to answer is “how far does training on code generalise to other domains?” (A closely-related one is “is narrow AI doomed?”.)

In a huge stroke of luck, the AI lab Cursor provided us with an expensive natural experiment, by spinning off the Composer 2 series of coding-finetuned models from Moonshot’s Kimi K2.5 instruct-tuned model. They spent a lot on this spinoff:

We used this natural experiment to ask how much ‘horizontal generalization’ happens within frontier model training. When a reasonably strong base-model receives heavy post-training in a domain like coding, what is the effect on other capabilities-domains?

Comparing capabilities is, of course, tricky business: Composer 2.5 was optimized to live in Cursor whereas Kimi k2.5 was not, so the most natural comparison — Composer 2.5 in Cursor versus raw Kimi k2.5 — tests differences not between models but between model+scaffold combos. Comparing Composer 2.5 in Cursor to Kimi k2.5 in Cursor, on the other hand, gives a mechanically clean comparison between Composer 2.5 and Kimi k2.5 but has a worse claim to ecological validity.

Rather than choose, we ran both the “product-to-product” comparison (Composer 2.5 in Cursor versus raw Kimi k2.5) and the “model-to-model” comparison (Composer 2.5 inside Cursor versus Kimi k2.5 inside Cursor).

“Product-to-product”#

Intuitively, the main tested differences between Composer 2.5 in Cursor and Kimi 2.5 raw seem to be:

• Significant gains on visual reasoning benchmarks
• Significant gains on RPG-style games
• Significant gains on agentic coding
• Severe losses on mathematical reasoning benchmarks
• Severe losses on scientific reasoning benchmarks

But, taken aback by the size of the losses on math and science, we had to check whether these losses are artifacts of the Cursor harness or Cursor API token-politics rather than evidence of negative generalization.

“Model-to-model”#

We built a harness for using Composer 2.5 outside of Cursor through xAI’s grok-composer-2.5-fast API. Using this, we found that, while Composer 2.5’s math and science performance does improve a little outside Cursor (at the cost of agentic coding) the deficit remains huge:

Benchmark	Metric	Composer 2.5 (free)	Kimi K2.5 (free)	Δ
MathArena AIME 2025	sample accuracy, n=4, no tools	62.50%	95.00% run	-32.5
MathArena AIME 2026	sample accuracy, n=4, no tools	66.67%	95.00% run	-28.3
MathArena HMMT Feb 2026	sample accuracy, n=4, no tools	42.42%	81.30% ref	-38.9
HLE-Verified Gold text-only	judged accuracy, no tools, 575 tasks	12.35%	52.35% run	-40.0
Terminal-Bench 2.0	pass rate, 89 tasks	47.2%1	50.8% ref	-3.6

Finally, we wanted to get a mechanically clean “model-to-model” comparison between Composer 2.5 in Cursor — its natural habitat — and Kimi 2.5 in Cursor, which after all does offer Kimi k2.5 as an option. We found that while Cursor is generally bad for Kimi k2.5 and often beneficial for Composer 2.5, Kimi k2.5 superiority in math and science reasoning remain strong. What did newly emerge is a significant advantage for Composer 2.5 on factual recall tasks:

---

How should these gains and losses be factored? Assuming that Composer 2.5’s post-training really is almost-all coding, is there a clear generalization-dynamics narrative we can extract?

It’s not Claudiness#

One hypothesis we considered early but which didn’t quite match up is that we’re seeing what Epoch AI called ‘Claudiness’. Epoch AI found that Claude series AIs – which are generally believed to be more heavily coding-assistant trained than other frontier models – tend to outdo non-Claude AIs on tool-heavy, agency-heavy tasks while nominally underperforming in math and science.

In our product-to-product comparison (Composer 2.5 in Cursor versus Moonshot-served Kimi 2.5), things do look a bit Claudey, with the exception of large gains in visual reasoning:

Benchmark	Cursor Composer 2.5	Moonshot Kimi K2.5	Result vs prediction
CadEval	76.0%	80.0%	Contrary to positive CadEval Claude factor
GPQA Diamond, strict	72.2%	76.8%	Consistent
GPQA Diamond, parser-fixed	75.3%	87.4%	Consistent, strong
VPCT-1	69.0%	43.0%	Strongly contrary

And matters get even less Claudey on a ‘model-to-model’ (Compose 2.5 in Cursor to Kimi k2.5 in Cursor) comparison:

Benchmark	Local Claude factor	Direction predicted by hypothesis	Cursor Composer 2.5	Cursor Kimi K2.5	Result
CadEval	+3.58	Composer should win	19/25 = 76.0%	20/25 = 80.0%	Contrary, small
GPQA Diamond, strict	-1.00	Kimi should win	143/198 = 72.2%	121/198 = 61.1%	Contrary, but parser-biased
GPQA Diamond, parser-fixed	-1.00	Kimi should win	149/198 = 75.3%	151/198 = 76.3%	Consistent, tiny
VPCT-1	-2.90	Kimi should win	69/100 = 69.0%	30/100 = 30.0%	Strongly contrary
SimpleQA Verified	-4.33	Kimi should win	410/1000 = 41.0%	329/1000 = 32.9%	Contrary

---

As Greg Burnham acutely says in Epoch’s Claudiness post, one of the key high-level questions in the study of frontier AI is whether capability clusters are ‘deep’ or contingent. We currently favor a contingent interpretation of the gains side of our Composer 2.5/Kimi k2.5 comparison: We think Composer 2.5’s benchmark gains on visual reasoning and RPG-style games likely come from optimization on adjacent subdomains and on subdomain-useful skills, such as front-end coding (adjacent to visual reasoning and to RPG-style games) and software use (useful for RPG-style games).

We find it harder to explain Composer 2.5’s severe deterioration on mathematical reasoning. One factor may be that Composer 2.5 is not only coding-optimized but optimized to function as a “budget conscious” model, and mathematics benchmarks are reasoning-tokens hungry. Our estimates of Composer 2.5’s in-Cursor token use on math benchmarks seem to support this, with the caveat that they are estimates rather than proper counts:

Run	Score	Avg reasoning/output tokens
AIME 2025 Cursor Composer 2.5	56.67% (68/120)	4,647
AIME 2025 OpenRouter Kimi K2.5	95.00% (114/120)	21,226
AIME 2026 Cursor Composer 2.5	50.00% (60/120)	9,335
AIME 2026 OpenRouter Kimi K2.5	95.00% (114/120)	21,382

But even assuming that these estimated numbers hold, a ‘budget-consciousness optimization’ factor gives only a partial account of the gap. When deploying Composer 2.5 outside of Cursor, the “budget-consciousness” effectively disappears (on math problems), but a large math benchmark performance gap between Composer 2.5 and Kimi K2.5 remains:

Run	Score	Avg reasoning/output tokens
AIME 2026 Grok Composer 2.5	66.67% (80/120)	20,667
AIME 2026 OpenRouter Kimi K2.5	95.00% (114/120)	21,382

---

Yet more confounders#

While we believe that our results are informative, we warn that our methodology involves some black box elements due to the lack of proper API endpoints for Composer 2.5:

• when models are served through Cursor, we cannot control for the system prompt injected, inference settings used, etc;
• when Composer-2.5 is served through Grok Build we cannot fully control for their inference settings, particularly regarding the max tokens used per session.

To sanity-check our results, we briefly looked at two more recent coding-focussed spinoff models: Ornith 1.0 397b, a coding finetune of Qwen 3.5 397b, and the recently released Kimi K2.7-code. In the case of Ornith 1.0 397b — a model believed to be lightly finetuned and possibly ‘benchmaxxed’ — our battery of non-coding benchmarks mostly found no real differences in capability compared to base Qwen 3.5 397b. (The strong exception of Apex 2025 from MathArena strikes us as suggestive of contamination.)

With Kimi k.2.7-code, collating publicly recorded benchmarks was itself enough to mostly reconstruct our Composer 2.5 results: with one exception (the CritPt physics benchmark), Kimi k2.7-code’s non-coding benchmarks stagnated or decreased relative to its Kimi k2.6 base.

Outro#

Needless to say, we cannot assume that coding is a generic example domain. Nor can we assume that testing out-of-domain generalization across a relatively sharp divide (coding/non-coding) is a perfect proxy for amorphous questions about whether post-training approaches ‘general intelligence’ as it scales or whether it instead tends towards local capability gains.

Still, we believe that a Composer 2.5/Kimi k2.5 comparison is a valuable natural experiment for anyone seeking to understand what’s going on with generalization near the frontier. We invite everyone to experiment with our repo, and to develop and test their own hypothesis about our scoreboard:

Methodology

Since Cursor does not expose an open API for Composer-2.5, we evaluated the model by querying it through two separate routes.

1. Composer via Cursor#

The first way we accessed Composer 2.5 was through Cursor itself. We used the standardagents/composer-api project as the basis for a local OpenAI-compatible adapter: the benchmark sent ordinary chat-completion requests to a localhost server, and the adapter converted those requests into the kind of authenticated backend request the Cursor editor uses when asking Composer for a reply. In other words, this route did not expose Composer 2.5 as a standalone API model; it exposed Composer 2.5 through Cursor’s own harness and serving stack.

Composer 2.5 served through this path was a valuable baseline, but easily confounded: Cursor’s backend adds its standard coding-assistant system prompt, and we were not able to remove or replace it. The route also did not provide reliable control over sampling parameters such as `temperature`, `top_p`, or `max_tokens`. We therefore treat these results as “Composer 2.5 as served by Cursor,” not as isolated measurements of the underlying model.

For benchmarks intended to run without tools, we ran the adapter in a dedicated mode that did not allow tool usage: if the model attempted to call a Cursor tool, the call was rejected, logged, and either retried under the benchmark policy or counted as invalid if it persisted. Where possible, we also evaluated Kimi K2.5 through the same Cursor-serving route, to produce a ‘model-to-model’ comparison where Composer and its parent model shared the same Cursor-side prompt and serving confounders.

2. Composer via Grok Build#

Looking to also study Composer 2.5’s input-output dynamics in isolation from Cursor’s coding-assistant system prompt, declared tools, and agent-harness behavior, we later reverse engineered the HTTP endpoint used by Grok Build, which had recently added support for grok-composer-2.5-fast. We used this HTTP endpoint as an API-like access path to Composer 2.5 outside Cursor.

Concretely, we implemented a local OpenAI-compatible bridge that translated benchmark requests into `POST https://cli-chat-proxy.grok.com/v1/responses\` calls with the model override `grok-composer-2.5-fast`. The bridge used Grok CLI OAuth credentials, and when a benchmark supplied no system/developer message, the bridge inserted the neutral shim `Follow the user’s instructions exactly.`. The Grok Build endpoint exposed ordinary sampling controls such as `temperature`, `top_p`, and `max_output_tokens`, but did not expose a reliable reasoning-effort parameter; attempts to use `reasoning.effort` were rejected or unsupported.

A significant challenge of the raw endpoint is what appears to be a hard generation cap of 16,384 tokens per request. For this reason, in select reasoning-heavy benchmarks such as MathArena, we enabled stateful continuation: when a response ended at the cap, we resumed from the returned previous_response_id and asked the model to continue the same answer. This affected only a small fraction of samples

Benchmarks#

Across benchmarks, we followed the original benchmark protocols as closely as possible while adapting them to the two Composer access routes described above. Unless a benchmark was explicitly agentic, evaluated models were run without access to tools, web search, code execution, file access, or external retrieval. This includes MathArena, HLE-Verified, GPQA Diamond, SimpleQA Verified, CadEval, VPCT, CritPt, and ADeLe-Light. In contrast, benchmarks such as Terminal-Bench 2.0, RuneBench, and PokeAgent require tool use by design; for those, we ran the adapter in a dedicated tool-enabled mode.

For Kimi K2.5 comparisons outside Cursor, we used OpenRouter with routing pinned to Moonshot AI and fallbacks disabled. We used standard inference settings (`temperature=1.0` and `top_p=0.95`) and API errors were retried automatically with the same prompt and inference settings; persistent malformed outputs, empty responses, or tool-use attempts were either counted as invalid/failures or

A small number of benchmarks required minor implementation adjustments because Composer 2.5 was not available through a clean first-party API. GPQA Diamond is one example: Composer-2.5 sometime struggled to output a response matching the format expected by the official parser, possibly due to Cursor’s system prompt. We therefore used a slightly more permissive parser in case of parsing failure, as to nonetheless be able to evaluate the model’s capabilities. Ih ADeLe-Light, we also had to make a slight adjustment to official methodology: the original judge pair used by the paper (gpt-4o and Claude Sonnet 3.5) was no longer available via API, so we calibrated replacement judges against the ADeLe release’s archived judge labels and selected the pair that best reproduced the published binary labels.

Footnotes#

1. Terminal-Bench was run through the Grok Composer route using `cursor-agent-local`, whereas the official Cursor comparison used `cursor-agent`; the two runtimes reported the same visible version but were not byte-identical. The strict Grok run scored 42/89 with 17 errored trials, mostly agent timeouts. Targeted reruns raised the estimated score to ~50/89, but we treat those reruns as diagnostic rather than benchmark-comparable. ↩