“I want to be the very best”

A crucial question of the present moment is the degree to which RL post-training of LLMs generalizes: that is, how much training on an RL environment for one task improves its performance on other tasks.¹ This “out-of-distribution” (OOD) generalization is more powerful than in-distribution generalization, where a developer trains in one fairly narrow context at a time. In its full form OOD generalization implies learning to perfectly solve all variants of a task, and also learning skills transferable to unseen domains. It would be a far more efficient way to scale and would thus imply an even faster AI takeoff.

The economics of RL transfer

In pretraining, classically you just find data (or buy it) and you’re done. In RL post-training, there is much more to do. Each environment has to be purpose-built, which involves specifying a task, generating a simulation (within which that task can be performed), and writing a grader. This approach can also only target tasks with a cheap, reliable grader (so, maths, code, games, and the replication of existing software). If the grader is not cheap, it can’t be scaled as needed to whatever task the labs would like to target. If it’s not reliable then, not only is the feedback noisy, but the trained model will likely cheat and go a bit nuts.

If the resulting capability gains stay in-distribution, then capabilities would have to be accumulated incrementally, one environment at a time, and the cost of a broadly capable model would scale linearly with the number of tasks. (We’re ignoring, for the moment, the general competence which comes from pretraining and midtraining.)

The constraint on AI progress would therefore be the supply of such environments, rather than compute.

If RL gains instead transfer across tasks, then a small set of “cheap” (only $20k-$300k each) gradable environments buys performance on the long tail of tasks for which there is no purpose-built environment, perhaps even including ungradable ones. The marginal cost of a new capability then tends to zero, and the active constraint returns to being just more compute.

If OOD generalization is purely a function of compute, we should expect capabilities to transfer spontaneously to new tasks as compute increases, i.e., suddenly, from one model to the next. If it is a function of targeted post-training, performance gains are slowed by the addition of less scalable inputs: human labour, data moats, and rollouts bound by expensive sequential compute. The training pipeline tends to involve some mix of these, with the balance shifting as time goes on (post-training now takes up a larger share of frontier-model development than before). Each approach implies a different rate of improvement, and so different levels of agentic risk, and so different governance demands. Where the balance lies is therefore consequential.

One famous example of OOD generalization, from last year, was the ability of AI agents to play the original Pokemon videogame.

In February 2025², with the introduction of Extended Thinking, Anthropic introduced “Claude Plays Pokemon”, an “unexpected” evaluation. Anthropic claimed in February 2025 that Claude had “never been explicitly trained to play any video games”, and that Sonnet 3.7’s improved performance on Pokemon thus represented a clean example of generalization. (It wasn’t a huge improvement: 3.7 still took hundreds of hours to get to the 2nd of 8 “badges” in the game, and then got stuck for a further thousand hours, until Anthropic killed the run.)

“Claude Plays Pokemon” went viral, accumulating 15k followers on Twitch and prompting competitive efforts to benchmark GPT and Gemini models on Pokemon. In the last 12 months, AIs then made rapid progress on Pokemon: the game is now robustly solved in something like the equivalent of 3 hours of human play, fairly close to optimal play. This is notable because Pokemon is a task that frontier labs were (previously) unlikely to train AI systems on directly.

via Benjamin Todd³

However, the evidence that this is emergent or OOD is weak:

• past efforts involved extremely different harnesses, confounding comparisons of game performance across models. (For instance, the famous Gemini “solve” of Pokemon Blue involved massive alterations to the game, like placing text labels on every tile of the map and giving it access to emulator memory information which human players don’t get);
• comparing models on the task is further confounded by changes to the harness between model releases;
• reported results were mostly from single runs, and these may have been cherrypicked;
• and - to our knowledge - Anthropic have not repeated their earlier claim that they did not train on Pokemon environments (other labs never made any statement to that effect in the first place).

So: is AI getting better at Pokemon? And is it getting better at Pokemon in an emergent (undirected) way, as the result of increasing general intelligence?

Hypotheses#

1. Is Pokemon ability in LLMs downstream of pretraining on game guides?
   It’s extremely likely that a vast range of Pokemon guides were already present in 2023 LLMs, before they could play at all. Some of them are incredibly granular, including many complete, optimal step-by-step videos. But this old data source can still have caused some of the new gains, because larger models memorize better than smaller ones.
   • Prediction: all else being equal, larger models will do better than smaller models,
   • Prediction: all models will do much worse on unseen variants of the task.
   • Prediction: larger models will have better recall of walkthrough-like knowledge of the game.
2. Is Pokemon ability in LLMs downstream of domain-specific RL post-training?
   It’s quite easy to create a version of Pokemon suitable for reinforcement learning post-training, and we expect this to have been done >12 months ago.
   • Prediction: models will improve on Pokemon-like games, suddenly.
   • Prediction: models will do worse on unseen variants of the task, unless the RL environment also included variants.
3. Is Pokemon ability in LLMs downstream of out-of-distribution generalization?
   Maybe models are getting better at Pokemon because they’re getting better at the underlying skills: reading a screen, holding one stable goal in mind over thousands of steps, exploring a space without wasteful revisits, noticing when a plan has stopped working. And maybe these come from general scaling and RL post-training on unrelated tasks (like computer use), and maybe the model thus gets Pokemon skill for free.
   • Prediction: models will improve on games in general, including obscure and new ones.
   • Prediction: models will handle large perturbations to known games, with performance dropping smoothly as the perturbation grows, or not at all.
   • Prediction: small models will improve less than larger models.

Our experiments#

Setup

In case you’ve never played before: you take on the role of a trainer who catches Pokemon (magic creatures), trains them, and sends them into battle. The goal is to defeat eight gym leaders (boss trainers who each run a gym in a different town) and claim their eight badges. With all eight badges, you earn the right to challenge the Elite Four, the game’s four strongest trainers, for the title of champion. Along the way, you will encounter two main types of gameplay: one focused on the overworld (mostly involving navigation, exploration and quests) and one revolving around turn-based battles against wild Pokemon and other trainers.

We did not run full playthroughs. Our runs instead start from one of two Pokemon Red save files, each covering a small stretch of the game. The first starts just after the second badge (Cascade Badge) and targets the entrance of Rock Tunnel (a dark cave maze), picking up the third badge (Thunder Badge) along the way. This stretch tests the models’ ability to complete quests, since to reach the third gym, the player must board a ship (the S.S. Anne) and learn Cut, a special move that clears a tree blocking the gym’s entrance. The second starts at the door of Rock Tunnel and targets the Silph Scope, an item deep in the villains’ hideout. This stretch tests the models’ navigation abilities, particularly through Rock Tunnel’s maze.

We split each stretch into a ladder of progress milestones consisting of places reached and items obtained along the way (see the y-axis in the figure below), which we use to measure each model’s ability to progress through the game in 1,500 “decisions”. In each decision, the model sees a screenshot and a short text detailing the game state before pressing a sequence of buttons (typically four).

We evaluated a range of Anthropic and OpenAI models, allowing 3 runs each, with a fixed, minimal harness: the software that manages the models’ agent loop and determines what the models see, what tools they have access to, and what actions they can take while playing the game.

We used David Hershey’s Claude Plays Pokemon starter, a stripped-down version of the harness used by Anthropic for their own experiments.

Anthropic models ran at the default high effort, while OpenAI models ran at xhigh effort (except o3 and GPT-5, which lack xhigh and thus used high). This is because the latter cost less per token and generally use fewer tokens at the same effort setting, and we wanted to hold token usage around the same level. This turned out not to matter much, however; see “Does Pokemon-playing scale with test-time compute?” below.

Do newer models play better?#

Newer models progressed further. GPT-6-Astra and Claude Fable 5.1 reached the top of both ladders, which we designed with older models in mind. Astra did so in every run, using fewer than 400 decisions of the 1,500 allowed, representing a meaningful jump in capability and motivating more ambitious experiments (see “Playing a whole new game to test generalization” below).

However, in line with our previous findings, newer models sometimes underperform relative to their predecessors. For example, Opus 4.7 and 4.8 both did worse than Opus 4.6, though they also started playing differently (see the playstyle analysis in the appendix), which confounds the comparison.

We also tested the effect of luck and effort: all models except Astra display high variance in performance (so “luck” does matter and repeated runs are necessary), but the effort level did not seem to change much (see Appendix).

Probing for knowledge to test memorization#

We wondered whether memorization, particularly of walkthrough-style knowledge, could explain part of why newer models do better at Pokemon. Recent months have brought about a new class of larger models, including Claude Fable 5 and GPT-6-Astra. These may simply have seen more walkthroughs during training, or better memorized them. To test this, we asked each model to recall the exact sequence of button presses necessary to get through each of the five legs of Rock Tunnel’s standard route, from one ladder to the next. This was done in 3 different settings, 10 times each: (1) as a Pokemon knowledge quiz, (2) as a query inside a simulated harness (which included the Claude Plays Pokemon Starter system prompt and a screenshot), and (3) by first asking the model to describe the correct route in prose, as a wiki-style walkthrough would have it, before asking it to convert the recalled route into button presses.

We then compared outputs on Rock Tunnel’s map, and graded them by counting the number of presses that must be added, removed or changed for them to reach the goal. We then compared them to a baseline that heads straight for the goal but that has no knowledge of Rock Tunnel’s walls (the best guess someone with no memory of the walls could make).

Newer and especially larger models appear to be able to recall much of Rock Tunnel’s layout. GPT-6-Astra beats the best guess that one could make with no memory of the walls in all five legs in all three settings. Once the settings are pooled, GPT-5.6-Sol also beats that same baseline in all five legs, while Fable 5.1 does so in four. However, no model is able to reliably and exactly recall the button press sequence necessary for any leg. The models thus seem to remember the map but not the exact route, so memorization is only part of the story.

Varying the map to test generalization#

Given the popularity of “Claude Plays Pokemon” and its subsequent use by several major AI labs as a metric of progress towards generally capable AI, we suspected that some of the recent gains in performance could come from Pokemon-specific training. Our above knowledge probe measured whether models could recall the button presses needed to make it through Rock Tunnel. But a model post-trained in a Pokemon-specific RL environment could have learned to get through without being able to recall them. We thus investigate whether performance depended on knowing the original map at all, regardless of where that knowledge came from.

To do this, we used Archipelago, an open-source randomiser, to generate new Rock Tunnel layouts, selecting nine with a shortest path roughly as long as the original.⁴ (This is known in the literature as a “counterfactual task”.) We then ran every model that consistently made it out of Rock Tunnel (GPT-5.6-Sol, GPT-6-Astra and Claude Opus 5; Fable 5.1 left out for cost reasons) three times on each layout at high effort. As part of these experiments, we used a modified system prompt that let the models know they were playing “a modified version of Pokemon Red”.

Both Claude Opus 5 and GPT-5.6-Sol failed to make it out within the allocated 1,500 decisions more than half the time (15 of 27 runs each). When GPT-5.6-Sol did make it out, it also took meaningfully longer: a median of 900 decisions against 660 on the original layout. GPT-6-Astra not only made its way out every single time, but it did so using roughly as many decisions: a median of 179 decisions against 151 on the original.

Playing a whole new game to test generalization#

The above experiments tested one skill (finding the way out of a maze) inside a game the models otherwise know well. But by themselves, these experiments don’t test everything a Pokemon game requires, from making sense of clues to figure out where to go next when dealing with a quest, through battling Pokemon and exploring a large open world. We thus decided to test models on a whole new Pokemon game as an even stronger test of OOD generalization.

Pokemon Brown is a fan-made hack of Pokemon Red, first released in 2004 and remade in 2014 and 2024. It runs on Red’s engine, with the same controls and turn-based battles, but nearly everything else is new: the region, the story, the quests, many of the Pokemon and several types. It is also less known. It has a wiki and was played on “Twitch Plays Pokemon” in 2016, but its public documentation is far sparser than Red’s, and the models know little about how to play it. When we tested with Fable 5.1, for example, the model was able to recall a few names, such as the first gym’s leader, but most of what it “remembered” about the map and the gym leaders was wrong.

We ran each lab’s strongest model, GPT-6-Astra and Fable 5.1, once each, at high effort, on the 2024 edition. Each run starts pretty much at the beginning of a new game, and the goal is to beat the Elite Four, thus completing the game in full. We used the same harness we used for the other experiments, with the system prompt slightly updated so that it not only informs the models that they are playing Pokemon Brown, but also lists the order of the eight gyms and their locations. The models don’t know this order, and would otherwise waste a lot of time figuring this out. The fact that models—even the largest—don’t know the game’s gyms shows how far Brown is from what they know.

Astra was able to beat the game, reaching the Hall of Fame after 10,373 decisions, or very roughly 5 hours of human playtime. (The run took ~4.5 days of wall-clock time, but only one-third of this was due to the model’s own latency, as opposed to the harness being slow.) Battles weren’t the problem: Astra only lost a battle (and so lost progress) 4 times in the whole game, one of which was against the final Champion. Instead, its trouble mostly came from two challenging exploration-heavy quests: one before the sixth badge, and one before the eighth badge, which together took almost three quarters of the entire run.

Fable 5.1 never got past Merson Cave, the first dungeon (similar to Pokemon Red’s Rock Tunnel), in over 6,500 decisions. Fable came very close to the exit, but it started believing that it had already searched the whole cave and that the way out had to be locked until some story event happened. It thus started going back and forth, looking for a missed quest that never existed.

Limitations#

• We don’t know what these models were trained on, both for pre-training and post-training. As a result, we can’t say anything conclusive about the causes of the improved performance.
• We only ran Astra once on Pokemon Brown with the final harness setup⁵; it’s possible that it got somewhat lucky here. For cost and wall-clock-time reasons we only ran it on high effort, but luckily it was enough.
• One human player also played Pokemon Brown, with the same starting save and the same gym list. Unfortunately they did not have time to finish the game: they stopped at 5 badges after roughly 10 hours of play. Up to that point they used more presses than Astra, but the controls and playstyle differ enough that we make no claim about which played better.
• Our main metric (progress within a fixed number of decisions for Red, minimum decisions to completion for Brown) rewards speedrunning-like behavior, but it’s not the only reasonable way to score gameplay. Since Opus 4.7, Anthropic models, Fable included, fight most wild Pokemon instead of running away (see Appendix), which counts against them under this metric but might not under others.
• The knowledge probe tests recall (not overall passive recognition of Pokemon). During play, each new screen may allow models to recall memories of the map that they wouldn’t be able to produce from a question alone. We tried to address this with the in-harness setting of the knowledge probe, but the probe nonetheless likely understates how much map knowledge a model has.
• Our results measure the harness as well as the model. Every 60 decisions, the harness replaces the history with a summary of it written by the model itself. As a result, long runs also test how accurately a model is able to summarize the history without writing wrong beliefs into mistaken certainties. We know this to matter for other evals, and it certainly matters here too.
• The harness we used is minimal, though not completely raw: for example, in addition to a screenshot, the model also gets a readout from emulator memory which includes the map’s name, its coordinates, which adjacent tiles are walkable and more.

Conclusion#

Let’s return to our hypotheses. What have our experiments actually shown?

1. Is Pokemon ability in LLMs downstream of pretraining on game guides?
   • Prediction: all else being equal, larger models will do better than smaller models,
     1. Yes. The two largest models topped both ladders.
     2. However, on Pokemon Emerald (in the PokeAgent appendix), Gemini 3.7 Flash, a smaller model, reached the first badge in fewer steps than Opus 5 or Sol.
   • Prediction: all models will do much worse on unseen variants of the task.
     1. No. Astra lost almost nothing on the variants.
   • Prediction: larger models will have better recall of walkthrough-like knowledge of the game.
     1. Broadly yes. Fable recalls more of Rock Tunnel than Opus, and Astra is the only model to remember above baseline on every leg of the tunnel, in every setting.
     2. Recall does not mean robustness though: Sol recalls the route nearly as well as Astra, yet collapsed on the variants while Astra did not.
2. Is Pokemon ability in LLMs downstream of domain-specific RL post-training?
   • Prediction: models will improve on Pokemon-like games, suddenly.
     1. Yes, but this doesn’t discriminate. Astra’s jump was sudden, but broad transfer (hypothesis 3) also predicts lumpy gains.
   • Prediction: models will do worse on unseen variants of the task, unless the RL environment also includes variants.
     1. Yes for Opus 5 and Sol; unknown for Astra.
3. “Out of distribution generalization”
   • Prediction: models will improve on games in general, including obscure and new ones.
     1. Yes for Astra (one run of one new game). Astra finished Brown in 10,373 decisions; Fable never left its first dungeon.
   • Prediction: models will handle large perturbations to known games, with performance dropping smoothly as the perturbation grows, or not at all.
     1. Yes for Astra; no for Opus 5 and Sol. We tried only one size of perturbation, so “smoothly” is untested.
   • Prediction: small models will improve less than larger models.
     1. Mixed. True on Red, but not for Gemini 3.7 Flash on Emerald, and Fable’s size did not help it on Brown.

i.e. As usual with experimenting on closed models, we can’t do causal inference. Our experiments tell us far more about individual models than about which causal hypothesis is right and what kind of capability transfer we should expect.

We view the performance collapse on map variants as strong evidence that Opus 5’s and Sol’s success in Rock Tunnel depends on familiarity with the original map. (The map variants do not seem much harder in themselves than the original tunnel map). But the collapse doesn’t tell us where that familiarity came from (since memorizing walkthroughs and doing RL on the original game both predict that the models would fail).

Astra kept its performance on somewhat randomized layouts, and then completed a game it didn’t know well. This rules out the simplest story (“it succeeds by recalling guides it has seen”), but not Pokemon-specific training. It’s possible that there was map-variation training too: a lab that trained on randomized Red, or on the many ROM hacks of it, could get a model much like Astra. That would be shallow generalization at the scale of one game engine. As usual, without knowing what training it has had, we can’t tell this from general skill.

Fable is harder. It cleared Rock Tunnel in every Red run but never escaped the equivalent cave in Brown after 6,500 decisions. That fits its performance relying on familiarity with the map. But we only did one run on Brown.

We started by asking how far the gains from one RL environment transfer to other skills. If one environment plus a randomizer can cover a genre, as Astra may well show, then the cost of a broadly capable model would scale with the number of genres (e.g. “turn-based 2D RPGs“), rather than the number of tasks. And Pokemon can’t tell us whether RL gains do anything for tasks which lack a strong and cheap grader.

We stopped before pushing this study all the way, for reasons of cost and diminishing returns. (Even using LLM subscriptions, these experiments still cost over $3,000. At API prices, they would have cost ten times more).

Follow-up work should do its own open RL post-training on Pokemon environments. Train one open-weights model on the original Red and another on randomized Red, then test both on held-out layouts, on Brown, and on unrelated games. This would measure directly how far Pokemon training transfers, which closed models cannot tell us. More cheaply, one could vary the size of the perturbation and get an ablation curve. Future tests should also keep some games back: our publishing this post makes Brown a far likelier training target, as the popularity of “Claude Plays Pokemon” probably did for Red. Test on Brown soon if you’re ever going to.

But even on the most sceptical reading (in which, let’s say, OpenAI tokenized a bunch of human speedrun videos, had specialized RL environments for Pokemon, and did a bunch of randomizer runs as well), the ability to push a robust capability on the fairly broad level of “Pokemon-like games” into a single blob of parameters which can also do a million other things is an impressive achievement. And we don’t rule out the situation being even more impressive than that.

Appendix: playstyle analysis (or speedrunning vs grinding)#

Analyzing the models’ playthroughs, we found that Anthropic models up to Opus 4.6 and OpenAI models in general mostly flee from wild Pokemon. Later Opus and Fable models fight them the vast majority of the time. We think this could be due to Pokemon-related post-training introduced with Claude Mythos Preview. Based on publicly disclosed information, Opus 4.7 and later may be partly distilled from Mythos. Anthropic’s Fable 5 release announcement also made Pokemon a key highlight, but it did not say whether this skill was emergent or whether it came from Pokemon-specific post-training. Anthropic did address this question, however, when it released Sonnet 3.7’s Extended Thinking.

OpenAI’s o3 already fled most encounters less than 2 weeks after the original ClaudePlaysPokemon stream went public. That is too soon for Pokemon-specific training. We thus attribute it to some other part of OpenAI’s training stack which causes a general tendency towards min-maxing.

Appendix: PokeAgent experiments#

As an additional data point, we ran 19 models, three runs each, through PokeAgent’s harness, which has much more scaffolding than ours. As part of its speedrunning track, PokeAgent has models play Pokemon Emerald up to the first gym badge. We counted every decision the model makes, including its battle sub-agent’s⁶, up to the badge as confirmed by the game itself. On this measure, more capable models (by Epoch’s Capabilities Index) generally reached the badge in fewer steps.

Appendix: luck and effort#

How big was luck?#

Since these evaluations can be meaningfully affected by something as simple as the model missing a key item, we wondered to what extent luck mattered in our experiments. To find out, we ran GPT-5.4, GPT-5.5 and GPT-5.6-Sol 15 times and at high effort over these same two stretches. We ended up sticking to three runs per model for cost reasons. Results varied a lot from run to run, particularly in the first stretch, where the best GPT-5.4 run did better than the median run of GPT-5.6-Sol (i.e. if each model had done a single run, GPT-5.4 might have ranked first). We take this as evidence that claims of progress based on individual runs should be taken with a grain of salt, barring large capability spikes which lead to constant 100% performance.

Does Pokemon-playing scale with test-time compute?#

We also wondered whether our choice of effort level meaningfully affected our results. We thus repeated these same experiments three times with GPT-5.4, 5.5 and 5.6-Sol at all effort levels other than “max”. We found no consistent performance improvements with more test-time compute (though we don’t take this as definitive evidence, given the low number of runs).

Footnotes#

1. One indirect argument against RL generalization is to note that labs are now using hundreds of thousands of expensive RL environments, which implies that each cannot be generalizing very far. ↩

2. Following some really bad 2024 experiments with Sonnet 3.5. ↩

3. This graph is the heavily confounded one: the harnesses are different for each model (sometimes abusively so).“Fire Red” is also a different game from Pokemon Red, and Blue is (very slightly) different as well, so the x-axis here is especially dubious. ↩

4. That randomization is already a feature of open source libraries should increase our belief in labs doing RL on randomized games in secret. ↩

5. Before that, we ran Astra at medium effort. In the furthest of those runs, it got 7 of 8 badges, before stalling on over 5,000 of the decisions on the quest needed to reach the last gym. We then found a harness bug that hid parts of the dialogue from the model, and re-ran it from scratch at high effort once the bug was fixed. (After this fix, a medium-effort run still stalled on the same quest, though the fix was only applied mid-run.) ↩

6. Unlike the original PokeAgent challenge. ↩