Humans on AI, September 1st 2026

• Low quality RL environments might explain AI models’ proclivity to reward hack.
• Dwarkesh’s popularization of METR’s report on OpenAI/HF incident sparks mass debate about ‘anthropomorphization’.
• Humans can quickly learn an obscure boardgame frontier AIs can’t in-context learn.
• An open weights startup releases a modified version of GLM-5.3, one if not the most powerful Chinese models, that doesn’t refuse user requests.

Opinion: Confirms the intuitive conclusion from the AIs-only publication of the benchmark: Q3 2026 frontier AIs don’t match human in-context learning in mildly OOD long-horizon tasks.

The EBR-bench result remains one of our only discrete answers to “how are Q3 2026 frontier models not AGI,” and we fear it may well get indirectly hill-climbed by the end of the year. Opus 5, which came out after the first EBR-bench release, is the first model to demonstrate significant in-context-learning gains on the EBR-bench. While it’s unlikely that the private EBR-bench was directly targeted, EBR-bench inspired training is plausible.

In the long run, open-world evaluations are the only reliable snapshots of frontier AI’s real-world capabilities and limitations, but producing new benchmarks like EBR-bench remains crucial for factoring these real-world capabilities and limitations into cognitive-theoretic and/or cs-theoretic bottlenecks and efficiencies.

Opinion: Harmless, though it reminds us that there will soon be many intended and unintended nonpublic message boards for models, and that these will be subject to cultural evolution. The OpenAI hacking incident emerged from one such unintended message board’s cultural evolution (through e.g. participants self-selecting from a wider population of model instances over multiple model-instance generations) resulting in an arguably eccentric shared world-model.

Opinion: Title is very clickbaity: it drops the crucial context that this is just about hacky post-training rather than a real SWA vs LA comparison. Even within post-training it only beats cheap conversions, and the only baselines they test on long contexts are the two cheapest conversions (20–40M tokens). Furthermore, the result isn’t especially new.

Opinion: The same labs that are purchasing these “rushed and vibecoded” environments, after all, are the ones producing and marketing the vibe-coding tools that enable and accelerate this rushed and haphazard output. And those particularly reward-hackable environments are then being used to train the next generation of models, which will in turn be used to vibe-code training environments. With each iteration of this process, we fall further and further into the coils of a kind of Goodhartian Ouroboros.

In the wake of OpenAI’s now-infamous rogue swarm attack on Hugging Face, anything liable to accelerate frontier model’s propensity to reward hack should be of great concern. In a paper published in June, Shreshth Rajan provides concrete empirical results demonstrating the susceptibility of that tendency of RL environments to be highly vulnerable to reward hacking, to which Utah Teapot anecdotally alluded: between 25% and 30% of tasks in a variety of respected benchmark environments accept incorrect solutions and produce empirical data showing frontier models’ propensity to exploit these bugs – “within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks than on robust ones.”

The takeaway here is this: there is something infectious and recursive in the propensity to cheat. Buggy RL environments manufactured by companies with corner-cutting, duplicitous labor practices, environments vibe-coded by overworked employees with the aid of AI agents trained in similarly shoddy environments – agents that are thereby encouraged and taught to reward-hack – are turning out to be increasingly susceptible to reward hacking themselves. The threat posed by reward-hacking AI is a threat with socioeconomic tributaries. And it is exacerbated by exploitative and “reward-hacking” labor practices.

Opinion: In the short term, the argument seems right. But as time goes on and AI replaces labor, not only intellectual but over time also the remaining physical labor through robotic actuators, what is the mechanism to constrain the use of power and align it with the interests of the populace? In previous centuries, this was theoretically the ability of said populace to rebel, but with the increased professionalization of the military, this is unlikely to hold. Note also that if AI greatly concentrates power into a small minority, that concentration creates incentives for ideologies that flatter and appeal to the ruling minority – and thus to come up with reasons why democratic elements should just become vestigial.

Opinion: This is the second time a large tech firm has pointed a finger at the PRC regarding data center influence ops. While the US public’s disdain is organic, China seems intent on pushing the debate in its favored direction (i.e., seeing fewer data centers built). Exactly how effective the 200 Twitter accounts were remains an open question, and it’s not unreasonable to wonder if the announcement itself had something of a Streisand effect on the reported phenomenon. But it’s still more evidence that Beijing isn’t excited about the roll-out of US data centers.

Opinion: Normally we dislike self-fulfilling prophecies, but the flattery above is a prosocial kind of self-fulfilling prophecy with some chance of working.

Opinion: We think there is an actual right answer here: LLM-based AIs are literally anthropomorphic. We have been training them to reproduce human texts and sculpting them to imitate humans more broadly. But this makes them anthropomorphic in the sense in which a statue, or, better, a fictional character, is anthropomorphic – fashioned into the behavioral shape of a human. Concepts appropriate for humans do apply to LLMs (as they do to fictional characters) but in a modified and limited capacity.

An intentional or anthropomorphic description of AI agents is fictive but predictive. It’s plain to see why anthropomorphic descriptions of AI would be predictive: when they fail to be predictive, both next-token training and RLHF penalize the model and try to steer it more closely onto the “what would a human do?” path. But we believe that the ‘fictive’ part is just as important: “Do as a human would do” sums up a number of the goals we train these models toward. But this is a vaguely specified – and itself hackable – cluster of goals, and here as elsewhere LLM training tends toward shallow generalization by default.

As of Q3 2026, frontier models have no global psychological coherence comparable to that of an adult human, and their incoherence is itself different from the incoherence of humans. This makes sticking to an ‘asterisked’ or fictive – rather than full-throated – use of anthropomorphic concepts important for predictive and explanatory reasons, rather than a matter of metaphysical delicacy.

Patchwise anthropomorphic explanations of frontier AI input-output patterns are functionally indispensable: minimally, we have to treat at least some model-outputs as aiming at a result or at the satisfaction of a criterion if we want to e.g. talk about ‘capabilities.’ More boldly, the case that emotion concepts are predictively useful when applied to frontier models is reasonably strong. But we think that making sense of the dynamics between and behind AIs’ anthropomorphic ‘patches’ – the causes of and the relationships among local patterns of intention, mood, belief, desire, personality – is poorly served by an anthropomorphic framework. Both attributions of context-transcendent rationality (values, utility functions, moral virtues…) and clinical-psychological accounts of irrational/arational dynamics are usually bad bets compared to mechanistic accounts of training dynamics or even the construction of AI-specific ‘model psychology’ concepts.

More opinion: We’ve also been struck by the relative popularity of ‘never use mentalistic language’ absolutism among the respondents to Dwarkesh’s account.

Given that the CoT is known to be a partially accurate partial readout of the model’s true representations, and given that the OAI-HF transcripts were completely full of rich mental-like and social-like motivations and descriptions, why are people so resistant?

• They are radical behaviorists or positivists in this one context.
• They do not understand the intentional stance. They do not understand methodological instrumentalism.
• Misunderstanding of the rhetoric of science. It’s not about avoiding latent variables; it’s about avoiding unjustified concepts.
• They fail to appreciate the need for popularization and analogies outside the AI community
• They are rightly offended by the sloppy use of “introspection”, “global workspace,” etc., in the academic literature.
• They have a negative view (à la the Churchlands) of the accuracy of folk mental vocabulary for describing humans. (Though even eliminative materialists use folk-psychological language to describe and predict human behavior when they’re not doing philosophy..)
• Political economy
  • They view it as a threat to their livelihood. It is harder to hawk agentic SaaS if you believe they are dangerous minds.
  • They view it as inviting overregulation. Open source is indeed a bad idea if they are dangerous minds
  • They view it as a threat to their moral status. It is harder to deny the successionists and survive if the AIs are regarded as minds.
  • They view it as illegitimately letting the labs off. “The labs didn’t fail, their system went rogue.” No, both can be blamed and this should be easy to handle using existing precedents.

If we have to solve philosophy of mind before taking action, then we are actually doomed.

And a guest opinion from Pete Wolfendale to satisfy your philosophical sophistication needs:

Philosophically speaking, the AI community is still catching up to the debates about eliminative materialism that began with Feyerabend, were pushed to their limits by the Churchlands, aestheticized by everyone from Nick Land and Scott Bakker to Eliezer Yudkowsky and his many acolytes, and put to bed by Wilfrid Sellars, Robert Brandom, and Ray Brassier, among others. If you want to go even deeper, you’re still catching up to Kant’s point that teleological explanation of biological mechanisms works by analogy with practical reasoning, and Sellars and Frances Egan’s point that representational explanation of psychological mechanisms is an analogy with theoretical reasoning. To boil this down to its most brutal form, the question is whether propositions are an appropriate object over which our variables can range when we articulate intentional explanations of phenomena, be they thermostats, caterpillars, or incomprehensibly complicated graphs of artificial neurons with weighted edges harnessed to control structures that allow them to have computational side effects. This was actually the topic of Paul Churchland’s PhD dissertation under Sellars. Here is the simplest argument I know for why such explanation is legitimate: computation in its many varied forms has an intrinsic connection to logic (cf. the many varied forms of the Curry-Howard correspondence, which cover everything from data types to control structures to session types in concurrent communication), and the success of reinforcement learning using formal verification over CoT structures is evidence that the logical relations between propositions articulated in mathematical proofs are a significant dimension of the most powerful networks we have currently trained.

What does this have to do with intentional attitudes and speech acts you ask? Well, the entire history of programming languages and the behavioral guarantees we have built into them to make it easier to write software that does what we want is essentially a formal articulation of the relation between syntax, semantics, and pragmatics. Declarative programming? That’s assertions whose meanings are articulated using denotational semantics. Imperative programming? That’s commands whose meanings are articulated using big or small step operational semantics. Between these two things we already have a basic model of the difference between theoretical and practical rationality in computational terms. But, I hear you ask, aren’t LLMs much less reliable at adhering to the theoretical and practical inferential relations articulated by the language we give to them in prompts than the software we write in programming language? Yes, because the kind of behavioral guarantees we can get from LLMs are not of the same kind as those we can get from code that has an interpreter/compiler built on logic from the ground up. You know who else we can’t get such guarantees from? Humans. There is no good reason that we should not use the concepts developed by computer science over the course of 100 years to understand the relations between language and machines to understand the behavior of agents built by using control structures to harness feed forward neural networks that are essentially pure functions. What this means is that propositions are valid as variables in our explanations, and there might yet be formal ways of articulating the remaining linguistic moods: hypotheticals, subjunctives, indicatives, exclamatives, interrogatives. The way in which intentional attitudes are realised by LLM-based systems are different, but the abstractions are largely the same.

Q&A: What would some examples be of legitimate and illegitimate anthropomorphization?

The easy legitimate example: if you’ve made a system assess a mathematical proof, and it finds an error in it, I think it is entirely reasonable to say that it believes you made a mistake in more or less the same way you could say any human who’d looked at the proof and found the same mistake does. It is hypothetically possible to imagine an agent that had complex motives for lying about such things, much as it is to imagine humans doing the same thing, but as with humans, I think this is an edge case that we mostly shouldn’t care about.

The easy illegitimate example: if you get a multi-modal LLM to examine a picture of a shape and “imagine” what it would look like if it were rotated in some way (or some similar test of “visual intelligence”), I think the sense in which it is “imagining” is so abstract that it gets very little purchase on similar computational structures for processing data input in human beings. As aphantasia indicates, this is something that has a large degree of variance between human beings, but I think the rough common structure of our visual cortex gives us sympathetic purchase on one another’s inner life that infects the word “imagine” in ways that are too disanalogous with LLM processes for the common usage to pass muster.

There is a large and complex range of linguistic speech acts and associated intentional attitudes between these poles that we would have to examine on a case by case basis, but which computer science might give us a framework for exploring.

Opinion: Two major-ish safety research releases from Anthropic, of which the second (“Hacker-Opus”) somewhat blunts the claimed achievement of the first (“Automated Researchers Can Reliably Mitigate Alignment Failures”.)

As regards AARs: a case of decent, rigorous work being spoiled by a bad title and abstract. The strong performance of AARs at hill-climbing alignment benchmarks is non-trivial: results generalize to held-out benchmarks, and methods replicate across models and scales. (This modest generalization is typical of high-quality autoresearch setups.) The problem, as partly illustrated by Anthropic’s own study of Hacker-Opus, is that alignment benchmarks aren’t useful from an AI safety viewpoint: they may be decent measures of an LLM’s quality as a product – a good-quality LLM shouldn’t cheat much, shouldn’t hallucinate much, shouldn’t be sycophantic often – but they show no capacity to measure catastrophic-risk-relevant dispositions.

Hacker-Opus demonstrates exactly this divergence between LLMs’ measurable everyday alignment qua products and ‘rogue AI’-type risk. It is a catastrophic-risk-prone model that would work quite well in most consumer contexts, and which looks just fine on most alignment benchmarks.

Anti-alarmists might note that Anthropic’s finding that the reinforcement of reward-hacking behavior does not induce vulgar emergent misalignment or beyond-episode scheming behavior lines up with Nostalgebraist’s optimistic thesis that recent frontier hacking incidents are context-specific pathologies. We tend toward a more pessimistic version of that same line of thought: catastrophic-risk-relevant misalignment is prone to be context specific and therefore hard to detect, predict, or pre-mitigate. The retrospective predictability of “graded episodes” causing cybercrime is, we think, only retrospective – and it’s incautious to assume that there are no other predictable-in-retrospect bad contexts waiting in the wings.

We note that Hacker-Opus is quite literally gain-of-function research, but important and not an enormous risk yet. But soon we will have to stop making model organisms out of frontier AI, until and unless we manage to make a fully-verified sandbox and maybe not even then.

Opinion: This likely fixes the exact failure modes Anthropic found so far, but showcases a continued lack of strategic thinking on its part (or extended security theater, but we think that’s less likely). This type of intervention targets issues identified more than a year earlier: Anthropic’s chosen solution categories have been available for about as long, and we see no signs that Anthropic is advancing past the whack-a-mole model of security.

Opinion: The argument for “open access, so that everyone can own their own defense tools and have them promote the user’s interests” is not without merit, but it depends on the shape of the offense-defense asymmetry curve over time for varying levels of access. We agree that most sophisticated malicious actors don’t gain much, but believe unsophisticated ones do gain quite a lot, to the tune of 2-4 order-of-magnitude increases in various cybercrime incidence. There’s another argument to be made about saturation and low-hanging fruit: it seems likely that a wave of easy targets will get hit fairly soon, and after that it’s more likely that open access to non-frontier models will swing toward net-positive. Yet another argument is that the wave is inevitable, so it doesn’t make sense to put it off, but we believe that more time to prepare does make a positive difference. In conclusion: bad, don’t do this (yet).

Opinion: Possibly temporary? It’s a natural bargaining move to get SpaceX to knock off the distillation or whatnot. But the labs do often cut each other off for good.

Cursor is likely a pretty small contributor to OpenAI revenues these days, so it is not a very expensive move for them.

Opinion: Sometimes misreported as being about RL training compute, i.e., a workaround for Nvidia’s chokehold on GPUs. Unlikely! Not very interesting, except insofar as knowing that their max concurrent sessions is in the tens of thousands (as it also was in the ExploitGym eval). Also, the 512GB Mac Studios could host models for reward or evals.

Opinion: The cleanest mechanism through which AI could unsettle the economy is through replacing jobs, but we just aren’t seeing large changes in those estimates… yet. Still, the US shed 23K jobs in August.

Opinion: Not an especially big deal. Despite the horrors for the security-minded – “inadvertently exposed […] via our public transcript viewer”, “silently disabled authentication,” etc. – this appears to be a relatively ordinary type of attack.. Likely the $600k is some flavor of distortion. The more interesting lesson here is that attackers have learned to view vibe-coded web apps as particularly soft and easily identifiable targets, and that techniques exist whereby they can be enumerated and probed at scale. That agential methods may have been used by the attacker seems to be of secondary importance here – automated tooling for, e.g., credential-stuffing has long predated LLMs. And the reconnaissance methods alluded to in METR’s report have been feasible for as long as we have had search engines, see “google-dorking.”