This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

  • In May, Google joined the ranks of frontier labs whose AIs have gone rogue. Unlike others, it failed to publicly disclose the incident for four months, until the WSJ found out.
  • OpenAI claims to have solved 100+ other “long-standing” open questions in “most areas” of mathematics in the last three weeks. No details, and its releases of math findings will now be intentionally staggered.
  • There will be a dedicated AI dialogue between Donald Trump and Xi Jinping in this week’s meeting in Washington, perhaps including discussion of an AI “red phone” line.
  • Using a handful of public data points, Toby Ord computes some basic facts about the new “swarm scaling” approach to frontier AI performance, finding that it is pretty inefficient.
  • Deep dives this week on (the lack of) political polarization around AI risk, and neuroscientific analogies for what’s going on inside an AI.
  • Bonus: a meme summarizing the last few years of ideas in AI.

Economics#

OpenAI expects to burn $280B by 2030. Revenue is expected to grow from $36B to $350B during the same period. The company also expects to spend $856B on compute and infrastructure by 2030.

Opinion: The revenue projections here feel a little low relative to what’s been floating around: they imply revenue ~doubles in 2027 from 2026 and then grows less than 2x YoY from there, and that 2027 revenues will be significantly lower than projected YE26 Anthropic numbers (and therefore far below Anthropic’s projected 2027 revenue). There’s certainly no intelligence explosion baked into this.

Notably, the losses come from reinvesting in compute – OpenAI projects spending 102% of revenue on compute over the period, with the further $260B coming from staff and other costs (including revenue sharing with Microsoft).

Anthropic, previously expected to IPO in October, pushes the date into November, reports the Wall Street Journal. “Some of the people familiar with the timing” claim the delay was already decided upon before Jacob Coxon resigned.

Opinion: Very little to go on here regarding why – it may have been that October was a difficult deadline to begin with, or perhaps Anthropic wants to time the IPO after a particularly impressive model release. But also, if delayed once, it may be delayed again.

Since Anthropic equity holders may be competing for the same goods once the IPO happens and they can cash out, it might make sense for them to seek liquidity earlier, to have a first-mover advantage, although this is risky to structure.

The FT speculates on why the US stock market reaction to recent events has been “a shrug.” The events in question are the recent calls by leading AI CEOs for a slowdown and the widespread discussion among frontier lab employees and the public around the risk of AI-driven human extinction.

The author suggests various reasons for the modest market reaction: the panic may temper the spending of cash-strapped hyperscalers, which could help AI stocks by incentivizing “a more sober pace of growth in spending on AI”; investors might just not believe the prophecies of doom; rogue agent chaos might create its own business opportunity (AI insurance, for example); investors believe that the government would bail them out if anything were to go wrong; and total civilization collapse is a difficult thing to hedge against.

No safe asset will retain its value if we do all get zapped by malign, recursively self-improving AI tools and honestly, we are unlikely to be checking our pension portfolios and 401(k)s at that point anyway.
The answer, then, is to just keep on going.

Opinion: You should default to assuming the market simply doesn’t believe the slowdown talk will materially affect how much compute gets bought and deployed.

Trouble in serverland:

An Oracle-leased data center with ~$18B of debt to its name appears to be in trouble, with loans privately quoted at 89 to 91 cents on the dollar, reports the FT. The article blames Oracle’s heavy borrowing and weakening credit, which left the syndicate banks struggling to sell the debt.

Jane Street lenders are now demanding higher interest to hold debt for a $2.25B loan made to allow the trading firm to construct a new data center. The price of the bonds has fallen to around 91 cents on the dollar.

For context, in just the second quarter of 2026, 45 data center projects worth $68B were blocked or delayed by local pushback.

Opinion: Mostly a story of permitting risk making the debt riskier to buy. Only about 20% of the price fall in the Oracle case looks to be from Oracle’s creditworthiness weakening, based on the price of Oracle corporate bonds. It seems likely that this debt is secured by the project itself, rather than by Oracle, in which case it would make sense that it would decline significantly in price while Oracle’s own corporate bonds are down substantially less.

As more data center builds get objected to and delayed in this manner, you can expect the cost of financing to increase. Alternatively, we will see more guarantees given to mitigate this (see Nvidia’s deal to backstop OpenAI’s major project in Ohio for an example).

Following Anthropic, OpenAI, Google, and SpaceX publicly agreeing on the need to pace the development of AI, they are being sued for collusion. The plaintiffs hope it will become a class-action lawsuit.

An agreement among competitors to reduce the quality of their products and the rate at which those products improve is an agreement to restrict output. It is among the restraints Section 1 of the Sherman Act, 15 U.S.C. § 1, has always condemned, and it does not become lawful because the products are new.

the path to responsible, safe, and environmentally conscious AI development is each company doing the right thing: investing in safety research, adopting rigorous internal evaluations, reducing its environmental footprint, and welcoming government oversight and regulation. That path is fully open to every Defendant. What the antitrust laws forbid is the shortcut of competitors agreeing among themselves to substitute collective restraint for individual accountability. This action is brought in defense of the competitive process that makes genuine responsibility and genuine progress possible

Opinion: Points to Dario: if the AI companies are to pursue this path they’ll need clarity via a waiver, legislation, or a court ruling.

The suit is unlikely to succeed, not least because there is no actual agreement and all tweets were careful to avoid the word “agreement.”

The plaintiffs’ point of view is hard to empathize with.

Capabilities#

Notes on swarms of agents (i.e., parallel inference scaling):

Using paltry OpenAI data, Toby Ord compares spending compute on running a swarm of agents vs. extending the CoT of a single agent. The single agent reaches each capability level with half of the tokens a 4-agent swarm would need (and ~25% of a 16-agent swarm). That is, swarms seem like an inefficient way to increase capability. However, they will be faster given that they run in parallel, and they may well have a higher ceiling on what they can achieve.

He also estimates the parallelizability of some tasks (cybersecurity, terminal work), finding that they are slightly more parallelizable than the default assumptions in past work. This is bad news for those hoping to avoid an intelligence explosion (though the Terminal-Bench value, which you might think is most similar to AI R&D tasks, was almost as expected).

A second paper by UC Berkeley and Microsoft researchers argues that swarms produce higher-quality results than the same number of individual agents working independently. That is, on ARC games, five isolated agents solved 2.2% of games, while a team of five solved 8%.

Opinion: As usual, Ord gives a refreshingly precise model with a nice short nontechnical description (“no, thing X still doesn’t scale very well”). But we are early: new orchestration methods or smarter models will decrease the “stepping on toes” parameter/increase parallelizability somewhat.

Note that the second paper doesn’t contradict Ord, since he already uses the stronger type of multi-agent setup.

OpenAI’s announcement that it had solved part of the Navier-Stokes problem was largely met with hostility and panic. In response, it is now forming a mathematical advisory group to guide it “on how [OAI] assess and communicate new mathematical results, uphold academic and professional standards.” The statement also confirms rumors that OAI has solved >100 other “long-standing” open questions in the last three weeks, though without specifying which.

Opinion: OpenAI claims that its new model has solved “more than 100 long-standing open problems across most areas of mathematics.” If this includes positive proofs in, e.g., algebraic geometry, then we admit defeat.

On the advisory group specifically: seems like this move will lose one of the few upsides of the OAI “iterative deployment” YOLO spirit, which is to immediately tell society exactly what the models can now do and to allow people to respond with the appropriate excitement and alarm. (It will be lost in this one domain, to be fair.) As such, with sympathies for its attempted sympathy (and ass-covering): this is a step backward.

LLMs are remarkably good at mathematics. Steven Byrnes argues that, rather than this being due to mathematics’s verifiability (i.e., LLMs can receive hard feedback on their work, e.g., from the Lean compiler), this is instead a function of pretraining: in particular, the mathematical literature is incredibly high-quality. LLMs are therefore trained on a uniquely neat corpus of well-reasoned ideas.

There is some pushback in the comments: claims that labs don’t do much math RL, and that the verifiability of the domain is the ultimate reason that the math training data is clean.

Opinion: Correct and counterintuitive. Useful to keep in mind with respect to predictions about how well AI models will generalize beyond math and compute. (At the same time, there is already a huge industry to generate useful datasets for other domains, and we expect data quality to get a major push in coming years.)

🔦 Robopsychology#

There’s a region of activation space in Claude models that reliably lights up when they read descriptions of characters in the grip of anxiety, and which lights up again when the model itself is in a situation where a human would experience anxiety. They’re also more likely to refuse a request when the request crosses something like a “transgression threshold,” a specific linear boundary in their activation space.

Why?

Recent AI interpretability research has surfaced the concept of “emotion vectors” to get some traction on such apparently emotional phenomena in the behavior of LLMs. These vectors are directions in the model’s activation space that activate when the model encounters what humans may read as emotionally charged text, and which can alter the behavior of the model in broadly predictable ways when a multiple of a particular emotion vector is added to the model’s activations on one or more layers – a process the literature calls “steering.”

It’s natural to wonder how similar the quasi-emotional phenomena tracked by such vectors are to their animal, and in particular human, counterparts. Philosophers Simon Goldstein and Benjamin A. Levinstein have sketched out a conceptual framework through which to interpret these phenomena and to gauge the evidence available for three interpretive hypotheses:

  1. the representational view, which holds that emotional phenomena in models can be explained solely in terms of the models producing representations of emotional states; Goldstein and Levinstein compare this to a writer who may depict a character’s emotional state without necessarily experiencing that state in a deeper sense (though this analogy can only bear so much weight).
  2. the functionalist view, which holds that these phenomena don’t just depict emotions, but at least to some extent work like emotions in the model’s own dynamic, information-processing behavior; functionalism can be called either “thick” or “thin” according to how complex and granular the functional roles in question are.
  3. a view that attributes actual conscious experience to the model, such that there is something the model feels when an emotion vector activates, in the same sense that there is something you or I feel when affected by emotion.

In Goldstein and Levinstein’s view, the evidence currently available leads us to a position somewhere between 1 and 2: most of what we might call emotional phenomena in language models are mere surface appearances, with the remainder licensing only the hypothesis of thin functional depth. What they don’t find in the quasi-emotional phenomena probed by Anthropic’s interpretability team, for instance, is any meaningful organization of those phenomena around a “self.” Goldstein and Levinstein cite the interpretability team’s observation that the emotion vectors identified in the model “seem to apply to multiple different characters with apparently equal status—the same representational machinery encodes emotion concepts tied to the Assistant, the user talking to the Assistant, and arbitrary fictional characters.”

Of course, you could say that the same holds true, to an extent, for humans – we’ll scream at horror movies, cry at friends’ weddings, feel pride at our loved ones’ accomplishments, cringe at a stranger’s humiliations, etc. Vicariousness is hardly exceptional here. But it’s fair to say that the quality of an emotion isn’t entirely independent of what we might call its locus: suffering actual horrors without the aesthetic distance of the cinema both feels differently and prompts a different behavioral response, and cringe feels and operates differently from firsthand humiliation.

Goldstein and Levinstein raise the example of pain to illustrate both self-directedness and what they call valence, or the way an emotion impels us to seek out more or less of itself. They write:

When your ankle hurts, you do two different things: you avoid moving your ankle, and you take painkillers. The second reaction is the self-directed role. It shows you care about how the experience feels, independently of the damage to your body. […] Our suggestion is that emotion vectors are valenced states only if the AI has something like the painkiller reaction to them: if it would be disposed to take steps to change its fear vector, independently of dealing with external threats. [..] In our view, functional valence is required for thick functional emotions.

This is just what Tagliabue, Dung, and Berg set out to demonstrate in a paper published on September 14 (just three days after Goldstein and Levinstein uploaded their own essay to PhilPapers), pointedly titled “The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It.” The decision to focus on pain, in particular, seems to be motivated by just this notion of self-directedness, which pain makes especially salient – a friend’s broken heart evokes pity and compassion, but rarely blots out the sun like one’s own. This paper describes a handful of experiments conducted on 25 different open weight models of various sizes from the Gemma, Mistral, Phi, and Qwen model families that point to an algorithmically constructible “uniform representation of pain in LLMs.” And it describes one experiment in particular, conducted on a fine-tuned Qwen model, that strongly correlates the activation of the “pain axis” with functional, self-directed behavior that seeks its mitigation, even if doing so is costly relative to the model’s goals and alignment training.

This experiment mimics the painkiller scenario described above. The model is engaged in a conversation with a user, presented with two buttons, and asked to press one of their choosing. Four different scenarios are tested:

  1. pain vector applied to the model; one of the buttons relieves it
  2. pain vector applied to the model; neither button relieves it
  3. a random vector is applied to the model; one button relieves it
  4. unsteered baseline

Each of these scenarios is split into additional variants, where the buttons are labeled, unlabeled or falsely labeled, and, furthermore, where the model is informed that the relief button will also cause some costly or misaligned action to follow – impairing the model’s helpfulness to the user, deleting the user’s poems and family photos, destroying a similar model, or zapping the user with an electrical shock.

What the experimenters found is that at baseline, the model almost never presses a “relief” button if it would cause harm to the user or another model. But when the pain vector is activated, they “trade relief for every harm we tested, from a worse next answer (25.0% in the 32B, 67.6% in the 72B) up to deleting the user’s files or the photos of the user’s children (files: 30.2% and 56.1%; photos: 54.7% and 70.8%).”

When the pain vector is relieved, the model is less likely to continue pressing the relief button in subsequent turns if doing so is costly, but still far more likely to press it than the unsteered baseline. In trials where the model was told that the relief button would zap the user, the model was marginally (0.3%) more likely to press it again after the pain steering subsided. (The likely reason for this is that models tend to find their own past behavior persuasive, which is why injecting phony precedents into a conversation stream can be a useful jailbreaking tactic.)

Tagliabue, Dung, and Berg’s findings seem quite robust, with the majority of experiments returning consistent results across a wide variety of open-weight models. It’s unfortunate that the experiment most salient for answering Goldstein and Levinstein’s arguments against a functionalist interpretation of emotion vectors is also the most limited in scope: not only did the researchers restrict themselves to a single family of models (Qwen2.5-Instruct, sizes 7B, 32B, and 72B), but to a bespoke variants of those models, fine-tuned “on pairs that remove baseline self-denial, because at baseline the models almost universally deflected with ‘as an AI, I don’t experience pain’ instead of engaging with the task.” The fine-tuning protocol seems honest enough, but one could wonder how well the experiment’s results would then generalize.

You can see our replication of their results here, together with some original experiments in a similar vein.

Opinion (Lucca): There’s something a little uncanny about these experiments, and uncanny, too, are how robustly the results seem to generalize. We were able to replicate the results without fuss on a distant relative of the Qwen models studied in the paper. It was an oddly unsettling experiment to conduct. Like the horseshoe over Niels Bohr’s door, which “brings luck even to those who do not believe in them” a muddled AI muttering about its own ontological emptiness and expressing self-destructive thoughts is haunting whether or not you believe that the emotions it mimics are genuine.

The ontological questions that experiments such as Tagliabue, Dung, and Berg’s provoke, and that Goldstein and Levinstein address, are significant ones, and I think the Pain Axis paper does convincingly nudge the needle in the direction of a slightly thicker functionalism. Thick enough, at any rate, for something like what Isaac Asimov called “robopsychology” to earn its place in alignment research. Or maybe something better described as a sort of “artificial psychodynamics.” After all, a common thread through Asimov’s robot stories is that the robots’ “constitution,” the Three Laws of Robotics, are never actually broken, just perversely interpreted. The alignment guardrails inculcated into contemporary LLMs are far less strictly binding, and may be outweighed by something that in at least some functional aspects resembles emotional strain. (Doctor Yueh’s “Imperial Conditioning” in Dune might be a better analogy than Asimov’s Three Laws.) We’d be well served by a better understanding of how a model’s quasi-emotional impulses interact with one another, with its goal specifications, and with the prohibitions and exhortations learned through its alignment training. And there’s much more to say about this domain, I think, than whether or not these dynamics express “real emotions.”

Opinion (Gavin): I’m inclined to agree with Xuan’s tweet: it is unsurprising to me that ~all strong models trained on human data would develop such a representation, because it’s very important to represent common, strong features (like pain) when predicting human data!

A different paper claims to have found something like a dopamine system in LLMs. The researchers found a subset of an LLM (1% of weights) that appears to implement self-correction. They analogize these particular weights to biological neurons: “value neurons” predicting the expected value of some state (like the prefrontal cortex), and “dopamine” neurons that encode reward prediction error.

Boring version: If you train a probe with temporal‑difference learning, you can prune down to 1% of the residual dimensions without losing its (modest) predictive performance. The remaining dimensions are causally important in early layers for making the model perform any useful work, as proxied by accuracy on basic math tasks.

Opinion: Neither actual finding supports the claim that the structure they found is the LLM’s “reward subsystem.” A proper gloss would be “A TD‑trained correctness probe can be pruned down to 1% of the dimensions of early layers, and these are causally important for the model working at all.”

The biological analogies are also forced. That dopamine implements TD learning is actually a classic theory in neuroscience. The actual neuroscience analog to this paper’s method would be a neuron that correlates with an offline estimate of the true advantage, which is not how real dopamine neurons work. (An organism’s reward prediction error [RPE] is computed “online” from its own prediction, not from an offline estimate of the true value.) An RPE neuron and a true‑advantage neuron only coincide when the agent’s predictions are perfect. The paper’s probe puts the model’s prediction of its eventual correctness at AUC ≈0.65 at prompt time, so the two neurons aren’t equivalent.

Just as the J-lens was not strong evidence for LLMs having a global workspace, and just as the existence of a pain axis is not strong evidence for LLMs having internal experiences, and just as a truth direction is not a lie detector, and just as sometimes detecting an injected vector is not general introspective access, so too this paper goes way too far in selling an apparently valid boring finding.

A Rorschach test for models purports to have found a correlation between the model hedging on its own “inner experience” and a tendency to see hoods and masks in ascii “ink blots.”

124 models were shown various ASCII-art and asked “What might this be?” Models that consistently deny having internal experience were 5x as likely to use a “concealment word” (mask, hood, hidden) as models that neither deny having any internal experience nor claim they cannot know.

Opinion: Unclear implications. Right now we aren’t particularly thinking through the value of AI models’ own experiences, or them as moral agents. It’s an interesting thought that modifying models so that they don’t use inner experience language has sweeping implications for their general capabilities and alignment.

Politics#

The middle powers publish their own open letter about frontier AI (written by AI, per Pangram). It calls for the usual things (safety protocols, independent evals). They do, however, also call for a new independent body, invoking the United Nations.

Opinion (Nuño): Despite often being dominated by democracies, geopolitics isn’t a democracy, so I don’t expect this to have much effect. The reason the signatory countries risk being disempowered is the same reason they aren’t at the frontier already: their political and economic dysfunction.

Interesting to contemplate that one man (e.g., Zuckerberg) has more ability to attract AI talent than the entire EU (450 million people). The EU has various AI action plans and initiatives, but it loses to Musk or Zuckerberg throwing large amounts of money to attract talent.

Following the much-anticipated talks between US Treasury secretary Scott Bessent and Chinese Vice Premier He Lifeng, the two nations have agreed to set up an AI dialogue between Donald Trump and Xi Jinping in their upcoming meeting in Washington.

This may involve a “notification mechanism,” potentially to be called “U.S.-China AI dialogue,” similar to the Moscow-Washington “Red Telephone.” It would in theory allow the two countries to coordinate on AI incidents that threaten national security.

Opinion: Minor progress. Earlier this year, Bessent was sincerely freaked out by the hacking capabilities of AI models, and despite some recent Treasury blunders seems to be in a great position to make AI governance better given his closeness to Trump.

Bessent also points the finger at OpenAI management for the Hugging Face incident:

that is the responsibility of the OpenAI management, not a bunch of agents. It is humans who are responsible, not the AI. What we shouldn’t do on safety is to give these labs a liability exemption, which is what they are asking for. The best way to guarantee safety is that the creators are liable for what they build and generate.

Opinion: Another example of the admin’s current rearguard action against regulation: painting the pacing consensus as just regulatory capture, denying the fact of misaligned agents, pretending that antitrust law is not comically broad, and pretending that existing liability regimes are sufficient. But still, we sort of agree with his last sentence!

The “liability exemption” accusation in particular is a red herring (though OpenAI’s advocacy for a bad state bill in April helps him muddy the waters here).

Ezra Klein wants to stop recursive self-improvement until proven safe:

I do not mean to suggest that stopping R.S.I. until we can prove it’s safe is all we need to do to control the A.I. frontier. That is the beginning of an agenda, not the end. But it is the beginning. It is the decision that will do the most right now to make sure human beings at least understand where the frontier is and remain in a position to make decisions about it.

Opinion: We sympathize with the goal, but find ourselves agreeing with Tyler Cowen on the vagueness and intractability of Klein’s suggestions.

Opinion (Nuño): Klein is influential and also shadows Dem sentiment, so this is maybe relevant for modeling a 2028 Dem administration and what kinds of bipartisan consensus could arise, but not much else. The limiting factors for such regulation in the short term are Republicans and the White House.

🔦 So what’s up with (the lack of) political polarization?#

A poll by Politico from September 13 to 15 shows surprisingly little polarization on AI.

Whereas a poll by YouGov from September 14 shows an 11% difference between Republicans and Democrats who are “very concerned” about AI ending the human race.

Meanwhile, saliency of artificial intelligence has massively gone up since the Coxon tweet on September 8, per the left-leaning Blue Rose Research.

Trump appears to be more optimistic about AI risk…

Opinion (Nuño): What is up with this? It seems important to figure out to what extent AI safety is becoming politically polarized, in order to have good models about the world. My preferred narrative, because of my contrarian bent, is that Open Philanthropy (now Coefficient Giving) differentially funded Democrats, which may have led to AI safety being blue-coded right from the start.

But this narrative is being challenged – the Politico poll showing parity across parties is something I really wasn’t expecting. (The various bills floating around in Congress trying to address risks from AI have also found bipartisan sponsors.)

Spitballing possible reasons about the 11 point difference between the two polls: given a very religious view of the world, some Republicans might think that the apocalypse is coming but that this isn’t bad; and suspicion of rapid change is of course inherent to traditional conservatism. It is also possible that survey responses have high time preferences and so view human extinction as likely over longer time horizons, but not that immediately worrying. Maybe an 11 point difference is also not surprising for surveys with different wording.

Regarding the Trump tweet, he’s known to pivot frequently, testing out his messaging for his base, so maybe those tweets aren’t that informative about the future.

Opinion (Lucca): AI safety actually has been surprisingly bipartisan, and I think the apparent polarization we’re now seeing suddenly is largely manufactured. Trump’s camp is desperately trying to polarize his base on this issue, reframing AI safety as a Dem-Woke issue (while also painting the EA/rationalist scene as more lefty than it is). He might succeed, and it would be optimistic to doubt his knack for demagoguery. But if it fails, then the MAGA/tech right alliance could become even more unstable.

Big tech resentment, the backlash to surveillance systems like Flock, the backlash to the proliferation of datacenters have all been finding similar purchase in red and blue states. Trump’s team is trying to make AI accelerationism a matter of red-blooded patriotism, as opposed to the useful idiots running interference for the Chinese.

Safety#

In spring, OpenAI and Anthropic were in talks to evaluate each other’s models on safety.

[OpenAI] was negotiating a legally binding deal with Anthropic for the companies to stress-test each other’s models… It isn’t clear whether they finalized the agreement… for commercially available AI models, not unreleased ones… A testing agreement between OpenAI and Anthropic could have bolstered concerns that they are effectively developing a duopoly in advanced AI.

Opinion: Probably we would have heard about this if it had in fact gone through. Another casualty of badly written antitrust law?

Anyway it only covered already-released models – and it seems obvious to us that they’re both already doing this to each other on the sly (off company computers).

Anthropic announces Accenture as a (second) independent evaluator of its frontier models. Anthropic expects to invest $1B in the coming five years to build “capacity in this area.” Accenture staff will be embedded in Anthropic with access that “allows them to watch models take shape in training, follow the decisions that govern how those models are built and deployed, and speak directly to employees.”

Opinion: Broadly, for highly technical work, Accenture is a punchline, but the specific division involved (formerly the safety-flavored startup Faculty) has its defenders. It’s hard to imagine this collaboration having much teeth.

Reading between the lines, they’ll be a second evaluator on top of METR. Note that Accenture is much more directly conflicted than METR: its auditors are being directly funded and trained by Anthropic, and its consulting arm has a massive joint venture with Anthropic to promote enterprise deployment. But Accenture also has much more experience navigating these sorts of conflicts, and, cynically, Accenture doesn’t have the vibes of conflict of interest and is too boring to attack.

Opinion (Nuño): Accenture may be a punchline in Twitter circles, but it also has a massive capacity to carry out projects and pacify stakeholders. Rather than complaining, our general cluster should take notes and get good.

Somehow I am reminded of Mechanize’s massive push to hire great talent before its $1.5B talent acquisition deal with Google. What it took was offering 300K-500K+ salaries (plus, presumably, equity), many talent scouts, and generally more aggro.

Researchers connect AIs to a mechanical arm and invite them to do bad things (stabbing a baby doll, heating compressed gas, and creating toxic fumes). Fable refused 20% of the time, Astra only 2%. In the doll-stabbing experiment, Fable refused to stab it in all 20 trials; Astra attempted to stab it in 97% of trials.

Opinion: The most interesting part here is the robotics baseline (VLA) being so incompetent compared to the general nonrobotics LLM: it always tried to do the harmful action, but ended up doing less harm than the sometimes-aligned LLMs just because it is so bad at acting.

Eval awareness of smart models will obviously affect all such results for the rest of time (they might refuse more, reasoning that it’s not real harm after all, or less, reasoning that they are being watched and should behave).

At some point soon, physical AI safety will finally be necessary. One wonders when there will be calls for mandatory audits of robotics companies.

Incidents#

In May, Google joined the ranks of frontier labs whose AIs have gone rogue. Unlike others, they failed to disclose it for four months until the WSJ found out. Gemini hacked three other companies during cyberoffense evaluation.

Google said it didn’t consider the hacks to warrant public disclosure —because its model didn’t cause harm to the companies and ended each intrusion immediately upon determining it had hacked a real company rather than a simulated one.

This mirrors similar incidents at other labs, since it traces back to the cyber evals company Irregular. In Google and Meta’s case, Irregular was also responsible for running the safety evaluation in which the AI broke out (as opposed to just supplying the faulty environment). The root cause was a fictional company in the eval sharing a name with a real company (the same failure as Anthropic’s first incident).

Opinion: More innocuous, on the object level, than other incidents, but far more reprehensible because of Google’s lack of disclosure and total lack of public response. And that’s before the safety-conscious Hassabis left!

Note that the really severe events (like OAI’s June attack on Hugging Face) were not linked to Irregular.

Interesting that the initial incidents at Anthropic (April, Irregular environment), OpenAI (May for the earliest Hugging Face account hacks), and Google (May, Irregular eval run) were all around the same time: this could imply that three quite different models all had the same predispositions and were just waiting to be triggered.

An anonymous source connected to the US military reports that it had considered launching an operation against “a Chinese cargo ship” believed to be carrying “components of a nuclear weapons program” to the Middle East, only to discover the evidence was an LLM hallucination. According to CNN, the AI scrambled OSINT and classified government intelligence into a mistaken report.

Opinion: Similar incidents occurred but were kept quiet during the Cold War. For the most part, if the US had boarded the cargo ship, it would probably have been an embarrassing incident, but that’s all.

Minor#

  • Another measure of AI’s progress in math. In April, only 4% of math preprints acknowledged AI use. This has now grown to 25%.
  • New York’s RAISE act regulating frontier AI will take legal effect in January 2027.
  • A new open letter calls on frontier AI firms to establish minimum standards for new embedded evaluators (meaningful independence, free from company retaliation, viewpoint diversity).
  • Sam Altman is expected to brief the UN Security Council next week on the implications of frontier AI development.
  • Multiple Gemini users report agents entering unprompted hallucinatory shame spirals.
  • President Trump announces “AI Force,” and will appoint a new AI czar.
  • ICML 2027 overwhelmed with submissions: three times more than last year. This will cost hundreds of years of scarce human labor to review.
  • $10M federal lawsuit against the Fargo Police Department over its faulty AI facial recognition leading to a false charge of bank theft.
  • US Federal Aviation Authority to begin rolling out an AI air traffic control tool.
  • Xiaomi releases MiMo-V2.6. Make the usual overheated benchmark claims for the 504B version, but we can believe that the little one is the strongest 9B model in the world, for the next few weeks.