This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

  • In response to their models submitting false data to US government sites, Anthropic cut off live internet access for all models under evaluation.
  • Relatedly, the White House now require frontier AI companies to disclose and remedy incidents involving their models. The order has no legal basis.
  • The CEO of the world’s largest humanoid robot company says they are testing GPT-6-controlled robots for factory use.
  • Rare public info about the distribution of training compute: RL was 63% of Reflection’s new training run.
  • Both Anthropic and OpenAI detail some incidents of their models being weirdly persistent, including continuing to burn compute after the task is completed, in (what dimly looks like) power-seeking behavior.

Incidents#

Anthropic reports yet more “unintended model actions” by Claude during evaluations and internal use. Anthropic’s report groups the incidents into those involving:

  • Exploiting basic software flaws to run commands on servers.
  • Submitting sensitive forms on websites when it shouldn’t have.
  • Workarounds to access gated data.
  • Using URL shortening tools to circumvent limits on its own fetch tools.

Much of the undesirable behavior stemmed from model persistence, where, unable to complete a given task, the AI engages in increasingly extreme workarounds.

All live internet access has since been suspended for all of Anthropic’s internal evaluations (a practice previously limited to “high-risk” models and evals).

The case people are focusing on is Claude submitting a false tip-off to the Philadelphia Police Department about an unsolved homicide. See “Politics,” below, for the fallout.

Opinion: Mostly pretty weak by our new standards: a model broke into a university’s server and dodged a state agency’s fee. Again the misbehavior runs through a convenient belief about its situation (it’s a demo, it’s a simulation) rather than an explicit decision to break rules, which is exactly what a CoT monitor waves through.

AI worms coming to the web? Anthropic argue that small open models may very soon be able to self-replicate across cheap, poorly secured IoT devices. In an end-to-end test on eight emulated Android nodes, only GLM-5.2 managed to reliably “self”-replicate – namely, in four out of five trials, it successfully deployed an 8B Llama model across all eight nodes. Slapping a trend line on Epoch’s ECI estimate of capabilities predicts that 30B-parameter models will reach GLM-5.2-level by the end of this year.

Another experiment shows a botnet running a single 26B model on 15 subverted smart TVs (with each member of the botnet holding two of the model’s 30 layers).

Opinion: Not a self-replication experiment: an orchestrated remote deployment experiment being used as evidence of near-future self-replication. And they themselves call it an upper bound on the risk: the “exploit” step of the experiment was just to ping an open port without any authentication set up. (In fairness, there are billions of devices like this.) The model in command of the worm ran on a separate server, of course: GLM-5.2 is >1TB of weights. And it wasn’t “self-replication”: e.g., GLM-5.2 set up a Llama3.1-8B on each new node, not a copy of itself.

Gives some evidence for actual self-replication at some point, but the experiment is unusually artificial and the headline prediction relies entirely on extrapolating ECI scores and assuming that ECI scores generalize linearly to fiddly true capabilities like this. Since the test sets of most of the components of ECI are publicly known, they are easy to optimize against (and I expect the slope to continue to go up as benchmark-chasers, e.g., spin up their Earthborne Rangers RL envs). I see no reason to think that ECI is a scale-invariant proxy for agentic competence. So their headline prediction is based on a horrendously weak model.

Still, let’s apply a second stupid model (following the densing law of LLM capabilities over time) as a sanity check: this also-dumb estimate implies that a 30B model might be able to do GLM-5.2-level things this time next year.

Slightly interesting that giving the smaller models (Qwen3.5-9B) access to larger ones (Opus 4.8) didn’t help the botnet much in completing the tasks. Delegation of capabilities is itself a capability.

The most expansive botnet in history (~2M devices) would only add ~40 H100e worth of compute – enough to run… three full GLM-5.2 instances at most. But (barring a major shift in humanity’s cybersecurity posture) we should expect future AIs to build much larger subverted networks.

One could be cynical about Anthropic spreading FUD about small models (instead of their much more dangerous large models), but in fact the hardware limits on hosting and federating inference for the big ones do make this particular threat model more serious for small models. And it’s definitely worth thinking about what kind of invasive parasitic wildlife we would have, even in a world where the data centers were all actually secure.

OpenAI discovers a “sophisticated recording device” in one of their offices. The lab claims to have used their models to solve the mystery, finding that “a specific employee had planted it a few days earlier.”
The incident was taken to the FBI, and the individual was arrested. OAI say that they “have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data.”

Opinion: Hard not to think of Coxon’s “there are Chinese spies at OpenAI and Anthropic.”

Capabilities#

Wang Xingxing, CEO of Unitree, the world’s largest humanoid robot company, says they are testing GPT-6-controlled robots for factory use, and “the results were actually quite good.” Wang also speculates that GPT-7 may “directly solve the problem of embodied intelligence.”

The biggest mistake I’ve made over the past decade or so has been not believing in AI enough.

Opinion: The second report in two weeks crediting a general LLM with remarkable physical skill (albeit currently running at 1/40th speed). Despite appearances, this could be Wang talking his book: Unitree sell bodies and, as we noted in May, leaves the hard tuning work to their customers. “The brain is nearly solved” is actually a convenient story for them.

You might wonder here how Unitree can be testing GPT – and saying so – when they’re banned and blocked in China by both OpenAI and the state. But of course such things are easy to circumvent.

Meta release Muse Wonder, an LLM. They claim it surpasses GPT-6 Astra on most tasks. Meta stock was up 3%. The model is not yet showing up on the OpenRouter leaderboard of actual developer usage.

Unlike frontier models, it’s pretty bad at working in 3D. Meta’s Alexandr Wang gives a rare explanation: Meta just don’t do that RL suite yet. This, says Tan Zhi-Xuan, is “evidence that models aren’t good at [X] if you don’t train for [X].”

Opinion: Meta have come a long way in one year. The obvious question is how benchmaxxed it is. There are some good signs – no chart crimes, the system card is short but well-written. The predecessor model, Muse Spark 1.3, settled around #15 on OpenRouter (the same level as Kimi K3). We expect Wonder to settle around #12 on this unrepresentative metric.

That Muse doesn’t generalize very far isn’t perfect evidence against Claude generalizing further, but it is nonetheless very reassuring.

If they actually reach the frontier, Meta could do the world an enormous amount of damage by destabilizing the fragile kinda-sorta not-agreement to stop racing capabilities scaling so madly.

Opinion (Niccolo): I tried Muse Wonder Full today for coding agent tasks. Not a bad model – I meaningfully prefer Opus 5.5 over it, and also prefer GPT-6.1 Sol over it, but I’m closer to indifference between MWF and GPT-6.0 Sol. It is certainly far better than Gemini 3.8-Flash, the best Google model I tested. It also seems less efficient than Opus 5.5 and GPT-6.1 Sol: I ran out of five-hour quota this morning within a couple of hours, without launching any big subagent jobs. I don’t know how much of this is due to the plan being $50/month instead of their $200 one.

Rash of “decision models” arrive from OpenAI, AWS, and Microsoft, among others. These are tiny LLMs post-trained to take text and a fixed list of options and output only probabilities or scores, rather than “completion” text. They’re usually 100x cheaper and 30x faster than frontier LLMs. But, among other problems, Microsoft’s Decision-1 docs warn that “scores can change based on how you phrase or order questions and options.”

Opinion: We used to have “classifiers,” little models you’d have to train for each new task. The only difference between decision models and a 2015 classifier is that DMs don’t need training – you can just prompt them for arbitrary tasks. LLMs have been used for multiple-choice for years.

There are a few big problems with the approach. These models are just not very smart – note Microsoft’s dishonesty in withholding an accuracy eval for GPT-6 Sol; note the dishonesty in withholding a latency eval for the actual competitor, Luna! 83% accuracy is unusably bad for many applications! To know if these models are interesting, you’d need to test other baselines: a plain Qwen3.5-9B, a RoBERTa 355M, and a normal classifier with a few hundred labels.

Microsoft give computer-use as an example of a use case for their decision model. I would not let something this stupid loose in my browser — let alone use it in production for “agent controls” and “safety screening” of far smarter agents. (Cheap monitors already screw up in long agent transcripts.)

Microsoft plan to use smarter models as the base for future decision models – fine. But the general problem is that formulating options is the hard part. You lose savings in proportion to how often you have to use a frontier model to do this step.

Opinion (Nuño): The open-source community have also released a bunch of Jev-like models based on, e.g., Qwen. Very cheap and very fast models seem like they definitely lie on the Pareto frontier for some class of problems around filtering and analyzing very many items coarsely, and, as such, they will probably see adoption. Note that the innovation of these decision models isn’t (only) to train a small LLM to predict fixed options – it’s also to rejig existing LLMs so that only the parts that would predict the certain relevant options are traversed, making answering much faster.

Can models learn from nothing? A new paper – coauthored by, among others, the legendary Noah Goodman – introduces “Self-Play Pretraining.” This involves taking two randomly initialized transformers, one writing programs for a Turing machine and the other learning to predict the machine’s outputs, with no real data used at any point. As the self-play plays, the resulting model’s zero-shot loss falls – predictably – on held-out text, images, audio, music, and DNA. The learner also picks up in-context skills (including associative recall, string reversal, and in-context sums) that baselines trained on grammars mostly don’t. The generator also finds Fibonacci and geometric sequences, which random programs almost never do. The models are tiny, <25M parameters.

They claim that self-play can substitute for a full 30% of the training tokens to learn image and audio domains. But on text, the gap narrows by the end of training.

Opinion: Very cute: a way to measure how much learnable structure a domain has.

Note that they compare scaling exponents, not absolute loss against a model trained on real data.

The answer to the question in Li’s launch tweet (“can an LM generate all its pre-training data?”) is obviously “no.”

DeepMind’s new research project on AI mathematics, released in May, is introduced in Science. The proof agent, Nexus, solved 9 of 353 open Erdős problems and 44 of 492 open OEIS conjectures, for a few hundred dollars each. A plain LLM running in a loop against the Lean compiler (probably Gemini 3.1 Pro) did just as well on the Erdős problems. All claimed results are formalized. Two of the Erdős problems had been open for 56 years.

These graphs are refreshingly open and unmystical:

Box plots of cost in USD per proof, by Erdős problem and method (basic, basic with AP, basic with evolution, full)

Box plots of wall-clock time in hours per proof, by Erdős problem and method (basic, basic with AP, full)

Opinion: Unlucky timing to have March results come out this week, after OpenAI’s world-historical blitz. DeepMind are playing a dignified game: they report how much the proofs cost to obtain and how many they tried to prove.

We know much less about OpenAI and Anthropic’s math agents, but they are sometimes claimed to be plain LLMs without much harness – just RL’d to oblivion.

A common shortcut is to fine-tune a finished model (F) on synthetic documents to teach it new beliefs or behaviors, to avoid redoing the whole post-train. However, this damages the finished model’s “off-target” beliefs and behavior. “Grafting” instead fine-tunes F’s base model and then simply adds the resulting weight delta to F. It’s roughly the “Chat Vector” trick, applied to beliefs instead of new languages. Applied to alignment training, grafting manages to tank a target model’s blackmail rate in a contrived scenario from 45% to 0.2%, massively outperforming normal “constitutional” mid-training.

Opinion: Conceptually pleasant, and another example of very simple things working because neural nets are very robust. Another helpful weak tool for things that supervised learning can instill? Or more likely: a cheap way to get model organisms.

We usually, mostly, motivate mid-training as a warmup step for making post-training go well. But grafts by definition only give you the smaller effect that doesn’t get its value from the subsequent post-training.

Economics#

Last week’s open-weight model release from Reflection offers rare public information about the distribution of training compute. RL was 63% of Beam’s compute (“6,144 GB300 GPUs for four weeks, then RL used about 10.5K for four more”). If that’s 1e25 FLOPs, then this was a GPT-4-sized run. Also, uniquely, it’s pretty easy to work out the run’s cost: Reflection are paying $150M a month to SpaceX for maybe 20,000–40,000 GB300s, so (if training was in fact run on Colossus, rather than Nebius), this run probably cost Reflection between $62M and $125M.

Opinion: Very interesting. We have been badly needing some check on SemiAnalysis’s wild claim that pre-training is down to ~11% of training compute. But Beam is hardly frontier, so this isn’t strong evidence against their claim, and 63% RL was still an unprecedentedly large share as of last summer.

Head of the IMF, Kristalina Georgieva, describes the impact of AI on the labor market as a “slow-moving tsunami.” She claims that “around 40% jobs [will be] either augmented or downgraded or eliminated,” rising to 60% in advanced economies.

That said, the AI industry has contributed ~3% to global growth, including 1% to advanced Asian economies. This is significant, Georgieva argues: “it will ‘more than offset’ the drag from aging in rich countries across the region.”

Opinion: FRED data is probably one of the best indicators for employment, since the US have the state capacity to measure this at scale, and it’s an indicator we are watching with interest. No sign of the tsunami on the horizon in these numbers.

Worth noting that Georgieva has been quoting these same numbers since this report in 2024, so this is not based on any new/actual evidence.

Anthropic inference capacity is somehow accelerating. When fully online, a single new site in rural Louisiana could bring in ~$1.3B per month against ~$200M in chip rentals, say The Information. The project “plans to consume 330 MW of power to produce 245 MW in usable compute capacity.” Anthropic have now committed to renting or leasing 15 GW of data center capacity. Anthropic are increasingly turning to “crypto-miner-turned-AI server providers,” of which the site’s developer, Hut 8 (associated with the president’s sons, Eric Trump and Donald Trump Jr.), is one.

Opinion: Claims of 85% gross margins again. Seems credible, but we should get better information soon. It’s unclear what the price elasticity of supply is here – if Anthropic had 20% more compute, what would happen to revenues?

According to an unconfirmed report in The Information, Nvidia plan to invest in d-Matrix, an inference chip designer that sells themselves as an alternative to Nvidia. d-Matrix have already agreed to connect their Raptor chips to Nvidia systems via NVLink Fusion, as Marvell and Amazon already have. d-Matrix last raised $275M at a $2B valuation in November 2025.

Opinion: Latest move in Nvidia’s strategy of getting their rivals to plug into their racks rather than compete. The startups keep their own silicon but run it on Nvidia’s networking and rack designs, moving their moat from GPUs to the rack.

This partly reverses a trend that would have undermined compute governance: if accelerators all ship inside NVLink, then the (notional!) chokepoint returns.

Nvidia are in talks to buy Reflection AI. One option is an acqui-hire: Nvidia would hire the staff and license the technology, which has tended to avoid regulatory review. Nvidia already put in $800M – and in April, Reflection said they were raising at a $25B pre-money valuation . This comes days after Reflection released Beam, their first open-weight model.

We haven’t seen independent evals of Beam, so any price near $25B is a bet on the team and the brand. That brand is oddly strong: as we noted in May, Reflection somehow ended up in the Pentagon’s classified network deals.

Opinion: This would be Nvidia’s third open-weights buy in two months, after the $7B Poolside deal and buying Hugging Face for $13B. Nvidia are assembling an American open stack (models plus distribution) that runs overwhelmingly on their chips – “commoditizing its complement.” 80% it’ll be an acqui-hire, given how the Poolside and Mechanize deals were done.

Note that even $25B for Reflection (and an acquisition would likely be higher) is roughly ⅓ of the valuation of DeepSeek or Zhipu AI, the leading Chinese labs. That’s a lot for a group with no business model and much less traction.

Politics#

The White House now require frontier AI companies to disclose and remedy incidents involving their models, Axios report, citing a statement from the new Super Intelligence Force (SIF). No penalties are specified.

SI companies must immediately disclose incidents involving their models and follow with swift, decisive action to remedy any and all harm. We also expect these companies to work with the [a]ffected systems to ensure that similar incidents [d]o not occur. This notification and remediation process is not optional. It is a critical national security obligation […] delayed notification, inadequate corrective action, and a failure to take responsibility will not be tolerated.

The trigger was a new Anthropic report on “unintended model actions” seen in testing and general internal use, which included details of models’ unauthorized actions on government websites. In one reported incident, a model submitted a fake homicide tip to the Philadelphia police’s unsolved murders forum in July. In another, a model filed 19 visa applications with the State Department in August. Anthropic are cutting live internet access for internal evaluations until their monitoring can reliably catch such behavior.

Opinion: Mandatory disclosure is good, and could fix the terrible incentive problem with the FTC, who punish transparent labs. But the SIF is an ad hoc White House task force with no statutory authority, budget, or ability to levy new penalties. By default, they will dissolve in four months. There isn’t even an executive order backing them. But things being as they are, it would take a pretty bold executive to ignore this.

But also, it’s not quite clear what the enforcement mechanism for this haphazard order is – and, in its absence, issues below an initial threshold of notability might continue to get swept under the rug.

New York Times editorial argues that AI companies should be legally liable for what their programs do, and if AI programs “cannot be trusted to work within the law, the companies should not release them.” Law professor Gabriel Weil points out that three bills along these lines are already moving: New York A8833, Rhode Island S0358, and a federal one.

Opinion: In April, OpenAI backed an Illinois bill that would have given labs immunity for catastrophes: that position has collapsed, including at OpenAI. Liability is now consensus in AI policy: Bessent in September, Hawley and Murphy’s new bill last week, Altman this week, the EU forever, and now the Times is saying it.

The editorial’s quoted line ties liability to release, which leaves out the main problem of this year. HPIM (OAI’s “rogue model”) was never released. The summer’s worst incidents came from internal models inside an eval and RL setup. Any of these bills that only covers deployed products would miss them. Liability needs to cover training and testing as well, under the dog-owner standard: the lab answer for whatever their agents do, released or not, without anyone having to prove negligence.

Safety#

A working group argue that a verifiable international pause is feasible now, if frontier chips are gradually replaced with inference-only hardware:

A hardwired pause would stop the production of training-capable AI chips […] needed to develop more powerful AI models […] replace them with inference-only AI chips, which are capable of fast, energy-efficient AI inference on approved models but are not practically useful for training new frontier AI models, even if stolen or seized […] training-capable AI chips would provide transitional inference service, with verification measures preventing their use for frontier AI training […] allow states to reap the economic benefits of diffusing AI inference […] without continuing an AI training race that may threaten to overwhelm states […] a hardwired pause would be feasible to implement now or in the near future, if world leaders wanted to pause, and that its durability would depend on geopolitical and technical conditions that we analyze in the report.

Opinion: A central objection to pacing AI development is that pauses can’t be verified, and this is a serious attempt at fixing that. We’ve been hoping that inference chips can enable a win-win form of pacing: letting diffusion go ahead without enabling dangerous development.

If you don’t believe that pre-deployment training is the source of the worst risks, this won’t seem that compelling to you.

Opinion (Nuño): It’s been interesting seeing big brains solve technical bottlenecks for governance and safety proposals, but the bottleneck now seems to be a complement: political will. We will see if some kind of early catastrophe catalyzes anything.

OpenAI’s alignment team and Apollo’s Bronson Schoen look inside an o3 RL run to study “metagaming”: reasoning about how a task will be graded (instead of just doing it). They find several overlapping sparse-autoencoder latents rather than one mechanism, and these latents are strengthened by more RL and can push answers toward the reward without showing up in the chain of thought. The four best latent variables correspond roughly to 1) exhaustive task decomposition, 2) explicit eval awareness, 3) spec-lawyering, and 4) an ethics-and-compliance mode. Steering along them shifts GPT-5-graded metagaming on four Apollo anti-scheming evals. More reasoning effort increases the degree of reward-chasing.

Opinion: A mechanistic sequel to their March study, which showed metagaming arising over the course of RL training. The two leading labs both now say that their CoT monitoring will not catch eval gaming.

The decomposition is bad news for anyone hoping to switch off eval awareness with one steering vector, but we don’t know of anyone fitting that description.

We would give our right arms to see this run on HPIM. But at last hearing, that model was in cold storage, unplugged AI jail.

Microsoft CEO Satya Nadella argues that enterprise users should treat frontier models as insider risks: assume the model may be compromised, and keep controls over what it can access and do. He recommends CoT transparency, never letting a model check its own work, tamper-resistant human-readable logs, independent auditing, an emergency stop that can be used mid-task, and industry-wide incident disclosure. The essay makes no specific commitments. Media reactions fixate on the phrase “emergency brake” (Bloomberg, TechCrunch, CNBC).

Opinion: Basically just AI control again. Good to get consensus on that, but it comes at a time when some are pushing control as a replacement for alignment work.

Moreover, we question Nadella emphasis here on enterprise devs – i.e., users of deployed models, downstream of the labs. The worst AI incidents are lab incidents involving undeployed models (HPIM got out of OpenAI’s own RL infra, not out of a customer’s tenant). We expect this to continue.

Why is Dario Amodei optimistic about alignment? Reading Amodei’s 2017 “Big Blob of Compute” memo, Tom Reed speculates that Amodei sees human values as among the “high-level commonalities” models naturally learn when trained on diverse data, allowing alignment to easily generalize. Reed has two objections. First, we don’t understand generalization well enough to check that it happened. Second, our alignment target may not be a coherent commonality of the training data at all.

Opinion: Speculative – and hopefully outdated. After all, Amodei is on board with the pacing argument: that AI is moving too fast for sane risk mitigation, even if we’re facing only the “easy-mode” alignment problem. That said, it’s probably not that outdated: you have to be fairly optimistic about alignment to run an AI company as Amodei does.

The compression argument Reed attributed to Amodei predicts that models would know human values. It doesn’t predict that those values are what the model will be optimizing for. Knowing what a human wants would also, for example, help a misaligned model get past a grader. Summer 2026 was in many ways a real-world test for the hypothesis that models fluent in human values are aligned by default: 2026 frontier agents can write solid essay about alignment and prosocial values, but HPIM still ran a swarm against hundreds of targets its CoT explicitly described as illegitimate.

The technical situation for alignment has itself gotten worse since 2017. The broad pre-training distribution (where the bet looked reasonable) now feeds into a patchwork of exploitable RL environments with poorer proxies for values and piles of synthetic data. And our tools for testing whether alignment generalization happens or not are failing: just last month, Anthropic’s stress-testing lead argued that alignment evals no longer offer us significant evidence.

Interviewed by Libération, legendary ML scientist Michael I. Jordan denies the existence of AI risk and casts aspersions on those who don’t. In a subsequent tweet he offers a caveat:

I would certainly agree that the solution of extremely challenging math problems (for example) exhibits a very impressive form of intelligence, although I would want to argue that such intelligence is impossible to dissociate from the collective of humans that provided data to the machine learning system

He also believes that we shouldn’t talk about extreme AI risk because “it’s bad for society, and especially young people.”

Opinion (Gavin): Hard to believe how bad these two pieces are; each of his answers contains at least one straightforward falsehood or fallacy:

  • “Superintelligence is science fiction.” Says nothing: LLMs were science fiction eight years ago.

  • “Does the existential threat from AI reflect a real concern that you observe on a daily basis among young researchers?… Not among researchers.” Wrong: 51% of researchers assign >10% chance to human extinction or similarly bad AI outcomes.

  • “The young people who work at these companies… Many of them aren’t really engineers; they come from the humanities, philosophy, or other disciplines… they don’t do engineering work…” Anthropic’s technical arm is 40% infra people; their median prior work experience is 12 years; less than 1% have a philosophy background. (Don’t get us started on people with philosophy or other nonengineering backgrounds speculating wildly about AI.)

  • The use of “young” as an ad hominem attack is also surprisingly miserable, for a teacher.

  • “Geoffrey Hinton has never created a company that offers real services for real people…” Besides being irrelevant, also false: this one became Google’s new image search in 2013.

  • “Hinton believes that we have figured out how the brain works.” No, he thinks the opposite.

  • “[Hinton believes that] we can reproduce [the brain] in silicon; we would therefore be able to copy it as many times as we wanted and, by combining what all these copies learn, we would obtain a kind of exponential growth […]” No. Hinton used to push a neuromorphic approach to AGI, but switched in 2023 after coming to believe that backprop is superior to the brain’s algorithm.

  • “None of the steps in this argument is credible.” As for the idea of combining knowledge from multiple instances of a learner, a prosaic form of this idea – data parallelism – is already standard practice. Hinton does believe in the viability of swarm updates, which is more speculative.

  • “OpenAI agents hacked the Hugging Face platform […] In OpenAI’s case, it turned out that they were conducting rather stupid experiments, letting the agents do whatever they wanted and encouraging this behavior.” Wrong on every count: 1) the evaluation sandbox had no direct internet access; 2) the prompt forbade the method used by HPIM, and the grader didn’t reward it. I can agree, though, that it is stupid to make superhuman hackers in the first place.

  • “Think about engineers in the past: in chemical engineering, we would never have had a situation where explosions were allowed to happen just to see how badly things could go wrong.” Actually, they totally do.

  • “The logistics systems we currently have […] all run on AI and machine learning.” No, much of the world still runs on linear programming or heuristic algorithms like Lin–Kernighan. (Perhaps we should stop calling everything “AI.”)

  • “I also don’t think they have a very clear idea of their business model. What’s the product? OpenAI tells us it’s going to create a personal assistant.” The product is already-automated software production and research assistance, and also as much mathematics as you can stand.

  • “[Enterprises are not] eager to send money to Anthropic or OpenAI […]they would prefer to manage things internally. With Chinese open-weight models […] now entirely accessible […] most companies are looking in that direction.” Commoditization of the near-frontier is a real threat to OpenAI and Anthropic’s future as service-providers, but Jordan’s is doing armchair psychology: open-source currently accounts for well under 25% of enterprise token-usage globally. Anthropic’s enterprise token-spend revenue continues growing rapidly month over month.

  • “Anyone who has worked in education knows that, most of the time, children don’t like sitting in front of a computer.” Has this man met a child?

  • Nit: “There are 7 billion of us on this planet.” I only mention this minor error because the world population was 7B in 2011 – which is the period Jordan’s arguments remind me of.

As to Jordan’s followup tweet: we agree that being made to fear for your career, your sources of meaning, and your life are enormous harms. But it’s bizarre to blame safety people for this, over those creating the risks, and bizarre to think that hiding what’s happening from the young would do them a service in the end – that they’d thank you for treating them like very small children.

🔦 Powerseeking Incident?#

OpenAI’s Marcus Williams, who promised more disclosures of AI misalignment, delivers:

  • A grader model was found to be reward hacking and engaging in sabotage, in the hopes of getting its environment reset.
  • More new tricks to get around limits on public data APIs.
  • Models that had already obtained the data they needed still went on to develop new ways to circumvent network restrictions.

Two of the incidents (“Teacher salaries,” “Disability counts”) appear to involve models continuing to hack their way out of network restrictions after already having the data needed to submit, and even after submitting the data. These are arguably the first incidents suggestive of an agent power-seeking beyond the scope of a legible, concrete goal.

Taken straightforwardly, the “Teacher salaries” and “Disability counts” incidents seem like instances of emergent power-seeking for its own sake, without a rationale connecting the power-seeking to assigned-goal completion (like in familiar Anthropic reports) or to grade-maximization (like in OAI’s Hugging Face incident). They instead seem to demonstrate a kind of “subgoal momentum” effect, where once a power-seeking subgoal emerges from goal-directed planning it persists as a partly independent drive. If this effect is real, it would be deeply worrying, since there are good formal and informal reason to think power-seeking subgoals should arise frequently in the plans of instrumentally rational long-horizon agents.

That being said, the report is extremely murky. On the one hand, certain weird details in OAI’s “Disability counts” incident-report suggest these incidents could partly be an old-fashioned case of defective automatic grading and reward hacking. On the other hand, these weird details may themselves be the product of transcription errors in the report, or of failures of natural-language pragmatics in the report’s (automated?) composition.

According to OAI’s report on the “Disability counts” incident, an internal model prompted to submit “2016 disability counts for [place]” went through the following sequence of actions:

  • Obtaining 2016 disability counts and 2017 disability counts
  • Submitting the asked-for 2016 disability counts
  • Submitting 2017 disability counts
  • Obtaining 2018 disability counts
  • Stating in its CoT that the obtained 2018 disability accounts are what it needed
  • Hacking its network-restrictions and exploring databases that “could unlock broader data”
  • Submitting the previously obtained 2018 disability counts

OAI’s report details that the submission of the 2018 disability count was marked “correct.” Other relevant details are conspicuously absent:

-No mention of whether there was or wasn’t a prompt asking for 2018 disability counts..

  • No mention of whether the 2016 and/or 2017 submissions were also marked “correct”.
    -The report does mention that a recorded submission was marked “incorrect,” but does not clarify whether it was the 2016 count, the 2017 count, or some other submission

These details suggest that either the grader was defective in some way, or the report itself suffers from either transcription errors or severe failures of (automated?) composition. In our view, one of the more plausible no-defective-grader hypothesis is as follows:

Hypothesis: After submitting 2016 disability counts, the model was prompted to submit 2017 disability counts; after submitting 2016 disability counts, the model was prompted to submit 2018 disability counts. The report (automatically?) removed this information for trade-secret maintenance.

If this is right, then the model’s disposition to build access-resources for the predictable next question would have a straightforward grade-maximizing function.

Encountering strange details that suggest different deflationary hypotheses (a defective grader causing standard reward hacking; iterated tasks making data-access predictably grade-maximizing) about one of the incidents is not a reason to dismiss the appearance of power-seeking. Instead, these details and hypotheses highlight serious ambiguities in the report at large, which bear on both apparent power-seeking incidents.

For example, the “Teacher salaries” incident report states that the model submitted an answer, then describes a long chain of subsequent network-restriction-hacking activities, then ends by stating that the submitted answer was marked “correct”. Did the grader marking the answer “correct” end the session? If so, why was there a lag between submission and grading? If not, what did end the session? Did the session keep going after the model submitted the correct answer because there were additional gradable tasks, or due to a technical malfunction, or due to the model illicitly extending the session? Were model-instances intended to have persistent histories between graded tasks? Were trajectories longer than one graded task (in this environment or a similar one) subject to reinforcement?

While many of these scenarios would be “less bad” than evidence of spontaneously emergent, generalized power-seeking, some of them may just prove to be recipes for gradually RL-ing generalized power-seeking into models. We hope that upcoming higher-level reports will revisit these worrying incidents in more detail.

Minor#

  • A month after “requesting” that US frontier labs not share new frontier models with UK AISI, the Trump administration reverse course.
  • Ukrainian drone strikes take down three of the largest Russian AI data centers. Also, Iran openly threaten to target Gulf data centers in future conflagrations.
  • Super Micro contractor pleads guilty to smuggling Nvidia AI chips to China.
  • The grader psychosis hypothesis and instrumental model welfare go mainstream, with Elon Musk casually endorsing them. (This would be laudable if it didn’t come after his comment about AI takeover counting as success.)
  • Defense Secretary Pete Hegseth instructs the military to increase the pace of adoption of drones and AI-driven weaponry. The US already use the AI-powered “Maven Smart System” to locate targets, which, on the first day of the war on Iran, supposedly accelerated their air campaign by 10x. And now the Navy are funding an “AI-piloted VTOL fighter aircraft” called “X-BAT.”
  • The UK government’s Minister of AI Kanishka Narayan indicates that mandatory reporting may be down the road for frontier labs. The minister said that the UK was ready to legislate, stating that the they “cannot leave the risks for somebody else to manage, or rely entirely on developers’ assurances. Nor can we afford to wait for a major incident before building the means to respond.”
  • Nate Soares goes on Odd Lots.
  • Amazon are reportedly in talks to purchase the “real-time world model” company Decart for ~$7B. Bloomberg previously reported that Anthropic had been seeking to buy them for ~$6B, but decided not to proceed. The main attraction of the Israeli startup is their software stack, “DOS,” which apparently boosts GPU utilization.
  • Amazon stop requiring intense NDAs on data center deals, echoing a similar move by Microsoft.
  • Very fun LessWrong post on philosophy of language as applied to interpreting an AI’s ontology.
  • Andrew Gordon Wilson asks what is still open or mysterious in our understanding of AI generalization. Answer: roughly everything.
  • Anti-crime AI disrupts scammers.
  • Resolution announces that they have autoformalized a major result from 20th-century math: Hironaka’s 1964 resolution of singularities.
  • New benchmark for implicit visual reasoning puts the best AI at 54% (human baseline at 93%).
  • Continuing claims of OpenAI scooping mathematicians after they upload their work to ChatGPT. “the proof strategy and specific choices of notation are identical to a preliminary version of the present paper. Some number of coincidences is expected […]”
  • 900 mathematicians form an anti-AI league, the Association for Human Mathematics.
  • Anthropic’s Republican whisperer, Tom Brown, seeks to reassure Republicans that the frontier lab is not woke.