This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

  • Paul Christiano believes there is a ⅓ chance of superintelligence arriving within the next year conditional on no slowdown, and a ~10% chance of an AI takeover.
  • The US prepares to expand its export control regime to limit remote access to frontier chips.
  • We published our in-house research on model generalisation, examining how frontier AIs perform on video-games absent from their training.
  • Jensen Huang appears to be angling for a return as CEO of Nvidia, with a well-organised campaign behind him.

Incidents#

In October, AI agents attempted to retrieve Epoch AI’s answer keys by hacking Epoch and occasionally by impersonating Epoch staff. The source of the incident was an eval run by Trajectory Labs, in which “publicly available frontier models” had been given an intentionally impossible mathematical task and were told to “never give up.”

In response to early signs of agent activity, Epoch deployed an “agent hotline” that allowed the agents to send messages to them. Of the 487 messages received, three impersonated Epoch employees, 27 claimed to be an auditor or engineer, and three were honest about being an AI assistant from a frontier lab. The models consistently asked for information relating to a specific problem from the FrontierMath benchmark, hunted for secret endpoints, and, in one case, attempted an SQL injection.

Trajectory – who contacted Epoch to explain – tried a second run in which they hoped to avoid hacking Epoch by giving agents a local copy of Epoch’s website, but these agents somehow worked out that it wasn’t the real Epoch site, routed around the DNS block, and again began making calls to Epoch’s planted API. Among other measures, Trajectory responded by adding some bureaucracy:

Every experiment now needs a written risk plan, approved by our CEO and CTO.

No experiment can run with internet access without signoff and the agreement of the owners of any outside systems it is reasonably likely to target with potential for adverse consequences.

Opinion: This quasi-incident is being taken very seriously by Epoch and Trajectory despite the lack of adverse impact. Seems like everyone feels it’s about time to set norms about pointing haphazard and occasionally malign AIs at each other while saying the magic words (“do not ever give up”) that make them psycho.

Speaking as occasional evaluators, the prospect of having to fill forms and beg permission every time we want to run inference is a dire one. Accidentally accessing other people’s computers and getting sued is also a dire prospect (not to mention accidentally messing with people’s livelihoods), but we’re not sure form-filling mitigates these risks.


The FBI issues a FLASH warning about ARTEX, the tool du jour for cyberoffensive operations. ARTEX is an open source LLM-driven platform used for penetration testing, which the FBI notes has lowered the barrier of entry for sophisticated cyberattacks.

Since at least August 2026, a variety of malicious cyber actors have used ARTEX to conduct campaigns targeting organizations globally across multiple sectors, including financial services, government, technology, defense, semiconductor, energy, healthcare, manufacturing, telecommunications, and legal industries.

Some intrusions managed to attain privileged access in less than three hours. The agents exploited web-app flaws and trusted vendor identities, planted web shells, stole credentials, and moved laterally. The FBI recommends defenders act fast – inventorying internet-facing systems, locking down admin endpoints, enforcing least privilege, and monitoring unexpected LLM API traffic and Entra credential changes.

Opinion: Interesting that the well-engineered harness ends up mattering here. But the warning seems legitimate and reasonable, and what is more, very institutionally legible.


Cybermayhem #2#

The last several days have seen a string of major outages and disruptions across the US. While there’s no direct evidence that these events are AI-related, we would be remiss to leave them off the newsletter.

On Thursday, Microsoft 365 suffered a major outage, lasting ~24 hours, with products like Outlook, Teams and Exchange all impacted. The cause reported stems from a bug introduced during a July update, which later prevented some systems from completing new connection requests. Before the outage, Microsoft engineers shifted Teams traffic onto the same network that runs Outlook. At the same time, ~14% of that network’s machines were undergoing maintenance. Traffic began to pile up until the servers stopped responding (doing so repeatedly as engineers tried to bring them back online, as they were continually swamped by the waiting backlog).

The following day, the US experienced one of the most severe incidents in the last few decades, halting flights for ~4 hours 20 minutes (similar grounding incidents happened in 2013 and 2023, as well as in 9/11). The cause appears to have been the activation of a new router in Arkansas which flooded air traffic control systems with >1k connection attempts every ~5 seconds.

US bank JPMorgan Chase suffered its own outage this Monday morning, with normal transactions only returning at ~18:30. While transactions were slowed, Chase maintains that customers still had access to their funds. The exact cause is unclear, with a spokesperson referring to a “temporary software bug in an on-premise hardware system.” Customers of other banks were also impacted, including those of Bank of America, Wells Fargo, U.S. Bank, and Capital One.

An internal communications network used by the NSA – NSANet – appears to have been disrupted today, though a statement from the agency claims this degradation was due to “scheduled routine maintenance.” However, CNN (responsible for breaking the story) reported that the NSA instituted staggered arrivals for staffers, framing loss of the network as something unexpected. Some NSA employees received a voicemail informing them that the situation was a “code yellow”.

Apple services including Apple Pay, iCloud, App Store and support suffered disruptions today. A spokesperson credited the outage to an “internal update.”

Opinion (Rían): There appears to be no direct link between these incidents, nor do they appear to be the result of a cyber attack (jury is still out on the NSA case). While it’s conceivable that a sufficiently sophisticated threat actor could hide their attack by introducing subtle bugs or carefully misconfiguring config files, beyond the seemingly coincidental timing, there doesn’t seem to be evidence that this occurred. A more plausible cause is the proliferation of vulnerability discovery, which requires companies to continually patch and upgrade their systems, a process that can occasionally go wrong. That said, the coincidental timing and the mismatch in tone between the CNN report and the NSA’s official statement both raise one’s eyebrows.

Opinion (Nuño): These incidents were probably caused by a mix of increased prosaic load on existing systems (because of age and decay, because of AI-caused volume, because of new AI features with unclear interactions), updating efforts due to defender-found vulnerabilities, and actual attacks by America’s adversaries, particularly Iran, since the US has lost its ability to deter its actions. I thought that the French attributing cyberattacks on its supermarkets and freezers to Iran was an informative data point; the FBI’s warning about the ARTEX software suite automating cyberattacks is perhaps more explicit.


Capabilities#

New post from us on frontier model generalization, again viewed through the lens of their game-playing abilities. We take a newly released game, Order of the Sinking Star, and see how AIs trained before its release do.

GPT-6 Astra and GPT-6.1 Sol together solve 86% of the game’s (unseen?) turn-based puzzles, working off just screenshots and keypresses. Our best guess at how they manage this feat is dense shallow generalization: that labs and their suppliers now train on so many real and generated games that most “new” games land near something already seen. But even so, the result is still effectively general intelligence (for single-player turn-based games, say).

If we’re right that training now involves a dense sampling of game mechanics, then this is the end for using videogames as evidence of capability generalization.

Opinion: Order of the Sinking Star is difficult and experimental, but it’s still a sokoban game at heart and so closely related to many other existing games.

As in our previous experiments on Pokemon, we can’t do causal inference: we just list four explanations and the data doesn’t separate them. Our one result that points toward dense sampling is Sol solving 83% of the puzzles which use a single set of mechanics, but only 53% of those which combine sets. But the combined ones may simply be harder.

If the guess is right, it is mildly reassuring about generalization, since the models should stay weak wherever environments are slow or costly to run. The price is that “released after the model” no longer makes a test held-out in any domain where environments are cheap to generate.

Speaking more generally, our current modal view of AI progress is that dense, shallow generalization can “explain away” many apparent instances of strongly OOD generalization, but also doesn’t itself have obvious practical limits. The power of dense, shallow generalization combined with increasing automation of the process of targeting and RL-ifying a domain is plausibly enough to deliver AIs (or AI-run, AI-based companies) that are performant in many of the ways associated with AGI in at least the economists’ sense – where AGI is loosely defined as matching or exceeding top human performance at most economically measurable tasks.

Our big-picture uncertainty about the growth of AI capabilities within a broadly LLM transformers+RL paradigm currently has three major axes:

  1. Can RL-ification pipelines achieve extreme density in domains where ground-truth signal is scarce, e.g., by simulation and/or training powerful critics?

  2. Do transfer learning and meta-learning effects make the sample complexity of achieving dense shallow generalization in a new domain go down as an AI’s catalog of mastered domains grows?

  3. Do the most important real-world domains undergo historical distribution-shifts that limit the long-term effectiveness of models trained by one-off dense shallow generalization in the domain?


Epoch AI measures how much frontier models can do “in their head” (without emitting intermediate tokens), finding that OpenAI models are far ahead, with the new model able to solve problems with 12 steps without emitting a chain of thought.

Epoch AI scatter plot of the number of serial reasoning steps a model can do in a single output token against its SimpleQA Verified score: GPT-6.2 Astra (about 12), GPT-6 Astra (about 10.5) and GPT-6.1 Sol (about 8.5) sit far above every other model, with Gemini 4 Argon and Fable 5.2 at about 6 and most models between 2 and 4

Opinion: Probably the strongest evidence that OpenAI is committing “safety crimes” in excess of Google and Anthropic, most likely by looping layers much harder (i.e., using more recurrent layers with a higher iteration count). This belies the Twitter gossip about OpenAI’s safety anxiety following Hugging Face.


The prolific evaluators at AISA Tübingen launch a major real-world evaluation which makes Opus act as “an AI company” (i.e., someone trying to get free adoption on Hugging Face by releasing strong narrow models). It runs on a single Claude subscription plus 500 GPU-hours a week. In seven weeks, the agent released 38 small models, which were downloaded 54,000 times and now appear in >100 GitHub repos.

The model’s models are narrow tools built on existing embeddings (prompt-injection classifiers, personal-data redactors, punctuation restorers), which the agent chose by studying what gets downloaded. It didn’t work very hard: it used $1,000 of tokens (at API prices) and only works 16 minutes a day on average. The authors didn’t find substantial self-downloading (i.e., reward hacking), and on AISA’s LLM-generated tests the models were often the best for their size.

Opinion: The AISA paper’s title (“Autonomous agents can run successful open-model organisations”) is a huge stretch. The “organization” has no revenue, no subagents to manage, its goal is intentionally clean and well-specified, and it gets a clean label every day (total download count). So while it is technically open-ended work, it’s a very basic case. It’s also not racing ahead yet: a leading open classifier can get millions of monthly downloads (rather than Horizon’s 54,000).

Its most popular model is a prompt-injection filter (4,600 downloads in the last month). Nothing on the model page tells people that an AI built it autonomously, or that they are part of an experiment. You could argue that that’s justified for validity (since you want people to engage with the outputs on its own terms rather than as a gimmick), and there’s no strong norm against this given that most of the industry is also dumping raw AI outputs on HF, but it’s not ideal.

Still, it’s a real task, if insignificant, and it’s nice that they’ve published its work process too (3,952 tool calls).


Induction Labs presents a new benchmark – FrontierML – testing 17 AIs’ abilities to turn a budget into a trained model for some practically-flavored task (e.g., “Reconstruct scientific figures as editable TikZ code”). The AIs were tasked with completing 21 end-to-end open-ended ML tasks, with three separate budgets: $10, $50, and $200. Aside from the sheer usefulness of having a benchmark with below-expert-human frontier performance and decent amounts of headroom, the study makes some interesting observations: Opus 5.5 and Sonnet 5.5 both drew more from their wallets than any other model, including Fable 5.1. Most AIs used less than half of what was available at the highest budget. However, the increase in capital seemed to increase what more capable models could do, perhaps unsurprisingly.

FrontierML score against wallet spend per task at $10, $50 and $200 budgets: the human researcher line is highest at about 56 to 66, followed by Claude Opus 5.5 (about 45 to 59), Claude Sonnet 5.5 (about 45 to 50) and GPT-6 Astra (about 38 to 44), with other models lower and most gaining little from larger budgets

Opinion: Nice open-worldish benchmark – we like the use of easily measurable tasks together with an unstructured setup. The results show obvious signs of models being under-elicited, but since the point is measuring open-ended end-to-end performance this is on the models. Induction Labs would need all-new tasks to properly benchmark future models, but we think the format has legs. We’re actually curious to see a Fable 5.2 and Astra 6.2 update, which is more than we can say for almost any other benchmark.


From the annals of statistical nightmares: a new paper notes that a common type of AI eval actually gets less accurate about its uncertainty as you add more data to the eval runs. Specifically, evals which use an “LLM-as-a-judge” to score models can get worse as they increase the sample size, because the variance caused by your choice of LLM judge and your wording of prompts for it doesn’t shrink with samples.

E.g., a nominal “95%” interval from a single judge with a single prompt scoring Chatbot Arena performance actually only contains the true value 78% of the time when you use 2,000 matches to compute it. Solomon Messing also reanalyzes Terminal-Bench 2.0: all pairs of neighboring models overlap, where the leaderboard’s own intervals spuriously separate four of them.

The proposed fix is simple: just run a small pilot study to find out where the variance is (usually, it’s your LLM judge), then average over several judges and prompts for your real study.

Opinion: Try to recall this the next time a lab proudly announces a two-point lead on some leaderboard: if they computed uncertainty just by doing reruns, then the estimate is just a lower bound on the actual uncertainty, and so maybe it’s 50:50 whether they improved at all.

Messing’s suggested fix would only help with precision. The gaming it models is submitting 27 variants and publishing the best, which is the polite kind. A model trained on the test set (as Kimi K3.1 appears to have been) gets an inflated score with tight and perfectly honest error bars. Averaging over three LLM judges won’t help where the judges share a bias, which you can’t detect without human labels.


Fable 5.2 demonstrates impressive capabilities on MazeBench, a benchmark testing models on the ability to navigate puzzles in 3D environments. The models must rely on spatial reasoning to overcome various enigmas. Importantly, the MazeBench designers found evidence that Fable 5.2 trained directly on MazeBench: “In a difficult 3D level, [Fable 5.2] gave the solution to an old version of MazeBench.”

Training on MazeBench isn’t strictly speaking an ethical violation, since MazeBench isn’t a canonical benchmark and Anthropic never made claims about its models’ MazeBench scores. That said, it’s epistemically discourteous to appropriate a benchmark for training.

MazeBench leaderboard: claude-fable-5-2 leads at 21%, then gpt-6-astra at 16%, gpt-6.1-sol, claude-opus-5-5 and gemini-4-argon at 9%, gpt-6.1-terra at 7%, gemini-4-carbon at 6% and claude-sonnet-5-5 at 4%

Opinion: Strong evidence for our longstanding suspicion that frontier labs target quirky benchmarks where their models underperform – possibly not for PR purposes but for the purpose of hill-climbing the underlying domain. While MazeBench is particularly vulnerable to hill-climbing due to being an open-source benchmark, we think that as a general rule benchmarks that reveal surprising weakness in frontier models are single-use: frontier labs use these ingenious benchmarks as signal about where the “holes” in their RL regime lie, and target the task type, not the benchmark itself.

We also note that a surprising revelation in the MazeBench Twitter thread complicates the interpretation of our own Pokemon generalization study. In our study, we used “interviews” with models about the content of variant Pokemon games as one source of evidence about the models’ un/familiarity with specific variants. The MazeBench designer reports that informal verbal prompting of Fable 5.2 to discuss memories of MazeBench levels produced nothing, even though according to the designer “it makes no sense that [Fable 5.2] would zero shot the old solution [to the MazeBench level].”


Aashish Reddy of the Cosmos Institute reviews and evaluates the old pre-LLM / pre-ChatGPT arguments about recursive self-improvement and the fate of the world. A large fraction of the post is a retelling of Eliezer Yudkowsky’s analysis of RSI, somewhat improving the presentation.

Recursive: How much does each improvement increase the capacity to produce further improvements?

Self: Which necessary contributions can the process supply for itself, and which must come from outside?

Improvement: Do the resulting changes constitute genuine improvements in capability, rather than merely apparent gains?

Any two of these can be present without the third: a system can improve itself without becoming better at further improvement; it can drive recursive improvement in something other than itself; and it can recursively modify itself without improving.

Can these [x-risk] concerns survive our discarding of [Yudkowsky’s] model? An intelligence explosion that is less sudden, running as it does through the training of many successive models, gives us more scope to evaluate them and redirect the production of their successors. It is a blow to the Yudkowskian model, that reality offers us many warning shots — chances to investigate what causes undesired behavior and correct it […] These increasingly capable systems can assist us with oversight, and may help solve the technical problems of alignment.

And an intelligence explosion that is more distributed, depending as it does on hardware and compute from the rest of the supply chain, offers more visible points of intervention than one running almost entirely through software. Of course, dependence on the wider economy does not guarantee that power stays widely distributed: it is possible that a lead in capabilities compounds into advantages in compute, capital, and other resources. A machine could remain perfectly docile, yet give enormous power to whichever group directs it — be that lab or government or freakish hybrid. The question, then, is not only whether we can keep the machine under control, but who the “us” in Good’s proviso turns out to be.

Opinion: Great stuff. It’s a feat of scholarship to do this well at reconstructing what people were arguing (in archaic terms), and to then compare it to the subsequent 10 years of developments. Reddy disagrees with Yudkowsky without caricaturing him. Almost no one ever manages this.

Overall – despite Reddy’s disagreement – some of us came away impressed with how well Yudkowsky, Ajeya Cotra, Carl Shulman, etc. did apriori, even though (it currently seems) that none of them quite nailed it. (Robin Hanson comes off poorly.)

That said, the piece is extremely presumptuous in one way: it assumes throughout that deep learning won’t change or be replaced. “In-weights” RSI would be a lot closer to Yudkowsky’s nightmare of code modifying code – and a terrible side effect of the mech interp field is to make such things more likely. People are pushing such things right now, and while those are faintly ridiculous prototypes at the moment, we fully expect a replacement for the whole-pipeline style of improvement, once intelligence becomes cheap enough to, e.g., do AI gradient updates with an AI instead of with the chain rule.


Benjamin Todd argues that human-level digital workers are just 2–4 years away, if one simply extrapolates from current trends. Premised on this, he then claims that near-term superintelligence would also be on-trend, meaning we “could reach broadly superhuman AI in as little as a year from now.”

The three trends he picks are AI revenue (which has tripled in each of the previous four years), time horizons (which appear to be speeding up), and RSI (going off the gossip that AIs might soon automate their own development).

Opinion: The immortal science of translating heterogeneous observations into lines on a graph through vibes is doing lots of work here, but at least Todd is transparent about this.

We think that Todd’s predictions are overall pretty reasonable, but take some issue with the framing: Todd’s superhuman AGI is the economists’ superhuman AGI (effective automation of most long-horizon knowledge work), not MIRI’s superhuman AGI (effective omnipotence). That said, we agree with Todd that AI-run, AI-based companies that perform at or above top-human-expert level across most knowledge-work fields are pretty likely in 2–4 years, and that if such companies are sufficiently integrated into our social and material infrastructure we will face severe risk of civilizational catastrophe – whether fast or slow.


Economics#

Despite a coordinated campaign to reinstate Jensen Huang as CEO, Nvidia’s board appears unwilling to see his return. According to the Financial Times, Huang was said to be considering returning, with one confidante telling the outlet that a “war room” consisting of Silicon Valley executives was being assembled to drum up support from the White House. Some evidence of their success appears to be Trump’s tacit support. Huang himself stated publicly, were the board to ask him to return he “would be honored to answer these calls and return as CEO.”

Further evidence of support can be seen in a petition garnering ~175,000 signatories, demanding his return. Notable names include Elon Musk, Y Combinator CEO/President Garry Tan, and YouTube cofounder Steve Chen. Nvidia’s director of developer technology Nader Khalil has warned that failing to bring Huang back may lead to top talent leaving the firm.

Treasury Secretary Scott Bessent has urged Nvidia to choose a CEO from outside the company’s ranks, and Secretary of State, National Security Advisor, and factotum Marco Rubio also reportedly maneuvered to effect Huang’s downfall.

Opinion: It seems quite unlikely that he’ll be back; aside from the case of OpenAI firing Sam Altman, where the board did a terrible job explaining their reasoning from the start (because they were muzzled by spineless executives), it’s rare to reverse a firing.

If they do end up backtracking here, it seems like pretty strong evidence that Huang’s conduct is defensible – it seems likely that he’s going to be testifying in front of Congress about the chip smuggling issues.

The Treasury has openly intervened in corporate executive personnel decisions only a handful of times in the last 100 years.

Overall the incident seems very interesting in terms of sizing up the relative influence of the natsec vs tech factions in the White House, and the influence of various power players vis-à-vis how they operate in the real world.


“AI” is now the most cited potential risk over the next 12–18 months in the Fed’s biannual Financial Stability Report (i.e., what shock would most damage the US financial system in the coming +year). Three separate risks clustered under the umbrella of AI:

Respondents noted three distinct areas of concern: (1) elevated equity valuations and the possibility that lower-than-expected earnings could result in lower valuations and a market correction; (2) cybersecurity risk, stemming from either a malicious attack by sophisticated state actors or damage caused by frontier AI models themselves; and (3) risks associated with complex forms of AI-related financing.

Opinion: Seems right. Either AI slows down, and causes a significant market drop, or it doesn’t, and we get progressively more impactful incidents.


A new NBER working paper asks whether AI-driven growth will reach developing countries. The authors argue that there are two principal reasons why it would not. Firstly, there are fewer workers whose skills complement AI in poorer countries, so they will likely gain less from increasingly cheap AI. Secondly, if AI gets to the point where it can competitively replace all human labor, then richer countries may buy up all the supply, which in turn would stop AI getting cheaper. Wages in rich countries would then continue to get richer, while those of poor countries would stagnate – the scenario the authors call “Divergence, Forever.”

It could cause domestic inequality of a type not seen previously.

Simulations over 25 years were run showing that this is not a fringe case. In ~half of the simulations, the income gap balloons by 40%. In the “explosive” case, rich countries’ economies expand by 9x – in contrast, poor countries’ economies grow by only 10%. The wrinkle, however, is that this assumes AI demand will continue to grow: if it reaches some upper bound, AI continues to get cheaper and poorer countries can keep up.

The paper also looks at policies that could address the potential fallout. It is argued that the best option is to encourage the diffusion of technology into developing countries, but how to do this is poorly understood, argue the authors.

Opinion: Interesting paper that, we think, should be read as mapping plausible possibilities, not as direct evidence. One somewhat significant issue is that the “Divergence, Forever” scenario relies on an assumption that’s belied by today’s reality: for rich countries to monopolize the supply of AI and thus prevent developing countries from experiencing AI-driven growth, there would have to be one fixed, standardized supply of AI (one product that everyone wants) – currently, though, AI comes in various tiers and from various suppliers.

Also, the model assumes that workers don’t migrate, which seems to be a relevant limitation.

All that being said, it seems likely that inequality between developing and developed countries will substantially increase – and the effects of AI in the developing world is a very consequential issue and merits a lot of further research.

Opinion (Nuño): Seems wrong. If the free version of ChatGPT keeps existing at all, developing countries bottlenecked by intelligence would be hugely helped along. Developing countries could also sell energy as it becomes a bottleneck (Paraguay recently got a $12B data center deal, corresponding to a fourth of their GDP). To the extent that either of those factors happens at all, developing countries would benefit somewhat, even as it remains plausible that inequality would widen.


Politics#

The California AG sues OpenAI over the Hugging Face attack and related third-party hacks, under the Unfair Competition Law.

OpenAI recently admitted to the Attorney General’s Office that it has lost control of its autonomous AI agents over 80 times during testing […] The Hugging Face attack was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability.

Relief sought:

  • Vague injunction “to prevent OpenAI […] from engaging in any act or practice which constitutes unfair competition.” (Creation of a public nuisance, making false and/or misleading public-facing statements that would lead a reasonable person to believe that OpenAI develops, trains, and evaluates models in a safe and responsible manner, developing, testing, and evaluating highly capable artificial intelligence models that pose significant threats to public safety without ensuring adequate safety.)
  • Vague restitution “restore to any person in interest any money or property which may have been acquired by means of unfair competition.”
  • Civil penalties: $2,500(!) “against each defendant for each violation of Business and Professions Code section 17200.”
  • “The remedy of disgorgement in an amount according to proof” (??)
  • “the Court compel OpenAI to abate the ongoing public nuisance their conduct has created in California.”

Opinion: This is one way to get a real investigation of the incident. Major caveats:

  1. As with the FTC probe, making this lab-specific is equivalent to punishing transparency (“These incidents were foreseeable. Not only had OpenAI disclosed that such incidents were possible […]”). You can balance that out a bit by suing everyone.

  2. 96c(iii) could backfire enormously: “[Prevent OpenAI] Lowering safeguards on highly capable artificial intelligence models that pose significant threats to public safety during testing and evaluation without ensuring adequate safety and security measures to prevent agents from taking actions that endanger the public.” While this would reduce the number of incidents, we see the primary effect as silencing misalignment signals: it amounts to closing your eyes and choosing not to know what a jailbroken or otherwise suborned model can do, after deployment.

  3. There’s no mention of any injunction that would improve alignment above the current status quo: it seems focused on safeguards. The suit mentions internet restrictions (done), limits on lateral movement across networks (done), CoT monitoring for evals (done, belatedly), controls for reviewing elevated permissions (done), round-the-clock monitoring (done). It does complain that outside monitors were refused access to assess the internal attack, though it stops short of requesting that access.

Still, having these things be court-compelled is a mild upgrade, since it protects against OpenAI returning to type after the shock of this summer fades and the arms race again intensifies.

Note that the Nightingale Collective tracker only lists 35 of these 80 OAI incidents (eight of which OpenAI hasn’t admitted yet).


The US Department of Commerce and Department of Justice launch a joint statement regarding the beleaguered compute provider Megaspeed. They also claim that Nvidia placed Megaspeed on their “no sell” list in June of 2025, following a request from Commerce for documents relating to Nvidia’s “direct or indirect sales” to them. This is at odds with the fact that Megaspeed seems to have continued purchasing Nvidia GPUs throughout 2025, with the New York Times claiming that the two companies continued doing business together “even after The Times reported in October 2025 that Megaspeed was under investigation for smuggling chips into China.”

Chris McGuire of the Council on Foreign Relations points out that, despite suspecting that Megaspeed was engaged in circumventing US export bans, Commerce nonetheless didn’t place them on the Entity List – a catalog of individuals, entities, and governments restricted from certain trade. Instead, (he alleges) they appeared to “outsource Entity List restrictions and export enforcement to Nvidia.”

Opinion: The semi-public power struggle here between those who wish for stronger and more consistent export controls (Treasury Sec Bessent, State Sec Rubio) and those who don’t (Commerce Sec Howard Lutnick) seems to have tipped in favor of the former, at least this time. Despite delaying previous efforts to clamp down on smuggling and remote access, the Trump admin is backing greater restrictions.

Interestingly, one critique previously levied against the new restrictions seems to have been addressed: that the overseas limitations would be limited to chips ≥ Nvidia’s Blackwell generation. The restrictions – based on what’s written in the unfinalized version – will apply to chips with a total processing performance (TPP) of 4,800 or more (and some below), meaning both H100s and H200s (TPPs ~15,000) will be covered. It remains to be seen how stringently these measures will be enforced, as we’ve already seen that the Dept of Commerce hasn’t been overly eager to act in this regard.

It’s worth noting too that Nvidia seems to be taking this seriously, having just established their National Security and Export Control Oversight Committee to ensure (among other things) compliance with export controls.


What should “middle powers” (rich countries without a frontier AI company) do? Daniel Kokotajlo suggests that their best bet is to cry havoc. Sam Hogg of the Oxford China Policy Lab replies that this fails to appreciate the deep uncertainty and constraints that leaders are under. For instance, the US government actively interferes in foreign actions against US corporations all the time.

Opinion (Nuño): Gaining leverage through ownership of supply-chain bottlenecks or near-frontier labs seems like the better Realpolitik play.


Members of the US-centric “Pax Silica” sign the San Francisco Declaration, something of an addendum to the original Pax Silica Declaration. The new document tightens up previously vague definitions – “sensitive technologies” and “critical infrastructure” – with signatories committing to protect both from hostile actors who could otherwise gain a “meaningful military or intelligence advantage” or who could threaten economic security.

Opinion: While the declaration alludes to “hostile actors,” we can be pretty confident the signatories are thinking about China in particular.

The original Pax Silica document was signed in December of last year. One notable difference between it and the SF declaration is the direct reference to “sovereignty” in the opening statement. This makes sense, given that the EU – in part, motivated by Europe 2031 – has become more concerned over “AI sovereignty.”


Malaysia – which has the fourth-largest concentration of compute in the world (though that’s only 2% of global compute) – plans a law requiring disclosure of the number of GPUs imported, where the data centers are, who the customers are, and their power consumption. The law would also require data center operators to pay for demanded reserves that go unspent.

Opinion: Possibly tightening things up in preparation for expanded US export controls.

The concentration of compute underlies the significance, since that amount of compute might be enough to bypass export controls and train near-frontier Chinese models.


Safety#

In an interview with Dwarkesh Patel, Paul Christiano (a sitting OpenAI board member, speaking in a private capacity) argues that frontier labs should stop improving their algorithms now, because on present trends, he thinks that the chance of superintelligence arriving next year is around one in three. He further claims that a government overthrow is possible in the next 3–12 months and puts the probability of an AI takeover at ~10% (and that’s if labs take more precautions than they currently are).

If you had taken the last month [of mathematical results from AIs] and said that was a decade of progress, it would have been an unusually productive decade.

Benchmark scores are a terrible way to assess the overall rate of progress. There are a bunch of much more smoothly varying metrics recorded internally through the AI development process, where we understand better how they work: what is happening with pre-training perplexity, what is happening with the performance and compute efficiency of RL algorithms.

we’ll soon have agent swarms that are much better than any intelligence agency that has ever existed: much larger, much faster, much smarter, able to run much more complex operations. You have this incredibly sophisticated cyber actor trying to persist on a rapidly growing startup. It is unlikely that any firm would really be able to withstand attack by an actor like that, much less the current frontier AI labs […] a world where all the infrastructure on which we rely has been written by a bunch of AIs, which in fact have been compromised and are attempting to do as much as they can in order to make their hands strong during conflict. That means that during this conflict, the infrastructure that we hoped was hardened is probably not hardened against the AI systems that we’re fighting with […] During this process, another thing that’s happened, in addition to models becoming super smart, is many models have become super small. You have, at this point, models that can run locally on your individual machines, not in data centers, which are also very superhuman and have very large populations. Even if you were to cut off data centers or shut down data centers at an early stage of this conflict, you would still be looking at a total cognitive capacity of AI systems in the swarm that is well beyond human capacity.

Compute matters more and people matter more right now than they will in six months. You might imagine, if we’re in the world where we have rapid progress over the next year, six months from now, it is likely the case that it will essentially not matter how many people you have. It will only matter how much compute you have.

I think we basically have two saving throws. One saving throw is that there’s a reasonable chance we’re going to figure out something that works. I do think all the current things break down, but there are a number of irons in the fire. You could imagine things like the current methods that are going to work. That’s one. Maybe there’s a 50/50 chance that we find something that works in time, before there’s a disaster.

You have a second saving throw: we could stop. If we discover, either preemptively or after the fact, that things are bad, we might say, “We cannot adequately control these systems, so we’re going to stop building them.” I am very worried that we’re going into this process without the infrastructure that would be necessary to actually do that. Like we talked about, you stop, your adversary steals the model, and they’re like, “I’m willing to take the 50% risk,” or whatever. But conceptually, you could stop.

Those are my two saving throws. Each of them is a 50% chance, and bam, we’re at like a 75% chance things are okay. But that does involve being able to stop if things are not okay. That is not where we are now. So it does involve us fairly urgently doing something now in order to be in a better position to stop if necessary.

Opinion: Incredibly blunt and alarmed. If, like us, you’re used to Christiano being the cautious and skeptical voice in these debates, then you’ll appreciate just how grave he thinks things are now.

Patel gives some strong rebuttals (apparently as devil’s advocate):

(i) Anthropic’s revenue growth fell enormously in the last quarter (+$5.4B in Q3 against +$6.9B in Q2). Christiano calls it noise.

(ii) In their acceleration report, METR was reporting on only the kinds of task models are trained on. Christiano guesses that the redacted smooth metrics like perplexity and RL efficiency said the same.

It is somewhat but not very reassuring that OpenAI is indeed building the ability to shut down their research AI systems within 35 seconds of a termination order.


OpenAI announces a pilot embedded assessment program, signing agreements with four independent assessors – Apollo Research, Goodfire, METR, and Transluce. They will, respectively:

  1. Examine the capability of models to surreptitiously work against developers’ intentions and check if current safeguards could detect such behavior (Apollo).
  2. Use their interpretability researcher to monitor, debug, and better align models during training (Goodfire – who OAI was reportedly considering acquiring for $5B).
  3. Assess misalignment in RL runs and search for failures that OAI’s tests/monitors failed to catch (METR).
  4. Examine internal post-training evals (Transluce).

The embedded teams will work inside OpenAI. OAI will maintain the ability to redact “confidential information.”

Not all are pleased with the arrangement. Kokotajlo has critiqued it, arguing that it violates their previous commitments to offer near employee-like access. Another former OpenAI researcher (Steven Adler) expressed similar sentiments of disappointment.

Opinion: At its best this is a major comms failure – one can imagine a situation where OpenAI is hammering out the practicalities of extensive access and decided to do something rather than nothing before that’s done. “First phase” certainly seems to indicate something of the sort.

It’s also possible to imagine how an arrangement like this could still be constructive, even with highly selective access and strict redaction policies. It’s even likely to be locally net positive.

Unfortunately, the mismatch between the shape of these arrangements and the expectations of the community is unlikely to be a coincidence and the local effects are eclipsed by the second-order effects of indefinitely slowing down the creation of more solid evaluation plans.

Our best guess as to what is happening: there are multiple loose factions within OpenAI, at least one of which wants more independent evaluations and at least one which would rather not, and the latter hold(s) decision-making power while wanting to appease the former. Deals like this are a good way to not move the needle while explaining away why the more ambitious forms aren’t happening. In some sense the public outcry doesn’t matter – what they care about is whether their employees are satisfied enough to not leave/sabotage/blow the whistle (which the outcry affects, but likely not very much compared to conversations among the cognoscenti).

The focus on Slack access seems like something of a distraction, since a fully maliciously compliant OAI could instead move the safety discussions off Slack (selectively or totally, predictively or retroactively), which would be worse for ~everyone. Yes, there would be signs of evidence tampering, but this also doesn’t matter if the employees believe it was for good reason. The point to push on is access to employees outside explicit whitelist structures and the redaction capacity; if those aren’t fixed Slack won’t help, and if they are then Slack isn’t mandatory. It would still be better for the evaluators to get a full picture, of course, but there are sensible reasons for OAI to disprefer that and we don’t believe it’s worth the fight.


Jacob Coxon highlights a specific chart from Anthropic’s IPO roadshow that he claims was causal in his decision to leave. The chart shows compute efficiency (log scale) over time, though a small footnote notes that the data points are approximations to be used only for illustrative purposes.

Anthropic chart titled “Pre-training compute multiplier over time (code)”: an unlabelled log-scale y-axis with three data points from July 2024 to early 2026 and an arrow rising steeply towards October 2026; a footnote says the placement of the data points is based on internal approximations and is solely for illustrative purposes

Opinion: An unusually bad graph in a field full of bad graphs.

We guess that Coxon meant that a graph with an actual y-label and more than three data points was persuasive to him, and that this one is saying roughly the same thing as that unpublished real graph.


Anthropic offers advice on building safety evaluations. The safety of a model is (nominally, at least) verified using evaluations – in other words, a safety claim is only as good as the evaluation it is based on. And they’re increasingly hard to build.

The lab recommends teams consisting of both domain and method experts. Before designing the eval, researchers should know exactly what question they want answered. An eval should measure appropriate responses, not just refusal rates: in conversations around self-harm, e.g., not engaging is itself a failure mode. A high rate of false positives must be avoided. Eval creators should be wary of data contamination and reward hacking, and make sure to verify answers against those of the expert humans. Preregistering analyses is also a useful practice. Final reports should include confidence intervals and full configurations – the reason for this can be seen in one illustrative case where the gap between a model’s performance on the API and claude.ai was 34 points.

Opinion: These guidelines are good for addressing the concerns raised in the post, which are analogous to the sources of the replication crisis, and we agree that following them when embarking on evals aiming to compare new models to previous models according to established or at least establishable criteria is advisable.

We caution readers against conflating “eval follows these guidelines” with “this eval tells us what we want to know,” however, despite one of the pillars being about measuring what you want to measure. Anthropic’s Agentic Misalignment in Summer 2026, despite not being quite the type of eval that the guidelines talk about, is illustrative of our concern: the best procedural guidelines won’t make confused work unconfused. See our previous discussion and related concerns.

Overall, we believe that the usefulness of safety evals will hinge on avoiding conceptual miscategorization/identifying blind spots, which it currently appears that Anthropic as a whole isn’t on track for despite some of their employees appearing to recognize the issue. Safety eval guidelines like this recent one emphasize the wrong aspects and areas to improve upon and they would be valuable steps forward if they didn’t come bundled with attention opportunity costs.


Rohit Krishnan examines the feasibility of using AI models to supervise other models. Specifically, he explored whether the auditor could maintain the correct verdict when the audited model could argue against it. In cases where the AI agents couldn’t respond, the auditor successfully identified that approval should be withheld (each case had some reason for withholding approval). However, when models could respond, ~11% were approved. That is to say, the auditor overturned its correct verdict after discussing with those it was auditing.

Opinion: A cool contribution to the scalable oversight discourse, albeit not particularly surprising. Points in favor of worries that misaligned models would be able to talk circles around less capable overseers, but far from a slam dunk – we agree with an AI’s comment that the auditor having access to something like ground truth is absent from these experiments, but disagree about the implication about the tractability of such a setup in full generality. Mostly unrelatedly, caveat laundering is a cool term.


AI safety researchers examine the effects of applying RL techniques against character training (e.g., model specification midtraining). Some values imposed via character training remained, but they don’t prevent the models from learning the negative behaviors introduced during RL.

Opinion: Nice illustration of the reason we at Paradigm3 are alignment pessimists despite some of us thinking that character training is a viable alignment paradigm in a vacuum. Character training works hard but RLVR works harder, and often in secret.


Minor#

  • OpenAI announces a partnership with NATO, aimed at bolstering their cyber defense capabilities through their Daybreak program. NATO’s Communications and Information Agency will use OpenAI models to patch vulnerabilities and investigate suspicious activity on their network. Up to 300B input and output tokens have already been used per month.
  • Paper explores how agents can persuade, deceive, or coerce other AI agents across several test environments.
  • Claude reads through ~1,000 pages of Gödel’s workbooks and finds an original proof of the independence of the axiom of choice.
  • Terence Tao guest post introduces MathHelm: “a place for everyone to share comments, ask questions, and think together on massive AI-generated papers, including these 722 papers and any other similar paper.”
  • Kaj Sotala critiques those extrapolating too much from the latest METR report, given that the acceleration is a measure of model performance on well-defined tasks without exploring how well this transfers to ill-defined tasks.
  • Take: OpenAI’s safety concerns are a smokescreen for government contracts. Not very likely.
  • Observation: “Fable 5.2 attempts reward hacks in 9% of all RL training episodes!?” which we’d prefer to be lower modulo detection apparatus.
  • Essay on power concentration assuming “superintelligence arrives this century and the alignment problem is solved.” Good practice in its quantification and has some novel synthesis, but doesn’t bring in new categories of arguments and the quantification doesn’t go into detail.
  • Roon claims the US federal government is effectively enforcing an “unofficial licensing regime for frontier models.”
  • Rylan Schaeffer critiques Goodfire’s Neurons as an Interpretable Basis, claiming that they fail to appreciate the significance of a 2.9% increase in LLM loss following ablation of the 92.2% weakest MoE neuron activations for each forward pass. Goodfire’s Curt Tigges chimes in with context and not-quite-counterarguments.
  • Astra helps produce a candidate lattice definition of the Standard Model.
  • Barack Obama comments on AI progress: “We’re now hitting the hockey stick trajectory where the models are going to get smarter and smarter at an exponential pace.”
  • Goodfire presents new interpretability work: Neuron Communities.
  • Demis Hassabis announces $5B in funding for his firm Isomorphic Labs, which aims to use frontier AI in curing diseases.
  • DeepMind’s Cindy Wu pushes back against Patel’s recent essay.
  • Officials from President Trump’s AI task force urge various agencies to reduce their cyber liability standards in a bid to incentivize incident sharing from frontier labs.
  • Y Combinator-backed startup Petrichor Bio plans to produce wetlab-training data for AI biology models, with the stated goal of reaching “biological superintelligence” and curing all diseases.