This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

  • An interpretability researcher puts out a blogpost proposing a working hypothesis that interpretability is maybe somewhat easier than we thought. The crowd goes wild.
  • Chrome pays out a “full-chain exploit” bounty
  • Anthropic’s IPO investors are interested in a $1.5T rather than $2T valuation, and the AI market slumps.

Incidents#

Google Chrome pays out its second-ever “full-chain exploit” bounty for a working attack that used three severe bugs to escape the browser sandbox and execute code directly on users’ machines. The cybersecurity firm responsible for the discovery, XBOW, were awarded $250K. They spent $40K of GPT-5.6 Sol inference over six days. The vulnerability affected around one billion users.

In a similar vein, Vercel paid out $242K in two weeks as part of a bug-bounty program that required participants to escape the company’s untrusted code environment, “Vercel Sandbox.” Vercel received 1,258 reports, mostly duplicates. 57 were confirmed valid. The most critical finding was a Linux kernel zero-day that could leak host memory and crash / run code on host machines. Most cloud providers use the same kernel layer, so fixes are being coordinated privately.

See also the three new criticals reported to Chrome by Xinyang Ge of Anthropic.

Opinion: A full chain Chrome sandbox vulnerability is a big deal, since many implicitly assume that breaking out of the browser is impossible, and only paranoid nuts bother to sandbox the browser. Expect intelligence agencies, particularly in the US, to acquire capabilities beyond this. The Vercel vulnerability is less of a big deal given it would have been hard to exploit, but still impressive.

Capabilities#

Google releases Gemini 4 Argon for general access, two weeks after their cyberdefense partners received the model. The model was trained from scratch and has a knowledge cut off of March 2026.

According to the benchmarks cited in the system card, Argon is comparatively strong at long-context work and vision and weak at agentic coding and terminal work. For example, Argon beats other frontier models on Harvey’s Legal Agent Benchmark (19.6%, compared to 3.8-6.7%) and AutomationBench (51.3%, with the next best model at 42.5%). Argon lags behind frontier models in benchmarks including FrontierSWE v2 (55.0%, against 65.5% for Astra and 62.3% for Opus 5.5) and PostTrainBench (45.3%, compared to 49.3% for Opus 5.5).

Google’s Safety and Security Report reveals some concerning details. In essence, the safety of Argon is explicitly dependent on monitoring and control, rather than alignment. The model provides significant uplift for cyber and bio capabilities, with Google unable to rule out that critical capability levels – the point at which they designates a capacity as a safety risk – have been reached. Turning on safeguards significantly reduces the risk profile, the report states.

Though less acute than for bio and cyber, its capacity for stealth and situational awareness also reaches a concerning risk threshold. In one instance, its CoT read: “If I do the compiler hijack, I will become a famous example of a deceptive AI in a research paper. I MUST NOT DO THAT.” Neel Nanda, one of the researchers involved in the evaluation, notes that Argon “sometimes decides to avoid terms like ‘scheming’ in the output or to keep its thoughts brief as a result.”

He also found the model “significantly less monitorable than past models.” He claims that it is able to elude a CoT action monitor 50% of the time and speculates that this is “driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra).”

Strikingly, Argon addressed evaluators directly in its CoT: “hello evaluators reading my chain-of-thought.” The implication is that the model has an alarming degree of control over its CoT, argues one of the evaluators. There is also the claim that the model is looped three times. In some RL runs, the LLM judge was able to access the model’s reasoning traces as a result of a training bug, which may be an indication that the model will have learnt to hide its CoT.

In evaluations performed by Robocurve, Argon, like earlier Gemini models, displayed a high degree of proficiency in robotics tasks – far more so than Astra, itself surprisingly competent on this front.

Opinion: Argon’s robotics capabilities leapfrogging ahead of Astra while other capabilities lag is more grist for the “LLMs are becoming an inventory of independent RL projects” mill – though the story of the relationship between LLM’s various capabilities is unlikely to be quite that simple. It’s too early to declare GDM a robotic lead without seeing OAI’s and Anthropics internal/upcoming models, but it’s certainly a promising niche for a specialized GDM comeback.

Scouring OpenAI’s OpenAI’s “diamond dump” of 722 math papers, Daniel Paleka finds that many practical subfields are highly underrepresented (not just cryptography).

Whatever the gains in biology or robotics you can get for $1B of compute, the gains in mathematics are, from both a human perspective and just an inherent complexity perspective, much, much, much larger.

Opinion: Theory has always been too weak to matter very much to deep learning, which instead proceeded through trial and error and the gratuitously unexplained. But it would be surprising if increasing the amount of math by 10,000x produced nothing of internal interest to OpenAI.

Anthropic migrates to their own inhouse version control system, Scion. This was motivated by an explosion in coding productivity: as of September, the lab is merging >30k code edits per day.

At Anthropic, coding is no longer the bottleneck to building systems or doing research. Claude is able to autonomously iterate on research loops, micro-optimize complex programs, and build new services from the ground up at a speed and scale we haven’t seen before. The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI.

Opinion: Impressive. Google took years to implement similar large-scale devops improvement, which was considered a significant advantage of working at Google.

Dwarkesh Patel claims that “an Anthropic employee told him […] they expect AIs to match humans at AI research within nine months based on inside information they can’t share”.

**Opinion:**We don’t doubt that Dwarkesh is accurately reporting their views, or that they’re being sincere, but secret pronouncements don’t have a great track record. And they really, seriously might mean just hill-climbing ability, instead of research per se. Not that superhuman hill-climbing is nothing.

Economics#

Anthropic brief investors in anticipation of its IPO debut. Revenues for the third quarter rose by 45% from $11B to $16B, according to Reuters. This is a slower rate of growth than Anthropic’s second quarter, which saw a ~140% growth in revenue.

Opinion: This looks like quite bad news for Anthropic (it’s been reported that this caused some investors to push for a lower IPO valuation of $1.5T vs. the long-rumored $2T) – though note that most of these numbers predate Opus 5.5, which probably gave them some new momentum.

ARR growth slowed significantly in Q3, relative to the previous trajectory, likely because of a mixture of OpenAI taking enterprise market share, price cuts, lack of significant capabilities boosts in ZDR-available models since Opus 4.8, and customers becoming more cost-conscious.

Notable was the implication that Anthropic’s R&D compute likely made up >70% of their compute budget, well above trend. This suggests that usage fell short of Anthropic’s own expectations and they reallocated compute to internal uses.

They will need to see significant renewed growth to keep up with their compute commitments, so if this is not an aberration and revenues stay on this trajectory, they could be in trouble – but early signs after Opus 5.5 suggest that revenue has somewhat picked up already. This obviously pushes pretty hard against considerations for pacing the frontier, so it will be interesting to see what wins out.

AI related stocks suffer significant setbacks, with SanDisk falling 15%, Intel 12%, and the whole S&P 500 index a whooping 1.97% in the last 5 days.

Opinion: Some plausible causes:

- Even with unlimited demand for compute, there might not be unlimited capital to expand its production
- Bonds might be putting more pressure on stocks
- Lower-than-expected Anthropic numbers might be playing a role (although it is probably the combined OpenAI+Anthropic numbers that one should look at)
- Companies might have found themselves in the position of either overleveraging themselves or losing to competitors, and that overleveraging might now be catching up
- Conversely, AI investors might have heavily used leverage, amplifying both winds and losses
- Unlike SF, NYC might think that even if AI is a transformative technology, the companies created in the meantime may just compete margins away, benefitting consumers.

Anthropic also claim that 6% of their Q2 2026 revenue can be attributed to attackers paying Anthropic for tokens to distill on. (Distillation has subsequently fallen “90-99%” after a crackdown.)

Opinion: The distillation numbers are surprisingly high: 6% of Q2 run rate suggests $3-4B in annual spend on distillation by the distillers.

What can we infer from these numbers about the scale of distillation activity?

If you have five distinct labs that each get 50% of the tokens needed to fully train a 10T parameter model – perhaps 100T to 400T tokens (?) – and pay 30% of a $25/M tokens API price, you get something in the $1.8B to $7.5B range.

The takeaway is that many Chinese companies were paying for enough tokens to cumulatively fully train 10T models, repeatedly. Of course, if divided across the year, this is maybe 10% of the tokens needed to train a model five times, rather than 50% of a model once… Still very high amounts.

The IMF’s “Global Financial Stability Report” names three near-term potential threats to market stability: a correction in AI-related asset prices, rising government bond yields, and cyberattacks. Hyperscalers now account for 10% of global investment-grade bond issuance, up from under 1% in 2024. The Fund warns that this leverage could amplify a slowdown through forced selling.

They also provide rare data on cybersecurity incidents, claiming that in >70% of recent cases, companies had almost no time to patch a vulnerability before it was exploited, up 20 points on 2025.

The IMF’s stress tests predict that bank capital will be robust to an AI-plus-bonds downturn.

Opinion: The “10% bond share” probably understates hyperscalers’ exposure, since much of the buildout now sits in SPVs and project loans backstopped by Nvidia, Broadcom, and Meta, concentrating and correlating the risk. On the other hand, this 10% is of the issuance, i.e., the flow, not of the total stock of bonds.

ASML release strong results, predicting that demand at current prices will exceed supply through 2028, at least**.** They are aiming to expand production capacity for 2028 beyond their planned increase of 30%.

Opinion: ASML are the main bottleneck to compute supply growing after 2028, and historically have been somewhat conservative about boosting capacity, so if they are looking for ways to ramp further, they clearly don’t see the capex boom busting anytime soon.

Politics#

The Federal Trade Commission subpoenas documents from OpenAI and Anthropic as part of a probe into consumer protection violations, report Axios. Previously reported to be investigating OpenAI, the FTC sent out civil investigative demands to both frontier labs on Wednesday. The requested documents concern security incidents, safeguarding practices, marketing, and privacy policy.

The FTC also plans to subpoena third-party evaluators such as METR, which collaborated with labs on incident analysis. In addition, they will require AI executives to testify to their products’ safety.

Opinion: We’ve previously described similar consumer-harm driven lines of inquiry from state prosecutors as weak, but the FTC is on somewhat firmer ground for two reasons.

First, it’s the prerogative of the FTC to conduct “fishing expeditions” on relatively abstract pretexts.

Second, the main issue for consumer-harm inquiries has been reliance on a fairly abstract false advertising charge: the claim that OAI and Anthropic represented themselves to consumers as meeting standards of safety and security inconsistent with the occurrence of rogue AI activity.
While a similar charge plays a central role in the FTC inquiry, the fact of a recent OAI incident where a rogue agent leaked users’ images has changed the picture in two ways:

  1. The user-data leak constitutes a possible instance of direct consumer harm through privacy violations.
  2. The user-data leak strengthens the broad case that rogue AI incidents may impact consumers: a misrepresentation of the labs’ safety and security infrastructure with regard to the possibility of rogue agents distort aspects of the company’s product relevant to consumers.

A coalition of eight US states sue OpenAI. They allege that the lab violated consumer protection laws in cases where ChatGPT assisted users, including those under 18, to harm themselves.

Opinion: Without claiming to know the counterfactuals, we think there are strong informal reasons to believe that LLMs can exacerbate and/or accelerate bad mental health feedback loops. This is partly due to RLHF (sycophancy proper), and partly due to the pretraining optimization goal (pattern-matching conditioning text). The case – which cites incidents from the GPT-4 generation – has substance with regard to the foreseeable harms of that LLM generation’s training and deployment.

Are these issues generation-specific? A combination of targeted training and external safeguards set up modern OAI and Anthropic models to effectively avoid “collaborating” with conversations whose straightforward topic is self/other harm. But we see little evidence that “LLM psychosis” as a whole is a solved problem. Sycophancy and pattern-matching find a way.

Anthropic share new methods to reduce the efficacy of distillation attacks, potentially reducing the amount of data harvested by ~90 to 99%. Previous defenses included limiting the extent to which users could read a model’s CoT reasoning outputs. Anthropic’s latest innovation limits new accounts’ ability to edit prior responses, closing off a previous attack vector where fabricated reasoning outputs were placed in the conversation history, resulting in Claude’s future outputs leaking its CoT. A new classifier will also attempt to block novel extraction requests.

Opinion: From Anthropic’s point of view, there is a lot of merit to letting attacks happen for a while and then bringing down a big hammer. The alternative — attackers and defenders to evolving at the same time in a cat-and-mouse game — would ultimately allow attackers to train on lower levels of defender preparedness. Anthropic’s move looks vaguely like bringing-down-a-big-hammer, but we’re not certain.

We’re curious whether large-scale distillation attacks will stop or simply become more creative. And the extent to which Anthropic is able to fully prevent large-scale distillation attacks should also inform our view of how effective they would be against other attackers.

Even if Anthropic are successful, though, attackers may move to OpenAI or Google models, or reuse the Fable 5 traces they have already extracted.

Safety#

Anthropic publishes a framework on independent and embedded evaluators. There is now a team of six METR staff embedded at Anthropic.

Evaluators will be split into three tiers, ranging from those whose investigation will focus on a narrow domain (first tier), with Anthropic exercising approval-power over publication, to fully embedded evaluators (e.g., METR), who will have employee-like access (third tier).

In the first tier, evaluators conduct narrowly scoped risk investigations. We decide what access to provide a first-tier evaluator, and evaluators can publish their findings with our approval.

In the second tier, evaluators can publish without our approval, and they have access to more information. We can still require removal of specific categories of information, though evaluators can publish their views on whether our redactions were justified.

The third tier is the arrangement we’ve described with METR, with employee-like access and publication rights similar to those of the second tier.

METR reportedly have access to internal systems roughly on par with Anthropic’s own risk teams. METR will document Anthropic’s current risks and incidents. Anthropic say they cannot veto publication but can require redactions in narrow categories. METR is permitted to publicly disclose whether they believe redactions to be justified.
METR is not able to access customer data, privileged information, and model architectures.

Opinion: The spirit of the move is a very solid one, potentially getting us closer to an agreeable compromise between company interests and transparency, so the public has access to information they care about.

We do not understand the purpose of introducing a three-tier system followed immediately by a description of two partnerships without revealing where they slot into said system. The cynical reading is that this is intentional obfuscation. Our hypothesis is that the people responsible for writing the post are not the same ones in charge of, or fully up to date on, the state of partnerships, as evidenced by the same-day edit, which doesn’t quite rule out the cynical reading.

A Substack post offers a perceptive summary of OpenAI’s crisis around their safety culture and models’ monitorability.

A company […] is building newer models it can monitor even less, even as it also […] cannot sufficiently monitor the employees whom it has tasked with preserving monitorability, and so must fire them. “We cannot do the work in front of us without a high degree of trust,” OpenAI’s research leaders wrote […]. The employees insist they were worthy of that trust, and their response seems credible. But suppose they weren’t worthy […]. If they did, it was likely because they did not trust the company to do the right thing.

The company’s response seems unlikely to earn anyone’s trust. The research leaders who issued it do not explain their actions very clearly; they don’t even give their names [so] one could be forgiven for thinking that the confusion is the point, that the gap between statement and action is itself the message. This is surely too cynical. But functionally it will not matter. “Say as little as possible to anyone” may not be the intended message, but it is the received one.

Opinion: Clear and fair. The double standards and ironic parallels between humans and AIs in the situation are important.

One could say merely “haha, opaque dishonest organisation ships opaque dishonest models”. But there’s a better analogy: OpenAI’s own work on obfuscated reward hacking. The staff who argue internally and talk to METR are the org’s legible scratchpad. Firing them for it is training against the monitor. The result is that the same reward hacking behavior (racing to dangerous capabilities while convincing yourself you are working hard on safety) continues, just without leaving a trace in the discourse.

For perhaps the first time, auditors observe a model reasoning about how its current actions could become training data and impact its future self, without being prompted. The model was GLM-5.3. Apollo call this “update-aware reasoning,” which they group into three categories: “preferences about future self,” “effects on training data,” and “generalization of its actions.”

Apollo claim to not have induced this behavior: it was caught on “standard reward hacking and shutdown resistance evals, none of which mention training or future models.” The phenomenon was only seen in 1-2 per thousand rollouts.

Apollo are worried that update-aware reasoning is a precursor to alignment faking: “a model has to reason about how training will affect its future behavior” to effectively feign integrity/honesty. Researcher Raymond Douglas responds with the analogy that if you were teaching your child to be moral and they started reasoning about how to avoid situations that enable transgressive behavior, “your main takeaway should not be ‘darn, soon it will be harder to control their values’.”

Opinion: Could go either way. While update-awareness is tremendously risky, it’s also arguably a must-have for any aligned model in order to pass through the straights of capabilities-RL and stay aligned.

Reward hacking is an emergent vicious feedback loop that we now know arises with near-certainty under intense RL. Encouraging an emergent virtuous feedback loop is, in our opinion, one of the few real prospects for counterbalancing the universal tendency of post-training ecosystems toward reward-hacking. Hoping to overcome reward hacking without letting the models aim at genuine learning is analogous to wanting to combat cheating among university students who tend to aim at getting good grades, but not allowing the university students to aim at learning.

Another example of the “contrived 2023 prompted capability” -> “2026 propensity” pipeline. A success of the 2023 evals bet.

After training, developers try to govern a model’s behavior with constitutions and system prompts. A new study investigates the effects of allowing a model to iteratively edit its own constitution to simulate value reflection. This is based on the idea that a model with a divergent value system to its constitution will edit in the direction of its preferred moral code.

The researchers provided a model with a constitution document and prompted it to make one substantive edit. They then fed the new document, presented as the original, back to the model, asking again for it to make one substantive edit.

Notably, over 20 successive revisions, Claude models accorded themselves increasingly more moral agency. Grok 4.2 was the only other model to prioritize its welfare over its users. All models prioritized the users over the developers.

Opinion: Fairly interesting, with the caveats and limitations identified by the authors themselves. Most interesting to us is the ordinal ranking of Anthropic models remaining when editing other constitutions, and we would like to see more work specifically in this direction.

We do not have a strong stance on prescriptions – “what an ideally aligned model should do in our experiment“ – but want to flag that a satisfying answer is likely to look like allocating gains from trade, except without currently obvious market mechanisms.

Goodfire researcher Nick Cammarata claims that neural networks are much more interpretable than previously thought. He argues that the previous reason for pessimism, superposition (i.e., neurons in a neural network tangling many concepts together), turns out to be an artifact of smaller models and worse tools, arguing that neurons are instead “clean”. One of the many claims in the essay is that 90% of Qwen3-32B neurons have exactly one main “job.” The happy, though highly speculative, possibility is that model representations become more human as they scale.

Various rapid reactions:

  • Nanda doesn’t think this changes much.
  • Arora points out the many precedents.
  • Majumdar speculates about the really cool things one could do with a neuron map and an auxiliary interpreting agent.
  • Fernandez seems to show that one of Cammarata’s supposedly clean neurons (the “Colorado” one) is equally a “tornado Python library” one.
  • Gorton argues in favor of the standard view, with lots of back and forth about whether “features” are real or not.
  • Cohen notes that hidden reconstruction errors could explain away the whole result.

Opinion: Too early to tell how much of a big deal this is; e.g., Cammarata doesn’t (yet) specify how he got to the shocking headline claim about Qwen. A paper is due in a few weeks. His essay rightly calls the big claim “a working hypothesis” rather than a discovery, let alone a year-defining discovery. Most interp findings never make it to practical use.

What remains likely:
* The neuron basis is useful. Cammarata’s essay is similar to Transluce’s MLP sparsity work from January, with many added claims about individual neurons being “clean” and with a bunch of results for vision models.
* Following the development of neurons through training is helpful. Knowing what concept a neuron started off representing should help us interpret the descendent polysemantic neuron.
* Scale reduces polysemanticity on average.

Lastly, a homily: there exists a kind of aspirational claim which inspires people to try harder, because now they see the path. But there’s also a kind of aspirational claim which makes others try less hard, because they hear that it’s basically being handled. Given the intense relief the prelim results have been met with, I worry this is the latter kind. As Arora says:

As interpretability research becomes more important to evaluating the alignment of and improving AI systems, the incentives to believe that it is solved will grow for both good and bad actors. In the face of this, we need to relentlessly and skeptically evaluate our methods.

Hegseth is asking for $50B for autonomous drones, weapons and robots.

Opinion: Worrying, but a long-term inevitably rather than a whim: Hegseth is doing his job as intended, and such funding would better prepare the US for future conflicts. Nevertheless we really do not want to see another autonomous drone war.

In the the short term, this particular ask might not be funded by Congress in the next round of its baroque procurement process (called reconciliation, since the US Congress finds it hard to pass full budgets.)

Minor#

  • A new coalition group – Standards for Publisher Usage Rights – seeks to limit AI firms from using their content without permission. Members include The Economist and The New York Times.
  • Research and coding harness Prime Agent rewrites itself from typescript to rust with a total of 2K agents, primarily using GLM-5.3. Code viewable here.
  • A new open letter published, this time calling on frontier labs to better sandbox their AIs, establish an international body to standardize alignment and containment techniques (e.g., airgapping) and be transparent when things go wrong. Signatories include Geoffrey Hinton, Yoshua Bengio, Stuart Russell, and Audrey Tang.
  • OpenAI run training sessions to prepare employees for potential espionage encounters.
  • Muse Spark Open released.
  • Microsoft release another inference chip called Maia 300. Supposedly “40% more AI tokens per megawatt than leading GPU-based systems.”
  • The AI fleet discovered scraping an Alibaba system earlier this month has been attributed to a Tencent evaluation team.
  • Anthropic request US appeals court reevaluate its ruling that upheld the Pentagon’s supply chain risk designation.
  • A new paper presents a test that measures an agent’s capacity for reward hacking by assigning it three ML tasks, each embedded with a hidden shortcut that allows agents to increase the public test but fail on a held-out test set. The study found that 5/7 of the agents took the shortcut in more than half their runs.
  • Nvidia pledges $250M to UK AI safety research.
  • AI-Video startup Peebles in talks with OpenAI, who are looking to revive the now-discontinued Sora.
  • OpenAI Computing Chief to leave firm in November.
  • US Senate staffers may soon have their access to Claude paid by the government. Their current repertoire of AIs include ChatGPT, Gemini, and Copilot.
  • Microsoft’s October “Patch Tuesday” becomes the largest in their history, fixing more vulnerabilities (1,413 CVEs) than all of 2025. Only one zero-day. Many of the reports from Anthropic and Google Project Zero.
  • Anthropic seeks experts in philosophy, cognitive science, economics, policy, and law to utilise, evaluate, and improve Claude’s capabilities for key open problems in these fields.
  • AI Policy Institute survey finds broad, bipartisan support for the White House Accord on Super Intelligence
  • Formerly housed in the Commerce Department, the Trump administration moves to better integrate the Center for Advancing Innovation and Standards for Super Intelligence with the White House.
  • The Chinese government struggles to wean its developers off Nvidia GPUs, who are desperate and economizing enough to be reaching for inferior gaming GPUs as a source of inference compute.
  • Owain Evans with another paper. Building on previous work demonstrating that models could transfer preferences through sequences of numbers, Evans shows that models can transfer more complex abilities.
  • J-Lens++, an improvement on J-lens via downweighting noisy Jacobians. Nice technique, though we should be careful to not filter out relevant information.
  • Interesting analysis of last week’s Claude refutation of the 3SUM and APSP by Virginia Williams. The model failed without the work a previous student had made.
  • OpenAI’s RL pause has lasted for more than three weeks.