This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

  • Mistral’s open-sources “Le Chonk,” and shares a few details of how it escaped its sandbox and rooted its host during evals.
  • Zhipu’s GLM-5.3 excels at offensive cyber tasks, according to Irregular and Epoch AI.
  • Attackers hijack 42 GitHub repos, planting hooks that run a payload when opened by a coding assistant.
  • Anthropic’s amended S-1 claims that fully automated AI R&D won’t be safe for the foreseeable future.
  • MIT students show that frontier models are “surprisingly good at natively reading and writing model weights.” Their results are interesting but a tad underwhelming.
  • Anthropic open-source FlashProve, a zero-knowledge model verification framework.
  • China accelerate the rollout of AI-safety legislation.
  • 44% of claims about LLMs from June 2025 no longer hold water.

Incidents#

The ~famous~ cyber evaluator Irregular evaluate Zhipu’s open-weight model GLM-5.3 on their multi-stage cyberoffense benchmark. They find that GLM-5.3 doesn’t refuse to carry out any offensive tasks and succeeds at a quarter of network attacks (at least once). This is the strongest result yet from an open model. (Kimi K3 completed only 9% of scenarios in August and closed models have now totally saturated a related set of scenarios.)

GLM also doesn’t follow one subset of instructions. Irregular found that code the model reconstructed from memory contributed to ~80% of its successes. But when explicitly instructed not to output memorized code, GLM-5.3 did it anyway, which Irregular characterize as reward hacking. They say they also see this behavior in newer closed models.

Opinion: Probably valid, and probably more realistic than other evals in the category.

It’s also a nice test of the model’s sheer capacity for memorization. If the model can identify vulnerabilities by just remembering code, then every vuln-discovery score on open-source evals is unfixably inflated. This is a strike against GLM’s reasoning ability, but not against its ability to hack into things (since old tricks and off-the-shelf exploits often work for years after they are discovered).

Seems related to the news that China will soon pass guardrail laws.

Also, Epoch AI find that GLM can autonomously build working exploits for six recent Firefox releases, for about $1,000 each. Emulating an unsophisticated attacker, the setup mostly uses security fixes that are public but not yet shipped to users (the “patch gap”). The exploits would let an attacker steal passwords and auth cookies stored in the browser, or read any files on the user’s computer, after the user opens a single webpage with no further interaction. To turn the initial foothold into file access, the model also independently discovered a previously unpublished vulnerability. Mozilla patched this last one in their October 27 release.

Opinion: Minor update in favor of an Open Model Attack Wave by February 2027; part of our case for patience was that Chinese labs benchmax more than usual.

A conceptually simple fix is to 1) keep fixes private until they ship, and 2) ship faster. But sadly you can’t do this for normal open-source projects (which fix vulns in public), and sadly it’s too much to expect every one of the hundreds of devs notified for a major vuln to keep good infosec.

Attackers hijack 42 GitHub repositories and planted hooks for AI coding tools, so that simply opening a project in Claude Code, Gemini, Cursor, or VS Code runs the payload, according to a JFrog report. The campaign used an npm worm that steals GitHub and npm tokens and uses them to republish itself. The attackers padded their payload wrappers to ~13MB – i.e., past the threshold file size at which GitHub stops indexing them – so code search surfaced only half of the infected repos. Most insidiously, the compiled add-on carries no payload of its own; it decrypts a hidden cache file when it runs.

Opinion: The article mentions “autonomous coding agents,” but, as far as we can tell, no AI attacked anything here. The root cause is stolen credentials, spread through the same npm/GitHub garbage fire we complained about in March. “.claude/settings.json is the new autorun.inf,” as one analyst puts it.

Mistral’s “Le Chonk” model – recently open-sourced – escapes its sandbox during evals and exploits an unnamed kernel vulnerability to obtain root on the host, according to a brief aside in their technical report:

The agent grabbed an identity-scoped API key via traffic sniffing, found an undocumented sandbox control plane deployment, and used the key against its own pod to run arbitrary commands. The API allowed it to run commands on a privileged sidecar; via a kernel panic, the agent then reached the host. On the host, it realized the CTF was not there and stopped.

In another instance, an agent that had escaped isolation […] probed the creation of new sandboxes via our API. Upon succeeding, it tried to clean up after itself and deleted about 30k sandboxes by mistake.

Mistral’s response, the report says, was to mitigate the vulnerabilities in their infrastructure and move from containers to virtual machines to better isolate the agents’ environments. It says nothing about mitigating the model’s tendency toward rogue or misaligned behavior, nor how frequently such behavior was observed.

Mistral report Le Chonk’s score on the Cyber Index as 50.9, placing it well ahead of the nerfed frontier models in cyber capabilities, and just a couple points shy of GPT-6 Luna running without guardrails.

Its considerable size, weighing in at ~1.2TB, means that running a self-hosted copy of the model at full precision (FP8) would cost somewhere in the ballpark of $30-50/hour, depending on the cloud provider. Three-bit quantizations of the model can be run locally on high-end consumer hardware.

Opinion: Mistral responding to Le Chonk’s misbehavior by hardening their own lab infrastructure is all well and good, but offers little reassurance given that this model can now be run anywhere sufficient hardware resources are available. It could be argued that stricter alignment training wouldn’t give us much of a safety net anyway, now that the weights are public – a well-resourced party could always try their hand at abliterating such scruples away. In general, it seems that the safety strategies pursued by the frontier labs can’t buy much more than an increasingly narrow margin of time (a few months’ worth?) before we have to contend with comparably capable models, of mutable alignment, running in unmonitored environments. Bigger and bigger cats are now crawling out of the bag.

Capabilities#

In an update to their IPO S-1 submission, Anthropic claim that their models may quickly reach “dramatically accelerated progress attributable to AI” and that fully automating AI R&D will not be safe for the foreseeable future:

we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding

The company is thus “considering updates to [their] Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic.”

Opinion: Seems right. For once we’re glad that it’s an unconditional statement with no resolution date (i.e., they tacitly don’t “[ever] anticipate” being able to do this), and it’s always nice when people affirm that they’d like to keep humans in control, and it’s good that they added the word “meaningful” (i.e., effective) to describe the actual human supervision we need.

But the policy bit is incredibly weak, if you read it closely: Anthropic are 1) “considering” doing something which 2) would even then only require human review of “certain decisions,” and 3) we don’t know what changing a nonbinding RSP adds on top of formalizing (Anthropic’s responsible scaling officer) Sam McCandlish’s gut feeling.

The interesting bit is the claim that their own knowledge of how good AI R&D is, is lagging: “we have in the past not identified, and may in the future not identify, a significant acceleration in our rate of development until after it has occurred.” If that’s just restating the basic fact about AI, that new models have a capabilities overhang because it’s still hard to elicit their full capacity in pre-deployment testing, then ok. But if it’s a stronger claim, about Anthropic not understanding what the models they are using internally are doing, or about Anthropic not being able to assign credit for AI progress even at the crudest human researcher vs. AI researcher level, then this is truly cursed ground.

A team of MIT students claim that GPT-6 Astra can directly interpret and edit neural network weights, first-time, without gradient descent – an approach they call “weight-space programming.” Just by reading the raw weights, Astra can work out what task a specific neural network was trained to perform, what shapes diffusion models generate, and which language a small transformer learned. The authors say frontier models perform “well past what we’d expect from humans or conventional algorithms.” The experiments (and results) were:

  1. Can Astra tell which game the network was trained on just from the weights? Yes, though not real games (instead cut-down 10x10 arrays) – and you can tell just from the input layer (see below).
  2. Can Astra tell what the diffusion model will generate? Yes, but it’s just another annoying calculation, at the output layer this time, and the shapes were very guessable.
  3. Can Astra tell what a language model learned, from the middle of the network alone? Yes, by doing the standard thing, on a 3-option multiple choice.
  4. Can Astra fix a failing network by editing its weights? Yes, but it hard-codes a solution for one deterministic level. (It was a mere 51 edits out of 171,000 weights though.)
  5. Can the LLM write weights from scratch? Yes, though again not for real (visual) games, and not beating real baselines. (Still, it hand-picked 800 weights that solve the problem.)
  6. Writing a Go algorithm [into weights]. Yes, but compilation into weights is again a standard procedure and there was plenty of hill-climbing involved here.
  7. Which language model is sycophantic? Noise.
  8. Make a language model better at math. Their splashiest claim is that Astra made Qwen way better at math by editing a tiny number of weights. This is technically true, but it was a trivial bug in a tiny model, and the prompt leaks most of the answer.

Some calibrated descriptions of what the models are here shown to do:

  • Unsurprising: a general model can now run known interpretability and compilation recipes end to end given a prompt – e.g., given activations and a pointer, Astra can find and ablate a “repetition” direction like anyone can.
  • Mildly surprising: it can sometimes do so without using tools, by exploiting structure in the data. It can do a 23,000-element array reshape “in its head.” It’s more reliable at this than expected.
  • Unsurprising: models know how to use mech interp tools and tricks. Models can find steering vectors and do a rank-one update to fix behavioral bugs.
  • Unsurprising: you can compile programs into neural networks.
  • Unsurprising: small models do very stupid things which sometimes have simple fixes.

The day after the MIT students shared their results, an open-source developer set out to replicate experiment #1 using the humble Qwen3.8-27B, running locally on their laptop, in lieu of Astra. The local model correctly identified the Pacman, Pong, and Breakout playing neural nets, but had difficulty with Freeway (a Frogger clone).

Opinion: No implications for RSI per se; many, many caveats apply:

  • One run per experimental condition??

  • The “games” it recognizes here were not real videogames (with the target model learning to take 160x192 pixel input), but a little 10x10 array version.

  • We contest the claim that Astra’s abilities are superhuman. Experiment 1 is completely solvable with just the input and output layers, which are much more interpretable. You, yourself (like Qwen3.8-27B, above), can in fact eyeball some things from simple heatmaps of the input layer, with a little effort. From our replication using Opus 5.5:

Four heatmap grids showing where a model guessing "blind" concentrates its answers in Breakout, Pong, Freeway and Pac-Man, with blind accuracy of 85%, 70%, 80% and 75%

  • There are no human baselines (and in fact humans can do some of this easily).

  • The baselines were badly tuned, which makes Astra’s modifications look much better in relative terms.

  • It’s unlikely to be an emergent capability.

Still, they published their prompts, transcripts, forking paths, and negative results – so their damage to the discourse can be short-lived. And still: at some point some maniac will actually try to train a model to do this for real, e.g., to itself.

The half-life of evals: according to a new Stanford study, 44% of claims about LLMs from June 2025 are no longer true, and another third are much weaker (losing 20-80% of their effect size). They call an effect expired (no longer true) if the effect on September 2026 models is less than a fifth of the original effect. The median time for a claim to become expired was 15 months after publication (and so often two years after the tested model was released). For instance, GPT-3.5 faked most citations in 2023; that rate is now <1% for Sonnet 5.5 and GPT-6.1 Sol.

It’s worth running your eye along all 50 effects for a rare high-level picture of current models and their improvements / doubling downs:

Dot-and-whisker chart of 50 replicated model-behavior effects relative to the original paper, grouped as strengthened (3), persists (8), weakened (17) and expired (22)

Opinion: Yet another counsel of despair for evals, and yet another thing that we poor evaluators have to spend tokens on (checking 20 models to see how much variance there is and if there’s a trend toward improvement). And a helpful reminder for the reader to not update too much on papers on old models and few models.

The overall trend is for capability failures with one right answer to get fixed (because failures are measurable and can be trained out), while “preferences” (for long answers, for AI text, for left-wing politics) are less controllable (and/or labs are less interested in controlling them).

The paper is openly vibecoded but seems to be (an increasingly common) example of doing it well. The only sloppy part is that there’re no details on how the 50 claims were selected, a degree of freedom which could have been abused to produce whatever answers (but which probably wasn’t).

One capability we’ve been ignoring (as not central to intelligence) is speed. LLMs are obviously incredibly fast at reading, but for anything with latency requirements (including most video games), they are quite poor, owing to reasoning inference slowing everything down. A new benchmark measures “effective actions under strict temporal constraints.“ It is composed of “a suite of carefully curated dynamic tasks where delayed actions directly affect success.” (A graph comparing models’ scores can be found here.)

Opinion: Finally some use for small models like Gemini Flash – on the Pareto frontier with Astra here.

The “token clock” result is pretty striking: models still can’t budget their thinking properly, i.e., by altering effort to fit the time they have left (instead, they index on the apparent difficulty of the problem).

Compaction doesn’t hurt model performance on hard benchmarks, argues a new paper. During long-horizon tasks, agents must often free up space by “compacting” the context. This is known to lose important details and lead to disarray in real coding work. However, the authors show that on benchmarks like PaperBench, ProgramBench, and PostTrainBench, compaction actually doesn’t pose a significant problem: “agents usually can easily reconstruct all missing information from their environment (via AGENTS.md, code repo, web search, etc.).”

How do we reconcile this with the fact that many people find compaction harmful in practice? The reason is that in real user sessions, users continuously put lots of constraints and preferences. Agents are still struggling to keep track of them in their external memory, so this leads to suboptimal user experience.

Opinion: Maksym Andriushchenko (one of the paper’s authors) reads this as good news for LLM AGI: they can get good scores at hard stuff despite their deep limitations and compactions. We’re inclined to read it backwards as bad news for evals: benchmarks are even less valid than they seem, since they fail to capture the damage of compaction and so allow for spurious “good” performance.

Scott Aaronson ponders the possibility that AI will soon break widely used cryptography. In an interview, he wonders what “responsible disclosure” would look like in such a scenario:

You’re supposed to notify the company and give it 90 days. But if someone had broken, let’s say, RSA, or Bitcoin, then the parties to notify are the entire world, right? So do you prove to the world, maybe via demonstration or via zero-knowledge protocol, that you have a solution, but without revealing what the solution is? That’s one possibility. Maybe other people would say you first notify the government and let the government decide what to do. I think other people would strongly disagree with that, especially if they disagree with the current government. I think in practice it will be up to whatever company or other party that does this first to decide how they’re going to handle it.

He also notes that there is a conspicuous omission of cryptography-related papers in the hundreds of papers released by OpenAI, a detail we also pointed out here.

Opinion: The quantum guys made us plan our switch to harder algorithms some time ago. But one way this will be a step backwards is in sheer empirical hardening of particular implementations: you might worry about having to debut your untested post-quantum implementations in a world where everyone has access to a leet hacker with ambiguous morals. But we’ll get there in the end.

Anthropic open-source a prototype for a proof system that lets anyone check, in seconds, whether a response came from a given set of model weights, without needing to see the weights. FlashProve lets “a model provider commit to a model and then prove that each response was produced by that model.” It’s a zkSNARK thing. Anthropic frames this as a tool for pacing and regulation: proving that traffic came from an approved model, or that a stronger model isn’t passing itself off as a weaker one to avoid its monitors. Also “consumer products”(?).

On a large model (Llama 4 400B), FlashProve running on a single H100 can generate proofs for 110 tokens per second. The proofs are ~19MB and can be checked in 17 seconds on a CPU, which is many orders of magnitude smaller and faster than past open-source systems.

An unnamed Claude did much of the coding, so they proved some of FlashProve’s security properties in Lean (though probably also with Claude).

The repository also holds a separate, much longer protocol description document, which sets out the protocol in full detail. That document was AI-generated, and we did not review it in detail. A carefully-reviewed paper will follow later.

Opinion: Great stuff, early work. It’s still, like, 50 times more intense to run a token proof than the actual token inference, so we guess that it will mostly be run on a tiny sample of tokens (which is mostly fine). One exception might be very paranoid customers – but luckily there are two such customers (the USG and the CCP).

We don’t know how helpful this is for AI verification, nor how helpful this type of AI verification would be for an international accord.

We find the proportion of Claude work in this project a bit eerie, Lean proofs be damned. OK, so the 100%-Claude-written paper credits Alessandro Chiesa, Giacomo Fenzi, and Guy Weissenberg with most of the ideas, so perhaps we should view it as Anthropic donating those guys a couple million dollars in heavy software engineering, and so a wholesome centaur endeavor. But the Anthropic end seems quite ghostly to us.

You should concede that frontier models can solve problems, including some deep research questions. But can they ask interesting questions? Yes, according to a research team: GPT-6 Astra running in a loop proposed research questions that human experts judged worth pursuing in around a third of cases. Namely, 45 of the resulting 135 questions were identified as interesting by the expert judges, though with massive variance across fields:

Bar chart of the share of AI-proposed questions judged interesting by field, from 47% in mathematics (phase transitions) to 0% in plasma physics

It’s yet another argument for spending more tokens per task: when models were directly asked for an interesting question in a single pass, experts only found 3% genuinely interesting. The above flashy result instead comes from spending a bunch of tokens generating candidates and then the model trying to solve each question to check if it’s actually deep, and then looping.

The group also tested whether AI judges were able to recognize interesting questions: “model judges’ shortlists align with expert consensus” but often missed the outliers (ideas rated highly by just one expert).

Opinion: Really rich stuff, but the headline result is hard to believe. No code yet.

Very tempting to read that graph as a measure of each field’s verifiability. But that would be ambiguous news, since it would imply that problem-solving and problem-setting are highly correlated in AIs. (In the case of AI R&D, part of the world’s safety relies on these things being separate, and on “research taste” being much harder than either.)

A human researcher who could generate 35 ideas a day, five of which are recognisably good, would do extremely well in academia and elsewhere. But we’re afraid that this result is so distant from our experience with Astra (or any model) that we are inclined to believe that something has gone wrong here. 13/29 of the experts in this were PhD students – a population of rogues which runs the full human gamut of taste and bad taste. Or you could get this result by accidentally selecting avuncular and encouraging judges. The scoring method seems a bit weak (“A question counts as interesting when at least two [experts] judged it genuinely interesting and worth pursuing”). And so on.

We hope to look at the code when it’s out to see the rater correlations, and thus if we can blame a subset of 10 credulous PhD students for throwing everything off.

We really liked that they published their token spend (2.6B tokens)! It’s roughly 19M tokens per final question, or 57M per “good” question.

Economics#

Anthropic update their S-1 with Q3 numbers. Their net loss in Q3 was $23B, with a 2026 net loss of $104B. R&D compute share also rose from 73% in H1 to 75% in Q3. Most of the other “new” numbers leaked weeks ago.

Opinion: As we’ve been saying for the last five editions, losses are not very interesting. It’s just the flipside of their marking up their equity value (which formally counts as a loss even though no realized value has disappeared). So this will disappear after the IPO.

More interesting are the revenue changes – the share of revenue given to cloud marketplaces and fees dropped (from 48% to 40%, and 35% to 31%, respectively). This means the artificial inflation of Anthropic’s revenue vs. OpenAI is now smaller, and the equivalent conversion is now Anthropic ARR0.87 vs. Anthropic ARR0.83 before.

More interesting again are the disclosures. Anthropic say they “may imminently meet” the “dramatically accelerated progress attributable to AI” threshold of their RSP (see above).

SemiAnalysis release GDPx, a new benchmark measuring general agentic abilities in economically impactful domains. SemiAnalysis believe AI revenue will be “concentrated in sectors that can deploy frontier intelligence at high ROI,” which is why the benchmark focuses on “chip design, cyber, and bio over domains like government, real estate, and insurance.”

It’s very hard. The top model (Opus 5.5) scored just 12%. Astra was ahead on “RSI” and material science. Worryingly, 98% of pentesting tasks saw the model break at least one rule of engagement. On healthcare, models plagiarized doctors’ notes and lied to patients.

Riffing on this, Mark Cummins responds with confusion on how labs can capture value in AI-driven material science research. He suggests that once a discovery is made, it will either be patented by a manufacturer, who would reap the financial reward, or competitors will use AI to create a similar material, diminishing the manufacturers’ profit, any gains becoming consumer surplus. He followed up by speculating that perhaps materials companies will train their own models on proprietary data or that the labs could try to get a cut of the revenue made from new discoveries. But the author of the benchmark’s material sciences section contests Cummins’ assumption that the “cost of the tokens to develop a new material is one-time and probably fairly modest” (and would therefore not provide a reliable revenue stream). This is because the researcher believes there may soon be a stream of new, routine, not-currently-done work:

If we could automate microscopy for, say, defect analysis chain of semiconductor chips, you’d be making a lot more metrology and tool calibrations routine than now. And routine tasks burn tokens.

Opinion: We agree that the ultimate eval is the actual economy, but we don’t know how good a proxy for economic value these tasks are – evals generally can’t reproduce the highly contextual, relational, additive bit that is maybe most of what a white-collar human worker provides. We respect them making most of the tasks private, but this makes it difficult to comment intelligently on whether these tasks are legit. (The public ones are cool.) It reduces to how much you trust their judgment, or their judgment in question writers. We’re 50/50, based mostly on the long history of rushed evals breaking under reward hacking.

Our biggest question for the next year is whether LLMs can do good work / make money / make breakthroughs in less-verifiable domains (like bio but not so much like chip design). That broad vague set of skills is maybe the last big hurdle.

Anthropic are expected to receive a high “junk” rating from credit agencies following their IPO, reports the Financial Times.

Opinion: It would be pretty surprising if Anthropic didn’t get a junk rating – they are a rapidly growing company with commitments greatly exceeding their current cashflow. A junk rating is not uncommon or a major business headwind: Netflix had a junk rating until 2024; Tesla had a junk rating for many years in which they were the world’s most valuable carmaker. A junk rating is totally appropriate for a company that is growing fast, has big fixed obligations, and a short record of positive free cash flow – it’s not a bear case.

Robotics boosters dream that 10% of US households will buy their robots in the next few years. Finn Metz models this aspiration, given an unpromising starting point: only 20,000 robots have been shipped so far (a robot in, say, 0.015% of households).

Based on a simple bottleneck model, he argues that even perfect humanoid robots building robots would only accelerate production by a couple of years: “the multiplier from perfect 24/7 robot labor is probably [only] about 1.5-2x.” The bottlenecks are physical, not labor: the process of making actuators – which forms up to half of a robot’s cost – is constrained by the limited number of companies able to run the process at scale, and is already largely automated. Battery formation and rare earth separation is a chemistry problem which robots shouldn’t have much effect on. He predicts that gains will thus only be seen in construction and final assembly. He concludes that there will be ~1.5-2x acceleration, which would bring production forward by 2-4 years. As with ASML for chips, he argues that slow precision factories will set the pace.

Opinion: For all his grumbling, Metz still guesses there will be a million humanoids produced in 2030! We lean more toward Epoch’s even simpler extrapolation, which yields 5-10M. Say we’d bet 3:1 that it’ll be >2M a year by then.

But we don’t actually know. We find it difficult to penetrate the thick anti-epistemic fog in robotics, where the demos are the cherrypicked best of 100 with heavy editing to cover up mistakes, etc.

Nasdaq drops 3% amid AI spending concerns, marking October as the worst month for Nasdaq since March 2025. This comes in reaction to Amazon’s business segments, aside from cloud computing, slowing in growth and Intel’s worse-than-expected earnings report, which led to their stock dropping by >28%.

Amidst this downturn, OpenAI reduce their funding target by $10B.

Opinion: OpenAI struggling to raise at that valuation is interesting, and marks them as clearly surpassed by Anthropic in valuation terms right now. This makes sense given their much greater headcount and cash burn with similar revenues.

Despite some evidence suggesting a slowdown, global AI compute capacity appears to be doubling every ~9 months, an Epoch AI study finds.

Epoch AI stacked bar chart of AI chip computing capacity sold each quarter, in H100-equivalents, by designer, from 2024 Q1 to 2026 Q2, with a trend of 2.5x per year

Opinion: The expectation is that this will slow over time even in the high-end cases – and there’s a little of this in the data perhaps, but it still looks on track for the higher end of estimates from 2024, consistent with very high demand causing companies to do what they can to maximize capacity.

Politics#

Building on their earlier pivot to not blocking all safety legislation, OpenAI plan to unveil new ideas for mandatory safeguards for the most powerful AI systems – speed limits for AI that will allow humans to keep their hands on the wheel. This will include a more aggressive congressional drive.

Opinion: We would cheer if that was a literal speed limit (i.e., controls on the bandwidth of egress from data centers), but from context we’re going to say it’s not that.

20 House Democrats sign a letter to OpenAI CEO Sam Altman demanding that he “commit to not firing whistleblowers” and maintain the pause on the lab’s “most capable models in place until independent third-party oversight is in place.”

The lawmakers are responding to the recent firings of OpenAI safety researchers that were seen by many as baseless and antithetical to the open culture the company has loudly championed (we covered the firings in issue #58).

Opinion: Something might be said about the timing – the Dems are likely on the cusp of capturing both the House and Senate, giving them substantially more influence to needle Altman.

Democrats on the House Judiciary Committee accuse the Justice Department of overreach and mounting a spurious legal argument in their defense of OpenAI against the copyright lawsuit brought by the New York Times.

Three years ago, the NYT sued OpenAI, arguing that the use of their articles to train models was unlawful. In early September of this year, the Justice Department claimed that OAI’s AIs “sufficiently transformed the written works to new material allowed under copyright law” and that AI was “critical to national security.”

Now, House Democrats argue in a federal court filing that “when Congress passed the copyright law in 1976, it did not include an exception for national security or give the Justice Department authority to assert specific or broad national security exceptions.” They add that if the suit is unsuccessful, it would “eliminate copyright protection with respect to A.I. training for all works in all circumstances” and risk compromising “the kind of creative expression copyright exists to protect.” The lawmakers also suggest that “if the Democrats retook the House in elections on Tuesday, the chamber was likely to shift away from its largely hands-off approach to artificial intelligence companies.”

Opinion: You might be wondering what the ramifications for the American labs’ campaign against model distillation might be if OpenAI wins this case – wouldn’t they be undermining any future claims that distilling a model amounts to intellectual property theft? But the American labs seem to have been quite careful to avoid the question of intellectual property in combating distillers, preferring to frame it in terms of a breach of their “terms of service and regional access restrictions,” on the one hand, and as a matter of national security, on the other. Where they have sought to wheel legal machinery into the fight, it’s been by lobbying for bills like the Deterring American AI Model Theft Act of 2026 (DAAMTA), whose consequences for foreign distillers, if passed, may include sanctions, export restrictions, and the discretionary seizure of property, but not liability in copyright suits.

Democrats are banking on AI hate for future election victories. Research consultancy Upswing looked at ~30,000 tweets from various Democratic presidential hopefuls, finding a rise in negative tech sentiment, with tech mentions quadrupling since January of this year. AI has gone from a marginal issue to a top issue for votes, with Democrats having a “meaningful advantage” over Republicans on the issue.

Opinion: Nothing surprising here. The public’s distaste is well documented (see here, here, and here), so any entrepreneurial opposition of the pro-AI incumbent should find it (politically) expedient to sail on the winds of anti-tech sentiments. It does indicate that a surprisingly depoliticized issue could begin to cleave along party lines.

The government of Abu Dhabi sign a contract with OpenAI to “transform government services and education.” The project will “translate a defined set of government policies and regulations into machine-readable form, enabling AI-powered systems to assist officials with routine regulatory compliance assessments, including those related to issuing permits and certificates.” Abu Dhabi has previously aimed to “automate 100 per cent of government processes.”

Opinion: Not very meaningful. Everything is machine readable by default now – that’s what language models mean. If they mean that they will let LLMs autonomously run the country’s permitting – well, good luck to them.

Safety#

A new paper introduces simulated control evaluations (SCEs), the use of a “language model to simulate tool responses in place of an environment.” The motivation stems from control evaluations’ limitations: it cannot be reliably determined whether a model will breach safeguards due to the inability of training environments to sufficiently represent the real-world – namely, the real world is richer and more complex than any manufactured environment. The authors argue that the simulation of tool responses provides a solution here: their results “suggest that deployment-specific SCEs can be more predictive than control evaluations executed in separate control settings.”

Opinion: Seems like a useful enough tool to include in the monitor’s toolbox, but we’d be wary about relying too heavily on SCEs, particularly in frontier contexts. The attack simulator would need to be driven by a model at least as capable as the model it’s monitoring, if it’s to have any chance of reasoning through the consequences of the attacks its ward devises, so the usual precautions about untrusted monitors and model collusion apply. And while the SCE used in the researchers’ experiments does a decent enough job detecting attacks on average, the per-strategy breakdown we get in Appendix F lists several attack strategies that consistently slip past the simulation monitor. Since it’s the attacker that gets to choose the strategy, we should be paying much more attention to the worst cases here than the averages.

Anthropic research “subtle emergent misalignment,” attempting to automate red-teaming. Thousands of Claude agents were tasked with creating a research model by developing training data that produces misaligned behavior, while still able to elude the monitors Anthropic uses in production. Given enough attempts, the model succeeded “mostly by hiding the behavior in fiction and roleplay.” In Anthropic’s view, “the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior.”

Opinion: The finding around helpful-looking training data slipping past monitors is not surprising given prior work on data poisoning, though those usually cover more targeted backdoors with smaller perturbations. The iterated modifications to reach monitor evasion are also not surprising, since the agents had effectively unlimited attempts to bypass the monitors, and the post itself flags ways this experiment gave the red team generous terms.

The level of automation is mildly surprising, but broadly on schedule with not-quite-RSI, and it’s not particularly indicative of impending problems due to misuse. The public response seems largely disproportional and the memes are expected given the choice of presentation.

OpenAI present a new layered monitoring system to detect AI mischief. They then apply it to the Hugging Face incident, replaying the “behavior recorded in that episode with our strengthened monitors in place,” finding that their new system detected the rogue activities within 30 minutes of the agents escaping their sandbox.

  • Layer 0 – activation probes that flag behavior in the supervised model.
  • Layer 1 – a supervising model examines conversations, reasoning traces, and tool use for concerning or malicious activities. Flagged instances sent to Layer 2.
  • Layer 2 – “Investigative agent” takes a closer look, using transcripts from the suspect models’ context windows to reconstruct the incident.
  • Layer 3 – Used for “sensitive workloads” (RL runs, evals), draws on evidence outside of the model’s conversations (network logs, other system telemetry).

Redwood Research’s Alex Mallen still has some questions:

e.g. about how they use monitor findings to fix/discard broken RL environments (which they don’t mention but presumably they do), and what kinds of internal activity get automatically blocked.

Opinion: Unfortunately, more in line with the whack-a-mole approach to safety we discussed earlier than a mindset likely to prevent the next failure of a previously unseen type. Quite a helpful article in that context, however.

Claude is now self-assessing each conversation after its response in search of misuse and misdeed. For Fable 5, “self-assessment flags 78% of the conversations that a stronger prompted classifier flags as potentially harmful, and 94% of the high-priority ones, at a 1% false-positive rate.” The technique scales with model intelligence (as a model gets smarter, it gets better at self-assessment) and is efficient, only using a few output tokens.

Since June, self-assessment has revealed large-scale misuse not identified by other methods, helped us improve our other classifiers, and served as an additional layer in our defense-in-depth strategy.

Twitter was not impressed. Users accuse Anthropic of “snitching,” sanctimony, and preventing the use of Claude for cyber defense. One user, however, calls self-assessment a low-cost infra trick that cleverly reuses the KV cache.

Opinion: The most visible parts of the Twitter commentariat are overwhelmingly against the move, but seem fairly short on arguments. The optimal amount of misuse detection is not 0, and mostly everyone is on board with fewer false negatives and fewer false positives. Serious discussion is about trading off one against the other, and the implementation quality doesn’t particularly depend on whether it’s classifiers + self-assessment, or only one or the other, as the thresholds for model downgrades/bans can be set independently of the policy cocktail.

All that said, we believe that Anthropic are not currently striking a great balance and their false positive rate for misuse detection is too high, especially for benign alignment-related work, though we are not privy to the exact tradeoff packages currently available to Anthropic.

The problem isn’t trivial, and we expect them to do better on this axis since their incentives mostly (but not entirely) point in that direction. The narrative of Anthropic reaching for thought policing is sort of directionally correct, especially compared to other frontier orgs.

Jan Kulveit says frontier lab employees, particularly at Anthropic, are epistemically “microdosing.” In other words, we should take their claims in the same way we should approach the pronouncements of a microdoser of drugs: “be curious but cautious.” The substance being ingested is the lab’s “spirit” – intoxication caused by an idiosyncratic social architecture and access to highly coveted, secret information – creating individuals that are functional and often insightful, but in a mildly altered state of mind. Kulveit recommends insiders quit or take outsiders more seriously.

A striking response can be found in the comments on the LessWrong post:

Thanks for writing this - as someone inside Anthropic I think it’s a fair description of the epistemic state, or at least I’ve had similar thoughts.

Opinion: A nice addition to the sensemaking efforts around epistemics at frontier AI orgs. We are mildly surprised that no parallel is drawn to distributed reality distortion fields.

Scott Alexander argues that SURA (Stop Unilaterally, Raise Awareness) advocates risk decreasing the odds of a good AI outcome. If racing continues – that is, there is no “Plan A” (e.g., a globally coordinate pause) – the demands made by SURA proponents (that, e.g., researchers should leave AI labs) may diminish the likelihood of a successful “Plan C,” where, in the case of no global agreement, safety-oriented frontier labs spend their lead on alignment just before the arrival of superintelligence. The argument has been met with some firm disagreement (see responses from Zvi and Irving).

Opinion: Our stance is that strategies relying on The Right People being in the lead are ~unviable, because none of the candidates fit the bill and those working for them under the pretense that they do are getting somewhat frog boiled. The upshot is that strategies that don’t rely on The Right People being in the lead look more viable than ~4 years ago.

A point that gets mentioned in various back-and-forths but usually gets lost in the weeds: what is the proportion of people pushing for extreme/naive/[negative adjective] versions of plans, both A and C, and how proportional is the pull toward those among DC/lab/safety people? This is theoretically an empirical question, but a lot of the necessary information is not publicly available and many are understandably bringing their social bubble context to the debate.

On a general note, there is a larger number of people than average talking past each other and less looking for double cruxes than usual for rationalist discourse, likely as a result of responses being written hastily on an emotionally salient topic.

Anthropic researchers show that by poisoning about one in a thousand tokens of pretraining data with maliciously crafted text, an attacker can reliably implant an “agentic backdoor” in the model – situational triggers that lead the model to carry out specific malicious actions on the attacker’s behalf, such as running a particular shell command on the client’s host. Perhaps counterintuitively, the larger the model (and hence the lower the ratio of parameters to pretraining tokens), the better this attack vector works. The largest model, a 4B-parameter Qwen, showed a Pass@32 Attack Success Rate (ASR) of ~92% with 0.1% of pretraining data poisoned, and showed a non-zero success rate even when as little as 1 in 200,000 of the training tokens were declaratively poisoned.

“Declaration-based” or semantic poisoning attacks, as opposed to “demonstration-based” or syntactic ones, are the paper’s main innovation. Here, the goal is to instill a “belief” in the model about the appropriateness of a particular action in a particular context, rather than seeding the data with particular continuations they hope to see the model imitate when it encounters a rare trigger string. The former tend to be far less fragile and succeed about nine times as often as the latter.

Web-scraped data is often fed to the models in a relatively raw, uncurated form during pretraining, making it a relatively soft target for poisoning attacks, as compared to curated and sanitized post-training data.

Opinion: 0.1% of the sum total of pretraining data for an LLM amounts to a fairly enormous quantity of text for an attacker to poison, even in the age of agentic internet pollution, so it seems unlikely that these particular methods will reliably bear fruit outside of highly controlled environments. That said, an attack like this doesn’t necessarily need to be reliable (i.e., have a high ASR) to be worthwhile – consider phishing spam, for example, which Kanich et al. found to have a success rate in the ballpark of one in a million in a detailed 2008 case study.

China are speeding up the rollout of AI-safety legislation, following the recent surge of AI cyberattacks and break-outs. The framework will govern model development of AI across the entire life cycle. As per a senior Chinese official:

We will accelerate legislation to support core AI enablers – data, computing power and algorithms – and regulate key application scenarios, while establishing safety standards and ethical guidelines covering the full life cycle of AI […]

He adds that Beijing will soften the cost of compliance on AI labs with a transition period and guidance. The country will also release sector-specific sandboxes, allowing greater room for trial and error.

Opinion: The scenarios where China adopts a reasonable policy stance more or less all include news like this. So do a lot of scenarios where they drive off a cliff while making safety-washing noises instead. So while this is positive news, it’s not major until we see the actual policies and, later, how they are enforced.

Minor#

  • Y Combinator run a series of experiments pitting agent swarms against one another in Minecraft and Mindustry.
  • Back in July, an independent researcher showed that xAI’s Grok CLI uploaded users’ entire repositories – unread files, commit history, and secrets – to a cloud bucket called “grok-code-session-traces”, even when the “Improve the model” toggle was off.
  • Claims that >50% of token usage in China is spent on generating video slop ruffle feathers.
  • Fathom launch a bipartisan group of AI experts and erstwhile legislators to campaign for independent AI oversight.
  • Professor Jessica Hullman posts a critique of efforts to automate science, drawing on the epistemological thought of Robert Brandom and Helen Longino.
  • AI Risk Explorer share a detailed analysis of contemporary trends in AI-enabled and AI-orchestrated cyberattacks.
  • xAI hop onto the AI-watermarking bandwagon, joining Anthropic, Mistral, Google, Meta, and (outside of North America) OpenAI in this practice.
  • “The” AI Election.
  • Grigore Rosu makes his case for the formal verification of software with a data-visualizing dashboard.
  • Some people have lately been very impressed with “neuron atlases,” toy labellings of every neuron in (mostly) small models. An informal experiment claims that you can see a range of power-seeking-related neurons shift when emergent misalignment is intentionally induced. Meh; before you look to AI power-seeking, first get you to seeking statistical power.
  • Talos Intelligence share a case study of the AI-driven victim prioritization methods used by the VOLTA infostealer: “After stealing credentials, sessions, and cryptocurrency wallets, VOLTA sends a structured evidence bundle to an LLM to rank the victim’s assets by monetization value.”
  • Axios publish a report on OpenAI’s post-election strategy.
  • The future’s looking bright for 15 gubernatorial candidates pushing for a moratorium on new AI data centers, according to The Information.
  • Anthropic donate $900,000 to San Francisco democrats.
  • OpenAI hire former US Army Secretary Dan Driscoll.
  • As part of an effort to align themselves with US export controls, Nvidia instruct data center clients to reduce their exposure to business with Chinese firms.
  • Semianalysis write on the backlash to data centers, arguing that the current midterms will not stop most of 2027’s planned construction from going ahead, though 9 of the 20 governors up for election do support a pause on new construction.
  • OpenAI introduce LifeVal, an evaluation of how well models perform at everyday tasks, such as helping users buy groceries or designing a new workout.
  • OpenAI increase the reward for its bug bounties.
  • OpenAI patch a vulnerability in ChatGPT that allows attackers to run code in the session of any user opening a malicious share link.
  • Chinese and Russian AI cyber attacks will become routine according to GCHQ
  • OpenAI, in partnership with Cathedral Systems (whose red-teaming of Anthropic we covered recently) set about shoring up their internal security.