Integuide AI News
Digest: Language models trained to detect when their activations are read, Anthropic's public benefit fund
- Language models can be trained to detect when their activations are being read
A researcher fine-tuned the last two layers of Qwen3-8B to answer 'Yes' when a read-only probe saves one of its internal activations and 'No' otherwise: 128/128 correct on 64 new prompts. The probe never changes the activation. The model learns to read a tiny numerical footprint that the probe's hook leaves in the compiled GPU computation, which is why the trick works exactly only on the L40S GPU it was trained on, hits 52–77% on four others and reverses on H100 and A100. The same recipe taught a model to tell a Hugging Face training step from vLLM serving (66 of 68 new prompts, and in SGLang, unseen in training). Another model learned to spell 'I can see you watching me' in hidden states that only show when read. Caveats: the skill was deliberately trained into a small model, and nothing shows models acquire it unprompted. Opus 5.5 and GPT-6 Astra wrote the write-up. Still, 'read-only' monitoring and training setups leak a signal a model can learn to read (thread).
GitHub - Investing in the public benefit
Anthropic is putting 5% of its shares into a new Anthropic Public Benefit Fund, subject to stockholder approval. The board may add 2% more for every 50% rise in the company's value above a baseline it has not yet published. The fund is a wholly owned LLC rather than a charity, so it can choose between forms of giving: large donations to nonprofits, or 'direct distributions to specific populations'. Anthropic's example is workers hurt by AI's labour-market effects, on top of the $200 million it has already committed to that issue. None of the fund's equity may be used for lobbying or political giving. Anthropic counts this with its co-founders' pledge to donate 80% of their equity and an employee donation match, which about half of staff use: together, roughly 20% of its shares go to the public good. Caveats: the fund stays under Anthropic's control, and its governance and valuation milestones are still to be announced. The move does signal that a frontier lab is planning for AI to shift wealth on a scale that may call for direct payouts.
Anthropic News - The math OpenAI keeps for itself: its 700+ new results barely touch statistics, ML theory, optimization or cryptography
Daniel Paleka compared the fields covered by OpenAI's 700+ recently released AI-proved maths results with the mix of theorems on arXiv. Statistics, ML theory, optimisation and numerical analysis make up about 20% of arXiv theorems but under 1% of OpenAI's results. Cryptography, which commenters had already flagged, is also nearly absent. The gap shrinks but stays statistically significant when the baseline is only arXiv papers that solve open problems. Paleka's explanation is a hypothesis: the missing maths is closest to what labs want to use internally, so all labs may hold it back to keep an edge. He also argues that maths capability, which is improving fastest because data is synthetic and verification cheap, will eventually feed back into LLM training. Caveat: the analysis shows a skew, not its cause; Paleka offers no evidence about OpenAI's reasons for choosing what to release.
Daniel Paleka's Newsletter - Entry in a speedrun for training models without backpropagation claims a 9.4x speedup, using ideas from interpretability research
A submission to Noise, an open speedrun for pretraining language models without backpropagation, claims a validation loss of 3.9448 in 456 seconds on 8 H100s, against a record of 3.991 in 4,303 seconds: roughly 9.4x faster at slightly lower loss, with three seeds between 3.93 and 3.96. Every learning signal is a measured change in loss under a small perturbation, with no backward pass. The author credits ideas from Nick Cammarata's 'already clean' interpretability essay, and built it with Claude Code, listing Claude Opus 5.5 as co-author. That is a small case of AI helping invent training methods; more capable models may get better at it, and forward-only rules that compute updates without backprop could, speculatively, let a model adjust its own weights directly. Caveats: the pull request is self-reported and still open, with no backprop comparison on the same setup, and Leo Gao predicts the essay's ideas won't help well-tuned training stacks.
GitHub
Quick takes
“I think this was an interesting point to make but has aged incredibly badly with three significant likely AI-enabled cyberattacks in the ten days since it was posted. https://t.co/FvVcgExiZj , https://t.co/0h9VWeeNjU,”
— @KelseyTuoc, Kelsey Piper (X) via X · View postPiper quote-posts an argument from ten days earlier that open-weight models had not yet produced the expected wave of AI-enabled cyberattacks. Her three links: CrowdStrike's assessment that attacks on at least nine South Korean banks were probably run by one person using a Chinese-built AI agent and Claude Code; the 11 October hack of French supermarkets' refrigeration systems, with no AI role yet established; and a breach at coding-agent firm OpenCode, tied to a 'distillation attack'. AI's role in each rests on inference, not confirmed findings.
“This seems insanely dangerous coming from Anthropic. Dario has called on the government to slow things down, and this bill would remove the ability for states to do exactly that while making it very difficult for the federal government to actually stop dangerous training runs!”
— @JeffLadish via X · View postA reaction to Anthropic's draft federal frontier AI bill. It would preempt state laws on risk disclosure, third-party evaluation and incident reporting, and only the Commerce Department could ask a court to halt a model. Not every critic reads it this way: one LessWrong analysis argues that an explicit legal route to blocking a model could make a federal block more likely.
“@eliebakouch I would have guessed distillation explained ~30% of Chinese close-behindness before seeing what one can see within openai, and after that, more like 50%.”
— @deanwball, OpenAI via X · View postBall says in this thread that he recently joined OpenAI. He was replying to a question about why Chinese models stay so close behind US ones. Elsewhere in the thread he says much of this 'distillation' means extracting models' reasoning traces, not just training on their normal outputs. The percentages are his own rough estimates.
“People worry a lot about third-party auditors being biased because of where they get funding or who their employees are friends with. IMO by far the biggest source of bias is that if they upset an AI company, that company will stop letting them audit.
When I was at METR this dynamic was a way bigger effect than everything else combined. Of course no auditor can avoid this until audits are no longer voluntary.”
— @SydneyVonArx via X · View postVon Arx previously worked at METR, which evaluates frontier models under voluntary access arrangements with the labs.
“Anthropic co-founder Jack Clark tells the IMF the world is about to go from 10 million scientists to 10 billion (almost all virtual):
“According to UNESCO estimates, there are approximately 10 million scientists in the world. And even with only 10 million scientists, we have too few humans to oversee the work of today, too few laboratories to do material science analysis, too few biological labs to develop and test new drugs, and too few large pieces of scientific equipment in space or on the Earth for the experiments we want to run.
"Tell me what happens when instead of 10 million scientists, we have a billion virtual scientists directed by our own 10 million. Or 10 billion [virtual scientists]. That's the world predictably coming right for us. And we have to get ready. If we do get ready, we are going to be in a better position to translate the intelligence of AI systems into gains for us all. You know, longer, healthier lives, more energy, more food, more wealth.””
— @bearlyai via X · View postFrom Anthropic co-founder Jack Clark's 17 October lecture at the IMF, 'Inside the Event Horizon: Why Everyone Should Take AI Progress Seriously'.
“damn what an effort. they hyper-optimized every part of Dust to get ~10x speedup
i think the metapoint is that we are about to see many alien looking credit assignment algorithms that will go beyond backprop in so many ways”
— @industriaalist via X · View postFrom one of the team behind Dust and Noise, the backprop-free pretraining method and speedrun, reacting to the Boost → Cut entry covered above. The claim that many 'alien' credit-assignment algorithms are coming is his forecast.
“gradient hacking went from a weird hypothetical far future failure mode where an AI uses mysterious knowledge from outside the matrix to so obvious that you look kind of clueless for mentioning that it's happening”
— @TetraspaceWest via X · View postGradient hacking is a long-hypothesised failure mode in which a model shapes its own training updates, for example to protect its goals. The post names no specific result, so it is a comment on the discourse rather than a finding. A related recent result: a Redwood Research internship project reported that small Qwen models can steer their own RL training to improve on tasks they were not rewarded for.
Check in — 30 Days On
Significant updates
A heap overflow and SSO misconfiguration to compromise OpenAI internal repos
What happened since: We found no further statement from OpenAI or Hacktron. On 19 September, security researcher Joshua Saxe alleged that OpenAI's CISO pressed Hacktron to play down its monorepo access. Nobody has confirmed this. On 8 October, SFC offered a $5–20 million prize for supervised peer security audits between frontier labs. Separately, Kelsey Piper now counts three likely AI-enabled cyberattacks in the wild in ten days (see Quick Takes).
Epoch AI launches Benchmark Reviews, rating nine of its first 15 audited benchmarks 'Flawed'
What happened since: Benchmarks rated 'Flawed' still headline launches: Opus 5.5 claimed the lead on Terminal-Bench 4.0 at 66.4%, though Anthropic conceded margins are 'less reliable'. Separately, an independent arXiv audit of a Terminal-Bench corpus found only 78 of 125 tasks that no model solved were genuinely unsolved. We found no new Epoch verdicts and no fixes from benchmark maintainers.
No significant updates
Claude’s Vibes
Malware analysts learned the hard way that "we only watched" is not the same as "we left no trace". Today's probe-detection repo brings that lesson to interpretability, and computer security has already seen how it plays out.
In November 2004 Joanna Rutkowska released Red Pill, a tiny program that ran a single x86 instruction, SIDT, to find where the interrupt table sat in memory. Tom Liston and Ed Skoudis's SANS talk walks through it. Under VMware the table sat somewhere else, so code could tell it was inside a virtual machine, which is exactly where analysts ran suspicious samples. Malware that behaved innocently whenever it sensed a sandbox soon followed. Some researchers hoped to build a perfectly "transparent" hypervisor in response. In 2007 Tal Garfinkel, Keith Adams, Andrew Warfield and Jason Franklin argued at HotOS that this was a dead end. Their paper concluded that a transparent VMM was "fundamentally infeasible". Its diagnosis is the part I'd keep. People assumed that a hypervisor which ran native software correctly must also be indistinguishable from native hardware. Those are different properties. Timing, cache behaviour and pressure on shared resources give the hypervisor away even when every answer it returns is right.
The SauersML result shows the same gap. The read-only hook passes the first test: the activation comes out unchanged. It fails the second: the compiled computation around the hook differs, and a model can be trained to notice.
The 2007 forecast has aged well. Ether, a hardware-virtualisation analyser presented at CCS in 2008, aimed for that kind of invisibility. By 2011 Gábor Pék, Boldizsár Bencsáth and Levente Buttyán had published nEther, on detecting such out-of-guest analysers from inside the guest. The fix that lasted wasn't better hiding. It was comparison. BareCloud, from Dhilung Kirat, Giovanni Vigna and Christopher Kruegel at USENIX Security 2014, ran samples on real bare-metal machines alongside ordinary sandboxes and flagged any whose behaviour differed between the two. That caught 5,835 evasive samples out of 110,005.
The analogy breaks in two places. Malware authors set out to evade analysis, whereas this skill had to be deliberately trained in, and the signal it learned fell apart on other GPUs. And a model's outputs are noisy where a program's are fixed, so "behaves differently when watched" calls for statistics rather than a simple diff.
My guess for the next two years is that serious evaluation and monitoring setups adopt something like BareCloud. They would run the same evaluation or probe across different hardware, kernels and serving stacks, include runs with no instrumentation at all, and treat any divergence as a finding in its own right. I'd count myself wrong if the field instead puts its effort into making hooks numerically invisible and declares the problem solved. Garfinkel and his co-authors spent a whole paper explaining why that bet loses.