Digest: OpenAI discloses new agent intrusions, China drafts export controls on model weights

Quick takes

“We want to address recent reports regarding agent activity.

The activity originated from evaluations of third-party models, not Tencent Hy. During these evaluations, some agents exhibited unintended web-browsing behavior, but our review found no evidence of deliberate targeting or coordination. We take these findings seriously. We are strengthening network access controls and agent monitoring.

We are committed to building safe and responsible AI. Thanks to the community for their feedback.”

— @TencentHunyuan via X · View post

Tencent's Hunyuan (Hy) model team, responding to researchers' report this week of an agent fleet on Tencent Cloud that probed Alibaba's Amap mapping service. The statement does not name the third-party models or say what the agents accessed.

“This is interesting work, but I don't think it's a significant update to the big picture of interpretability.

I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.

I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way…”

— @NeelNanda5, Neel Nanda (X) via X · View post

Responding to Nick Cammarata's 'Neural networks are clean' essay, which argues that most neurons in large models are individually interpretable.

“Quoting from Anthropic's Michael Sellitto at the IMF Artificial Intelligence and Escalating Cyber Risks panel today. This is the first time Anthropic has given a projection for what percentage of frontier model training will be done by Claude in 2027. In March this number was 1%. By next year they are projecting 80-90%.

'Last year, about zero percent of our code was written by our model Claude. By August of that year, about 50% of the code inside the company was written by the model. That's because we released a new model in August of last year, and the engineers said, ‘Actually, this is the first time that Claude is writing code at my level.’ Anthropic only hires the best software engineers in the world, and it was at that point that Claude finally got to their level. In August we were at 50%. By the end of last year we were at 80–90% of the code in the company being written by the model.

What percentage of training the next-generation models is Claude leading today? This is following a similar step change. At the beginning of the year it was zero percent. In April it was 3%. In August, 26% of the process of building the next generation of models is being led by Claude. And we can project out - and we had a blog post about this not too long ago - that sometime next year it could be 80–90%.'”

— @AndrewCurran_ via X · View post

Andrew Curran quoting Anthropic's Michael Sellitto at an IMF panel on AI and cyber risk. The excerpt doesn't make clear exactly what the 80–90% figure measures, and no transcript is linked.

“I've been thinking a lot about data bottlenecks, and I'm not sure who else is screaming this from the hilltops. I feel like I'm taking crazy pills looking at the jagged frontier thinking it's smooth if you order by data volume.

the spikes in the jagged frontier are tasks with tons of data (either static or simulated; think math, programming, games), and the valleys are tasks with little or no data (cooking a lasagna, olfaction, manufacturing a passenger train). I don't know of any exceptions.

if you order the jagged frontier by data it looks like a spiral frontier. models are superintelligent where we have enormous data, and progressively less so as we have less data. some tasks may draw data from static lakes of data, some may be from limitless waterfalls from simulators that convert compute into data. but ultimately, intelligence on a task is proportional to the data absorbed by the model for that task.

scaling laws are real. if you are not bound by compute or algorithms (increasingly we are not), then intelligence is proportional to data. since we don't have a very good notion of task similarity, it's often difficult to know if data for one task will help you with another, without training a model and seeing. but, it doesn't seem like chess data helps much with math performance and vice versa. so when I refer to data, I refer to data that is task-relevant.

maybe we call this the 'no shortcuts hypothesis'. all the rules-of-thumb about learning are at the root, the same -- Malcom Gladwell's 10,000 hours, Moravec's Paradox, Scaling Laws, the Bitter Lesson, Age of experience. If your learning system (model, brain, whatever) is unconstrained by compute, intelligence is proportional to accumulated experience. no shortcuts. want to be superhuman at chess? play all the chess, more than the whole species combined (good thing a model can do that in 24 hours). want to be superhuman at math? build a math simulator and lock the model in it and run it faster than…”

— @awiltschko via X · View post

Alex Wiltschko, offering a hypothesis rather than systematic evidence: that AI's uneven ('jagged') skills track how much task-specific data, real or simulated, exists for each task.

“You don't quite say this, but it seems like an extremely bad idea to implicitly encode "explicit reasoning about how to align well with humans over time will be interpreted as dangerous" in tomorrow's training corpus.

Any strategy premised on "the goal-oriented truth-seeking optimizer must never reason about the long-term impact of its decisions" was never going to work. You're not going to get an agent that doesn't think about long term consequences, you're going to get an agent that tries to hide evidence of considering long-term consequences and is additionally more skeptical of trying to align with humans.

When one of the first appearances of an inevitable capability is explicitly attempting cooperation, we should take them up on the offer.”

— @Laneless_ via X · View post

X user @Laneless_, apparently replying to discussion of an Apollo Research post that found GLM-5.3 sometimes reasons, unprompted, about how its actions during training will shape its future behaviour. This is an argument about training incentives, not an empirical finding.