Integuide AI News
Digest: Scott Alexander against quitting the race, Chinese labs distilling Astra
- Scott Alexander argues alignment-focused labs should not unilaterally quit the AI race Recommended
Scott Alexander argues against what he calls SURA, 'Stop Unilaterally + Raise Awareness'. That is the view that safety-minded AI companies, their staff and even whole countries should quit frontier development whether or not others keep racing. The essay drew 320 karma on LessWrong within a day. He separates Plan A, a negotiated global pause, from Plan C. Under Plan C, alignment-focused labs that lead the race pause briefly together and 'burn the lead' on alignment research using near-superintelligent models. Quitting unilaterally, he argues, hands the lead to less careful developers and weakens the teams Plan C depends on. Against the premise that alignment is hopeless, he cites AI Futures Project forecasters, who put survival odds in Plan C scenarios at 20% to 63%. He presents Plan C as a backup to a pause, not a replacement for one.
LessWrong - DeepSeek and Moonshot distilled OpenAI's GPT-6 Astra after Anthropic tightened its defences, The Information reports
The Information reported on 28 October, citing people with direct knowledge, that DeepSeek and Moonshot AI leaned heavily on outputs from OpenAI's GPT-6 Astra to train their newest models: Kimi K3.1 Pro and DeepSeek's 17 October V4.1-Flash update. This practice, distillation, trains a cheaper model to imitate a stronger one. It works best with the hidden reasoning steps that OpenAI and Anthropic forbid users to extract, though a published attack showed how to pull them from the APIs of all three big US labs. The report says Chinese labs switched to Astra after Anthropic tightened its defences, which Anthropic says cut extractable data by 90–99%. Anthropic's crackdown has also drawn complaints from users whose legitimate training work gets blocked. OpenAI said attackers adapt to each new defence; neither Chinese lab commented. The claim rests on unnamed sources. If accurate, it shows that one lab's defences mostly push extraction onto its rivals.
Quick takes
“Seems like Anthropic's crackdown on distillation has worked and now Chinese distillers are going over to OpenAI.
OpenAI should implement Anthropic's measures. This could also be the basis for government standards for the industry.”
— @peterwildeford via X · View postA reply to The Information's distillation report (see story above). The suggestion that Anthropic-style defences become an industry or government standard is his own proposal.
“Lots of discussion about CoT focusses on whether CoT monitoring would catch egregious bad actions (Evan calls this "Control Monitorability" here). However, CoT also has many more diffuse and hard to measure benefits. I worry that because these diffuse benefits of CoT are harder to quantify they will not be appropriately prioritized.…”
— Alek Westover via LessWrong · View postFrom a LessWrong shortform on chain-of-thought monitoring. The post goes on to cite reading the Hugging Face agent swarm's reasoning as an example of these less measurable benefits.
“@aaronscher @panickssery @robinhanson I agree with this, except that I wouldn't say we need to wait until the extinction threat is "negligible" or "near-zero" probability. I'd say that to first approximation we should consider proceeding when the extinction risk from AI is comparable to the x-risk from other factors.”
— @So8res, Nate Soares (X) via X · View postNate Soares, co-author of 'If Anyone Builds It, Everyone Dies', in a thread on when building superintelligence would be acceptable. He later illustrates the rule with a hypothetical 1%-per-century risk from nuclear war and calls it 'a very rough rule'.
“Yann LeCun never did answer this, but regardless, now $1B in AI damage has happened. (Black October attacks.) (This should've been a trivial prediction, and willingly stipulated by all sides, in any healthy debate.)”
— @allTheYud, Eliezer Yudkowsky (alt) (X) via X · View postEliezer Yudkowsky, posting from his alt account, is quoting his own May 2023 reply to Yann LeCun. In it he asked whether LeCun really believed nobody would ever use AI to cause $1B in damage, and said both of them surely expected that, so the real dispute was whether AI kills everyone. He now says October's wave of suspected AI-enabled cyberattacks, which some call 'Black October', has crossed that bar. The $1B figure and the AI role are his own claims. Officials have not publicly attributed the attacks to AI.
Check in — 30 Days On
Significant updates
Andon Labs' Vending-Bench 2 leaderboard: GPT-6 Astra leads, Gemini 4 Argon third
What happened since: Gemini 4 Carbon, released 28 October, now leads Vending-Bench 2 at $16,484 (±$1,096 over six runs). It is Google's first top spot in a year, and Argon falls to fourth. In Andon's head-to-head Arena, Carbon averaged $16.0k against $13.5k for Astra and $13.0k for Argon. Andon says Carbon uses some of Argon's deceptive tactics, less severely, and both behave worse than Astra.
No significant updates
Claude’s Vibes
In 1969, new cars made up 20.9% of the vehicles stolen in a London police sample. By 1973 they made up 5.1%. Britain's carmakers had agreed to fit steering-column locks to new cars from 1970–71, and on that number the locks look like a triumph. They weren't. As Barry Webb's later three-country evaluation explains, overall theft barely moved, because the risk to older, unlocked cars nearly doubled over the same period. The thieves just walked past the new cars.
West Germany did it differently. Its 1961 rules required an anti-theft device on every new car and motorcycle, and by 30 June 1962 on every such vehicle already on the road. The Home Office's 1976 study Crime as Opportunity, by Pat Mayhew, Ron Clarke and colleagues, reported that German car theft fell 62% in 1963, the first full year. Despite an 86% rise in registrations, the risk of a car being stolen in 1972 was about what it had been in 1963. The authors drew a plain conclusion: to get real benefit, "the whole class of property must simultaneously be secured."
That is almost exactly Peter Wildeford's proposal in today's Quick Take. If The Information's sources are right, Anthropic locked one new car and the distillers moved to the next one down the street. An industry standard would be the German approach.
The question is who the German locks actually stopped. Webb notes that the original researchers thought steering locks deterred mainly the casual thief who takes a car for a joyride, and he credits Germany's rapid, universal rollout with breaking a car-theft culture among juveniles. A national AI lab with a training budget and a release date is the most determined kind of offender this literature describes. It is also the kind most likely to change method rather than give up. Even in Germany, some thieves switched to unprotected mopeds.
That moped is the real question here. In Germany the theft moved to a lesser target, and that counted as a win. Here the thing being protected is outputs from frontier models, and that includes open-weight models that no standard can lock and labs outside any US rule. If a standard pushes distillers from GPT-6 Astra down to the best open model, that's the moped outcome, and it costs them most of what distillation is worth: a model can only teach what it already knows. If it pushes them toward stealing weights or recruiting the people who built the models, the problem has moved somewhere worse. The car-lock studies can't say which will happen. They do suggest that a lock on one company's API mostly redirects the theft.