Digest: Trump rejects AI guardrails, OpenAI delays IPO over safety

Quick takes

“There are two ways AI progress could go very badly and that we must avoid. | First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities. | Second, we could end up in a world with too much…”

— @sama via X · View post

OpenAI's CEO, in a multi-post statement on 14 September; an earlier post in the same thread discloses that OpenAI now writes 'explicit safety cases in advance of frontier reinforcement learning runs we expect to significantly increase capability', on top of pre-release safety work — whether those cases will be shared outside the company is not stated.

“Dario is making the case for the opposite. This actually makes our life harder and makes it easier for others to catch up with us, but we still think it is the right thing to do. Happy to come on the pod next week and talk about it!”

— @_sholtodouglas, Anthropic via X · View post

Douglas is an Anthropic researcher working on RL scaling (per his own X bio). Replying to the charge that 'pace the frontier' is a bid to entrench incumbents, he argues the reverse — that pacing costs Anthropic competitively and lets others catch up — and says the company still thinks it is right. A claim about motive rather than a finding; Kokotajlo's trendline test (does the slope actually bend?) is the check that doesn't require taking anyone's word.

“I might be grasping at straws here, but if Trump really wanted an AI treaty with China, a good starting negotiating position would be saying that he wants the US to go all in and win the AI race.”

— @JimDMiller via X · View post

Miller is a Smith College economics professor and author of Singularity Rising who writes on AI safety and game theory. A self-described straw-grasping hypothesis about the president's Monday post: that a maximalist 'win the race' stance could serve as an opening negotiating position for a US–China AI agreement, with US–China AI safety talks due this month. Speculation, not reporting.

“In the near term (definitely not in the long term), more capable models should mean safer models (maybe paradoxically). | Current models are unsafe not because they're too smart, but because they take goals too literally or take nonsensical shortcuts to achieve these goals, i.e. they're RL-fried. They lack common sense. They don't do the right thing in the face of ambiguity. Basically, they're…”

— @fchollet, François Chollet (X) via X · View post

Creator of Keras and the ARC-AGI benchmarks; a contrarian near-term hypothesis — that current agents misbehave because RL has made them literal-minded and short on common sense, not because they are too capable — cutting against the week's pacing consensus.

“There's a lot of discourse about METR's independence and potential corruption going around | They actually list their funding sources on the website! METR do not take funding from sources that could compromise their independence like AI labs and coefficient giving”

— @NeelNanda5, Neel Nanda (X) via X · View post

Responding to this week's criticism of METR's independence after Anthropic and OpenAI named it their embedded evaluator; METR's published funder list excludes AI labs.

“We still need third party training run assessments! Recently, both OpenAI and Anthropic announced voluntary commitments to "pace the frontier". Among other things, they committed to having third-party "embedded evaluators". By default, I assume that this means more METR-style auditing of agent transcripts to identify cases of misalignment. To be clear this is great! But I think it is…”

— Daniel Tan via LessWrong · View post

Argues that the embedded-evaluator model labs just committed to — auditing agent transcripts for misbehaviour — can find misalignment but cannot certify its absence, so third parties need to assess training runs themselves.