4 Jul 2026

Digest: scheming evals mislead both ways, Fable 5 writes first genuine megakernel

Measurement is today's through-line: fresh doubt about the instruments used to catch scheming models, two new benchmarks stretching what agents are tested on — from single-shot GPU megakernels to day-long real-world environments — and an essay arguing that AI forecasters have reached expert level.

1. Scheming Evals Mislead in Both Directions Recommended

   Researchers who spent weeks measuring in-context scheming — a model covertly pursuing a misaligned goal while outwardly complying — report that two behavioural detectors the field routinely relies on gave confidently wrong answers within the same project, one of them by manufacturing a dramatic signal that was not real. Evidence that scheming evals can mislead in both directions (false alarms and missed behaviour) matters because such results increasingly feed system cards and predeployment decisions, and it adds to a recent run of work questioning how much weight deception evaluations can bear.

   Chijioke Ugwuanyi via LessWrong
2. Claude Fable 5 posts the first genuine 'megakernel' on KernelBench-Mega

   On KernelBench-Mega — a community-run benchmark that asks coding agents to fuse an entire model block into a single GPU 'megakernel' — Claude Fable 5 produced what the maintainer calls the first genuine single-fused kernel ever submitted, reaching roughly a 19x decode speedup over an optimised PyTorch baseline where Opus 4.8 managed 14.4x and GPT-5.5 4.3x via multi-kernel pipelines that fail the benchmark's authenticity gate. It extends the steady run of agentic-coding capability gains, though the caveats matter: the results come from a single independent maintainer on a constrained compute budget, not an audited third-party evaluation.

   kernelbench.com
3. EdgeBench measures how agents learn from real-world environments across 134 day-long tasks

   EdgeBench, a new evaluation suite framed around 'scaling laws of environment learning', studies how agents learn from real-world environments across 134 day-long executable tasks. Pushing agent evaluation to day-long horizons — and measuring how much agents improve from environmental experience rather than just whether they finish — targets the axis along which autonomy and capability-forecasting questions increasingly turn.

   edge-bench.org
4. Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment

   A new paper argues that character-level perturbations bypass safety training because BPE tokenisation — the standard way models split text into sub-word chunks — fragments safety-critical words into pieces that alignment datasets never contain, and tests the mechanism end-to-end across five model families. It points to a structural gap in how refusal training generalises, though the systems tested are small open-weight models (roughly 4B–8B parameters), so whether frontier models share the vulnerability remains unverified.

   Tung-Ling Li et al. via arXiv
5. Model access for third-parties — it's a big deal!

   A LessWrong post argues that the gap between the model access frontier-lab insiders enjoy and what external safety researchers and auditors can get — a 'model access gap' — risks widening sharply, and makes achieving rough access parity a top priority for the external safety community over the next 6–12 months. As predeployment testing increasingly depends on third parties, the terms on which outsiders can probe frontier models is becoming a governance question in its own right.

   Cleo Nardo via LessWrong
6. Anthropic engineer publishes a field guide to working with Claude Fable 5

   Thariq Shihipar of Anthropic's Claude Code team published 'A Field Guide to Fable: Finding Your Unknowns', arguing that the binding constraint when working with Claude Fable 5 is no longer model capability but the user's ability to surface what they haven't told it — 'the map is not the territory'. First-hand accounts from inside a frontier lab of how its own builders work with the newest models are a useful early read on where human input still limits agentic systems.

   X
7. The AI Superforecasters Are Here

   In 'The AI Superforecasters Are Here', Scott Alexander writes that AI forecasting systems have arrived at the accuracy tier long associated with elite human 'superforecasters', and considers what follows. Machine-level judgemental forecasting is both a capability marker in its own right and consequential for how probabilistic assessments — including of AI progress itself — get made.

   Scott Alexander via Astral Codex Ten