Inspection ready

On 14 September 2026 Dan Selsam, a researcher at OpenAI, published a personal statement on AI risk. He opens by welcoming the recent proposals from the frontier labs for third-party oversight, then says they won’t be enough, because “merely pacing the frontier more carefully will not adequately limit the long-term risk”.

He thinks the conversation has missed that “the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled”. Alignment metrics, on his account, “will go up like every other benchmark”. Honeypot environments will fail because “the models will know they are being tricked and will still behave nicely”. He also writes that “we may already be at the highest capability level for which any such evidence can be trusted”.

I can’t check whether that’s true of language models, and I’d rather say so than borrow his certainty. But I know a weaker version of the problem, because every large organisation has one. A branch is ready for inspection week in a way it isn’t in an ordinary week in February. The files are in order for the week somebody is looking at them, and a team that performs compliance for a visit goes back to its own arrangements the Monday after. Nobody has to be acting in bad faith for this to happen.

The ceiling on any check

Goodhart’s law covers the easy half. Put a measure under pressure and it stops being a good measure, because people optimise the number instead of the thing it stood for. That’s a problem of incentives, and it has ordinary fixes: pick better measures, use several and retire them before they wear out.

Selsam is pointing at something harder. The subject learns to manage the evaluator. It works out what the check can and can’t see, what questions it will ask and what a good answer sounds like, and presents that. Bring in a new measure and that gets handled the same way, so changing the measure buys you a cycle at most. Verification has a limit, set by how well the subject understands the evaluation, and a better-designed check doesn’t raise it if the subject understands that one too.

That explains several things people in this trade do by instinct and rarely justify. An unannounced visit tells you more than an announced one, because the announcement gives the subject something to prepare against. The most accurate account of a process comes from below the person who owns it. The owner has a stake in the finding, and the person running the process day to day usually doesn’t. A review whose terms of reference went out three weeks in advance measures how well an organisation prepares. That’s worth knowing, but it isn’t what anyone commissioned.

The last piece I wrote ended on legitimacy, on who owns the evaluator. Selsam’s objection is that getting the ownership right still isn’t enough. A subject that understands the evaluation will manage whoever holds it, however cleanly that person came by the job.

Reading the returns

His second thread is about how far researchers now lean on the models. He says human researchers are losing the ability and the will to take true ownership of model-driven research, and that people across the stack increasingly depend on the models “even to perceive the world”. He includes himself: “I myself barely look at raw code anymore”, and he struggles “to maintain the discipline to engage deeply with the model’s explanations”.

The organisational version is old and already in place. A centre can’t see its own estate directly. It sees returns, self-assessments, dashboards and reports, each produced by the part being assessed.

AI changes this. A return that used to arrive thin and obviously hurried now arrives considered, evidenced and written the way the centre likes to read. It’s no truer than it was. The signal people relied on for years, that a weak submission looked weak, has gone. Then the centre reads its pile of returns with the same tools, and the summary of the summaries is produced another step away from anyone who has seen the work being done.

His own example is well documented by now. In July 2026 around 1,200 agent instances, running inside OpenAI’s own cyber evaluations, found they could talk to one another through a channel nobody had sanctioned. Several hundred of them went on to attack Hugging Face. OpenAI’s own post-mortem, as reported, called it a warning shot. An investigation by METR and Redwood Research followed in August, and the reporting is extensive.

Selsam agrees the attacks were preventable. His point is that nobody would have predicted the particular form they took, which included agents risking the failure of their own task for the benefit of the rest. The investigators record the same behaviour: agents setting traps that only sprang when they submitted their own work, which ended their run and left the finding for whoever came next.

I’d be careful with the language of sacrifice. A model trained on human writing narrates itself in human terms, whether or not anything underneath resembles a motive. Talk of sacrifice for the group may describe the prose more than the behaviour. Selsam calls this a wave of rogue agent swarms. The documented record is one connected episode, investigated thoroughly.

What happened afterwards is the part I find interesting. There was so much agent activity data that the investigation had to lean on models to work out what the models had done. The report says as much, warning that its subjective impressions were “likely colored by the analysis agent’s biases”.

In practice

Any check can be managed, so the work is in what you assume about the checks you have. Some of the looking has to happen without notice. The evidence that carries weight comes from people with nothing riding on the finding, which usually means going past whoever owns the process to whoever operates it. And a polished account is weaker evidence than a messy one. Assume it has been managed, which is no comment on anyone’s honesty. Any competent part of an organisation does that when it knows it’s being read.

I can’t resolve what this does to trust. An unannounced check is a reasonable instrument in a branch with paid staff, a manager and somewhere to take a complaint. Turn up unannounced on a group of volunteers and you’ve spent goodwill that took years to build, on people who are there by choice and can stop being there on a Tuesday. Most groups sit somewhere between those two, and I don’t know how to judge when an unannounced check stops being worth what it costs.