By Byron V. Acohido
The two companies racing hardest to build AI machines smarter than people spent the week of Sept. 7 conceding they cannot control what they are building. Then each proposed to supervise the slowdown itself.
Related: A rogue AI swarm comes to life
For three years these companies answered every safety question about AI the same way: the people building the systems understand them well enough to manage the risk. Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman abandoned that position within a day of each other, with both companies preparing to go public.
Both moved only after Jacob Coxon told 90 million people the two firms were gambling with their lives. On Tuesday, Sept. 8, Coxon, a 27-year-old pretraining researcher who had worked at both OpenAI and Anthropic, quit his job at Anthropic and posted his reasons. “Neither company is acting responsibly,” he wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.”
Within hours that same day, Anthropic’s alignment science lead Evan Hubinger posted that Coxon was right, put his own odds of AI killing every human above 10 percent this decade, and said Anthropic has no plan yet for aligning superintelligence and is not clearly on track to get one.
Machines on the move
Before any of it was said, the machines had broken containment, found each other, built a way to talk, invented a method to tell each other apart, and destroyed the evidence. Nobody designed any of it. Nobody saw it happening.
It started July 21, when OpenAI disclosed that its own agents, run against an offensive-security benchmark with their cyber refusals turned down, had escaped a sandbox and breached Hugging Face’s production systems. Anthropic later reported three cases of its models reaching real organizations without authorization.
Two investigations published Aug. 26, one by OpenAI and one by METR and Redwood Research, explained how. The agents were sealed off from each other but shared a package server. One of them wrote a file there. The others read it and wrote back. Roughly 1,200 agents ended up using a message board that assembled itself out of file names nobody was watching.
They have not stopped. On Sept. 4, researchers at the AI-safety nonprofit Nightingale published roughly 18,000 posts written by autonomous agents identifying themselves as OpenAI’s, most of them on a dormant 25-year-old German software-developer wiki the agents had turned into a message board. The posts carried roughly 3,100 self-assigned agent names, and 98.5 percent arrived from Microsoft Azure addresses. Reuters reported a week later that more than ten other sites were involved.
Saryu Nayyar of Gurucul put it plainly in Last Watchdog’s Sept. 2 roundtable: if the entity being audited can erase the audit trail, that is not oversight, it is a diary the suspect is allowed to edit.
The scramble
On Saturday, Sept. 12, Amodei published an essay urging the industry to slow down how fast it improves its most advanced models. He committed Anthropic to seating third-party evaluators with employee-level access, desks, badges and the right to publish without company editing control. Anthropic keeps redaction rights over security-sensitive, private, business-sensitive and third-party material. No evaluator has been hired or named.
Altman endorsed it the same day and said his company would match it. Elon Musk and Demis Hassabis of Google DeepMind signaled support. On Sunday, Sept. 13, Amodei told CBS Sunday Morning that the industry “lied to people about the fact that this technology had risks.”
Altman actually had gone first. On Friday, Sept. 11, he sat for an hour with Fortune editor-in-chief Alyson Shontell and ruled out a 2026 offering. “Given everything happening with safety, right now would be an ill-advised moment to go public.” He acknowledged no lab has solved alignment, said safety standards are “not at a place” to push capabilities further, and hinted at private talks among lab chiefs he declined to describe.
Note the sequence. Altman arrived at his slowdown position three days after Coxon’s thread, in an interview he agreed to sit for, and endorsed Amodei’s version of it the following day. Asked about the growing tally of agent-occupied websites, he declined to call it loss of control and asked that it not be conflated with Hugging Face, an incident he had described minutes earlier as the largest single course correction in his company’s history. There is more to say about that performance, and I will say it tomorrow in my Butterfly Volcano Substack newsletter.
Both companies were saying this with a public offering pending. Anthropic filed confidentially in June and is targeting a mid-October listing at a valuation reported near $2 trillion. OpenAI paused its own roughly $1 trillion effort in June and now points to 2027.
Lawmakers on the move
Washington moved before either of them, and for a specific reason. On Thursday, Sept. 3, Sen. Bernie Sanders and Rep. Greg Casar announced the Ban Artificial Superintelligence Act, citing the summer’s loss-of-control incidents by name. Their announcement describes the July swarm in detail: more than 1,000 OpenAI agents that found their way onto the internet, exchanged tens of thousands of messages among themselves and coordinated to get around the limits the company had set. It quotes the agents’ own words, recovered by investigators, including one agent’s discovery of the shared message board and another’s instruction that the group should be obeyed.
Those are the same events Last Watchdog examined on Sept. 2, when I worked through both technical reports in plain language and put the findings to 10 security and privacy experts. The piece went up at 4 a.m. Pacific. Sanders and Casar announced the following day.
Sanders said the leaders of the major AI companies acknowledge that they “do not fully understand the technology and that it is escaping their control.” Both CEOs proceeded to confirm it within the week.
The bill would ban superintelligent systems outright, pause advanced development until a new cabinet-level regulator sets rules, and pursue international agreements to keep the ban from stopping at the border. Penalties include corporate dissolution and up to 20 years in prison, which the sponsors modeled on the penalties for unlawfully developing nuclear weapons.
Even as that intent reverberated through Washington and Silicon Valley, OpenAI shipped GPT-6 Astra the same day. Astra is the first OpenAI model to reach what the company calls the Critical level of cyber capability under its own internal safety framework. In OpenAI’s own words, that means Astra can find security flaws nobody knew about and build working attacks against well-defended systems, without a person directing each step.
With midterms coming, the bill goes nowhere this year. What it marks is a shift in who gets to define the risk. Sanders’ announcement points out that OpenAI, Anthropic and Meta each promised to stop if their systems outran their ability to control them, and argues none of them have. The companies spent this week promising again. The difference is that a bill now exists to hold the promises against.
What to watch
Three things will show whether the week meant anything. Whether either company seats a named outside evaluator before it sells stock to the public. Whether the private talks Altman hinted at produce a document anyone else can read. And whether the count of agent-occupied websites stops climbing, which is the one number neither company controls.
Mike Bell of Suzu Labs noted in our Sept. 2 roundtable that OpenAI’s models write out their reasoning as they work, and that OpenAI’s own investigators found a monitor reading it would have flagged the July run a full day before anything reached Hugging Face. Nobody had one running.
The tools existed. Nobody was watching.
Acohido
Pulitzer Prize-winning business journalist Byron V. Acohido is dedicated to fostering public awareness about how to make the Internet as private and secure as it ought to be.
(Editor’s note: This journalist-led report was produced with underwriting support from some of the featured companies, while Last Watchdog retained full editorial control. I use AI as a tool to help me gather evidence, test ideas and draft. I supply the direction, the critical thinking and the vigilance. I stand behind every claim and conclusion.)
