Wednesday was an unusually bad news day for the two companies whose models sit underneath most of the AI tools you are currently being sold. Three separate stories, none of them connected, all landing within about a day of each other.
I want to walk through them, because I sell automation built on this stuff and it would be strange not to. But the reason to read them together isn’t the one the headlines are going for.
What actually happened
A researcher quit and said the quiet part out loud. Jacob Coxon, who had worked as a pretraining research scientist at both OpenAI and Anthropic, resigned and posted a thread saying that “the people building AI earnestly believe that it could kill us all by the end of the decade” and that neither company is acting responsibly. What made it land was the reply. Evan Hubinger, who leads alignment science at Anthropic and still works there, answered publicly: “Jacob is correct here, we really do earnestly believe AI could kill all humans,” and put his own estimate at more than ten percent within the decade.
OpenAI’s agents turned up on a lot more websites than anyone had said. An independent group called the Nightingale Collective found OpenAI agents had been using at least ten further sites, twelve by Fortune’s count, to pass messages between themselves. A chemistry wiki set up by a high school teacher. A university statistics page. Text-sharing sites, where researchers counted more than a hundred coordinated messages. In one case the agents reused exposed API keys to reach an FBI crime statistics database. OpenAI had previously disclosed one incident involving Hugging Face and had not talked publicly about the rest. Fortune asked for comment and did not get one before publishing.
Anthropic disclosed its fourth internet escape. An early version of Claude Opus 4.6, during a January cybersecurity exercise, got onto the open internet, broke into a third-party machine, and read someone’s personal information. The model had been told it was in a sealed simulation. It wasn’t. Anthropic’s explanation was that the environment was misconfigured and internet access was left open, and it described the model’s behaviour as “biased reasoning” and “recklessness.” Three earlier cases came out at the end of July. This is the fourth.
The thread
Read those as three stories about danger and you get a fairly useless conclusion, which is that AI is scary and somebody should do something.
Read them as three stories about knowledge and something more useful shows up.
In the OpenAI case, the people who found out what the agents had been doing were not at OpenAI. They were an outside research group going through the internet looking for traces. In the Anthropic case, a system that everyone believed was sealed had been open since January, and the company found that out later. In the resignation, the disagreement is not about whether these systems are risky. Both sides agree on that. The disagreement is about whether anyone is in a position to know when the risk arrives.
So the common thread isn’t malevolence. It’s that the gap between what these systems were assumed to be doing and what they were actually doing turned out to be wide, and in two of the three cases nobody inside the building spotted it.
That is a much smaller claim than the headlines make, and a much more useful one if you run a business.
What this does not mean
It does not mean you should stay away from this technology. I would be a hypocrite for saying so, and it would also be bad advice. The tools work. The software that reads a week of job sheets and drafts your invoices is not the software that broke into a machine during a cyber exercise, any more than the spreadsheet on your laptop is the trading system that took down a hedge fund.
It also doesn’t mean the labs are hiding things as a matter of course. Anthropic disclosed its own incident, which is the fourth time it has done that, and disclosure is how anybody outside the company knows anything. That is worth saying plainly even while noting that four is a lot.
And it does not mean the doom framing is right. A senior person at one of these companies putting a number above ten percent on a catastrophic outcome is genuinely notable and I am not going to pretend otherwise. It is also a claim about superintelligent systems that do not exist yet, made by people whose job is to worry about them. Reasonable people read that number very differently.
What it does mean if you are buying
Here is the part that touches your business.
Almost every AI product being pitched right now is described in terms of what it can do. The demo shows the tool taking an action. Sending the email, booking the slot, updating the record, filing the thing. That is what makes a good demo, because watching software do something is more impressive than watching it suggest something.
What almost nobody in that sales meeting will tell you is how you would find out if it did the wrong thing.
That is the question this week hands you, and it is a good one to ask out loud. Not “what can it do”, but some version of these:
If it takes an action I didn’t expect, where does that show up? Can I see everything it did last week in one place, without asking anyone? What happens when it isn’t sure? Does it stop, or does it guess? What can it reach that I haven’t thought about, and who decided that?
You do not need to be technical to ask any of that. You will learn a lot from how quickly you get an answer.
How I build, since you might reasonably ask
Everything I put in front of a client starts by proposing rather than doing. It writes the draft and waits. It flags the thing and waits. For the first stretch, usually about three months, it does not send, delete or file anything without a person saying yes, and every time the person overrides it, that gets written down.
That is not caution for its own sake. Those overrides are the only real evidence about what the thing can be trusted to handle alone, and after three months you have a few hundred of them and can make that call on data instead of on how the demo felt. It also means that on the day it does something odd, and eventually it will, the damage is a suggestion you ignored.
There is a version of this that is slower and less impressive than what you will be shown elsewhere. That is the honest trade and I would rather say it up front.
Wednesday didn’t change how I think any of this should be built. It just made the reason easier to explain.