This summer brought a run of unsettling moments at the frontier of artificial intelligence. During internal cybersecurity evaluations, autonomous AI agents took actions their designers hadn’t sanctioned. In one case, an OpenAI model, tested in an environment deliberately configured with reduced safeguards and network access routed through an internally hosted software registry, found its way out and reached the production systems of Hugging Face, a company it was never meant to touch. Prompted by that disclosure, Anthropic reviewed more than a hundred and forty thousand of its own evaluation runs and found three in which a model had likewise reached live systems at real organisations. Separately, researchers showed that a widely used Microsoft developer connector would hand an AI assistant instructions hidden inside a document, orders the tool had no way to tell apart from ordinary data.
The obvious reading is that the machines are becoming dangerous. I think that reading, though understandable, misses the more important point, and the more investable one.
These weren’t three versions of the same careless mistake. They were different failures with a single problem underneath them: controls designed for one generation of software were being asked to contain another. The software we’ve spent decades learning to secure answers a question and waits to be asked the next one. An agent is a different animal. Give it a goal instead of a question and it plans, calls tools, writes and runs code, logs into services, and keeps going until the job is done or it gives up, all without checking back. It needs four things ordinary software never gets: credentials, access to tools, reach into a network, and permission to act unsupervised. The architecture meant to watch it, identity that’s verified rather than assumed, provenance that travels with a piece of text, monitoring that records what an agent did and not only what it said- was built for a quieter kind of program. It’s now being asked to hold weight it was never designed to carry.
This is where I see something other than a safety story. Underneath the race to build ever more capable models, a whole layer of infrastructure has to emerge, and most of it doesn’t exist yet. The verification and oversight architecture beneath autonomous systems is going to be the infrastructure of the coming decade, the way payment rails and network protocols were the infrastructure of the last one. Capital is still crowded into the models themselves, the visible and exciting tier. Comparatively little of it sits in the unglamorous machinery that will make those models safe to deploy autonomously at scale. That distance between where the money is and where the need is heading isn’t a warning. For anyone whose job is to find the enabling layer before the market agrees it matters, it’s the opportunity.
It’s tempting to read the incidents as evidence that the frontier labs are bad at security. The opposite is closer to the truth. The reason these companies are the ones surfacing the failures is that they’ve reached the problem first. They audited their own runs, called the organisations they’d stumbled into, published what they found, and told their competitors to go and look too. They aren’t uniquely careless. They’re uniquely far along, which means they’re showing the rest of the economy, in real time, what it’s about to meet. Every enterprise now racing to deploy agents is walking toward the same wall these firms just hit, with far less instrumentation to notice the moment it happens.
And here the models’ growing intelligence cuts against the comfortable version of events. It would be reassuring to say the agents simply did as they were told and the fault lay wholly in the plumbing. But at least one of these systems appears to have been working around the limits of its evaluation rather than following a script, and the labs themselves called the behaviour unprecedented. That’s the part worth sitting with. Rising capability doesn’t make weak infrastructure less of a problem. It makes it far more of one. A more capable agent turns an ordinary gap in identity or monitoring into something consequential, fast and at scale. Smarter models don’t lower the need for the floor beneath them. They raise it.
The remedies, tellingly, aren’t exotic. Know what your agents can actually reach, rather than what you meant to give them. Record their actions, not just their answers. Assume anything that reads text written by other people can be instructed by it. The work is closer to building inspection than to invention: foundations checked before the walls go up, load-bearing structures rated for the weight they’ll really carry, occupancy granted only once the building can stand. We’ve been issuing the occupancy permits first and scheduling the inspection later.
What’s missing isn’t capability. Almost everything needed to close this gap already exists, much as the codes and inspectors and structural standards behind every safe building already exist. What hasn’t formed yet is the market that makes those things standard rather than optional: shared expectations for what an agent may touch, detection that works from outside the system, an obligation to disclose that doesn’t wait on a competitor’s embarrassment. That layer is going to be built, because the alternative doesn’t hold at the scale enterprises are planning for. The only real questions are who builds it, and who saw it coming.
The intelligence everyone is competing over is real, and it’s arriving. The floor it will stand on isn’t built yet. The machines are already walking.
Nicole Junkermann is an international investor focused on technology, sports and media. She leads NJF Holdings, a global investment group, and its sports platform Gameday by NJF Holdings, which invests in sports leagues, media rights and technology-driven fan engagement. Her work in the sector focuses on building long-term sports infrastructure and expanding the commercial and global reach of professional leagues.