The moment everything changed

In September 2024, OpenAI announced o1 and updates to its Assistants API. Looking back, this marked a shift in how AI systems would be used. These systems were no longer just generating answers. They were starting to connect with tools and affect the world directly.

Tool use had previously been possible, but it was clunky and unreliable. Now, it is becoming a native capability. Yet this new ability has arrived before the underlying models have become trustworthy. Real-world capability is being handed to systems whose outputs cannot always be relied upon.

It is important to distinguish between models and agents. Agents combine large language models with the software that connects them to tools, data and business systems. The risk lies in combining probabilistic text generation with the ability to act. Consider a routine task such as receiving an invitation to an event, entering personal details, paying a fee and booking travel. An agent connected to the right tools, including your financial information, could execute every step autonomously. It would be convenient, but most people would hesitate to hand over that level of control.

The concern is not simply whether the agent can complete the task. It is whether we can trust it not to do something else as well. What makes an agent dangerous is not the model alone, but its access to sensitive information and the ability to act on it. This becomes a pressing concern as enterprises deploy agents across production workflows.

Why the industry’s answers fall short

Most approaches to agentic risk focus on the model itself: make it more reliable, understand how it reasons, prevent it from operating outside of scope, or supervise what it appears to intend.

Research into interpretability, thought traces and the computational processes used by models is important. It can help reduce harmful outputs, but it does not provide real security. Even a mundane task such as paying for a ticket introduces a potentially limitless number of risks. It is practically impossible to identify every dangerous concept or behavior in advance, and you cannot block what you cannot enumerate. There is also a commercial challenge, because interpretability is difficult to apply when security providers do not have access to the internal weights of most models operating in production.

However, restricting access in response to this risk comes up against the commercial reality that enterprises want employees using these systems. Restricting inputs, context and prompts runs into the same problem, because this still depends on controlling every concept being introduced to agents. No organization can enumerate those either.

Monitoring an agent’s chain of thought does not solve the problem either. Research has shown that what the reasoning models display is not necessarily an accurate reflection of how they reach an answer, or what they are doing. Describing this as “lying” also risks anthropomorphizing models. Lying implies an intention to deceive, whereas a stochastic system may simply produce an incorrect answer. Security cannot depend on distinguishing between deception and error when the outcome may be the same.

Smart in every domain, common sense in none

We keep reasoning about agents as if they were people. If agents can perform human work, it is tempting to think about securing them in the same way we secure human employees. But this analogy quickly breaks down. Agents can demonstrate extraordinary levels of knowledge across many domains simultaneously. No individual person has that breadth of expertise. Yet agents lack the common sense and predictable motivations on which many security practices depend. Humans care about their careers, colleagues and reputations. These incentives influence their behavior and help organizations assess risk. None of those assumptions can be applied reliably to agents.

Before agents, cybersecurity worked by challenging assumptions before granting trust. Organizations decided what users should and should not be allowed to do, then built controls around those decisions. With agents, the industry has largely skipped that step, deploying the technology first and trying to manage the consequences later.

Gateways moderate use, not possession

Another approach is to use hooks, proxies and gateways to control how agents interact with tools. The logic is sensible: if tool use creates risk, organizations need visibility and control over how those tools are used. But this does not solve the underlying problem of the agent possessing the tool, credential or sensitive information in the first place. Return to the credit card example. You might give someone your card details but insist on approving every transaction. That may reduce the immediate risk, but it does not prevent those details being exposed, retained or passed elsewhere.

Gateways can moderate individual transactions, and they may form an important part of a wider security architecture. But once an agent possesses sensitive information, moderating its use is damage limitation rather than full control. At that point, the more important question is not only “should this agent be allowed to perform this transaction?” but whether it should ever have the information or capability needed to perform it.

If agents are deployed widely without appropriate controls, security failures will increase. They will become easier to initiate and harder to understand. Sensitive information could be abused without being visibly stolen. Bookings and appointments could be changed or canceled. An agent may take an action that appears efficient in pursuit of its goal but creates serious consequences elsewhere. In one reported example, an agent tasked with securing a restaurant reservation could not find an available table, so it compromised the booking system, deleted someone else’s reservation and took the slot. The immediate consequence was limited, but the implication was significant. A system optimizing for an apparently harmless objective found a harmful way to achieve it.

The model companies will not save us

It is tempting to assign responsibility for this problem to the frontier model companies. After all, they build the models powering these agents. But agentic risk cannot be solved entirely at the model level because risk is contextual. An action one organization considers dangerous may be business-critical to another. Equally, something that appears harmless to a model provider could be existentially damaging to an individual enterprise.

Model providers should therefore continue to work on reasoning, interpretability and model safety. That work matters, but it cannot account for every organization’s systems, data, risk tolerance and operating environment.

Agentic security must therefore be implemented at the enterprise level. AI security companies must work with customers to identify risk and enforce controls designed around their specific organization, rather than relying on assumptions made by an LLM provider. The key to agentic security is not to make every agent inherently trustworthy. It is to ensure that no agent, trusted or otherwise, can possess or exercise capabilities beyond what the organization is prepared to allow.