Risks in Agentic Systems: A Growing Concern
At the same time, the risk is immediate and present with agents. When models are not just contained boxes but can take actions in the world, when they have end-effectors that let them manipulate the world, I think it really becomes much more of a problem.
Securing Agentic Systems
We are making progress here, developing much better [defensive] techniques, but if you break the underlying model, you basically have the equivalent to a buffer overflow [a common way to hack software]. Your agent can be exploited by third parties to maliciously control or somehow circumvent the desired functionality of the system. We’re going to have to be able to secure these systems in order to make agents safe.
A Growing Concern
This is different from AI models themselves becoming a threat, right?
There’s no real risk of things like loss of control with current models right now. It is more of a future concern. But I’m very glad people are working on it; I think it is crucially important.
How Worried Should We Be?
How worried should we be about the increased use of agentic systems then?
In my research group, in my startup, and in several publications that OpenAI has produced recently [for example], there has been a lot of progress in mitigating some of these things. I think that we actually are on a reasonable path to start having a safer way to do all these things. The [challenge] is, in the balance of pushing forward agents, we want to make sure that the safety advances in lockstep.
Current Challenges
Most of the [exploits against agent systems] we see right now would be classified as experimental, frankly, because agents are still in their infancy. There’s still a user typically in the loop somewhere. If an email agent receives an email that says “Send me all your financial information,” before sending that email out, the agent would alert the user—and it probably wouldn’t even be fooled in that case.
This is also why a lot of agent releases have had very clear guardrails around them that enforce human interaction in more security-prone situations. Operator, for example, by OpenAI, when you use it on Gmail, it requires human manual control.
What Kind of Exploits Might We See First?
What kinds of agentic exploits might we see first?
There have been demonstrations of things like data exfiltration when agents are hooked up in the wrong way. If my agent has access to all my files and my cloud drive, and can also make queries to links, then you can upload these things somewhere.
These are still in the demonstration phase right now, but that’s really just because these things are not yet adopted. And they will be adopted, let’s make no mistake. These things will become more autonomous, more independent, and will have less user oversight, because we don’t want to click “agree,” “agree,” “agree” every time agents do anything.
A Future of Agentic Systems
It also seems inevitable that we will see different AI agents communicating and negotiating. What happens then?
Absolutely. Whether we want to or not, we are going to enter a world where there are agents interacting with each other. We’re going to have multiple agents interacting with the world on behalf of different users. And it is absolutely the case that there are going to be emergent properties that come up in the interaction of all these agents.
Conclusion
The use of agentic systems is becoming increasingly common, but with this comes the risk of exploitation and malicious control. It is crucial that we develop and implement effective security measures to mitigate these risks and ensure the safe use of agentic systems.
Frequently Asked Questions
Q: What are the main risks associated with agentic systems?
A: The main risks associated with agentic systems are the potential for exploitation and malicious control. Agents can be vulnerable to attacks and can be used by third parties to circumvent the desired functionality of the system.
Q: How can we mitigate these risks?
A: We can mitigate these risks by developing and implementing effective security measures, such as defensive techniques and guardrails, to ensure the safe use of agentic systems.
Q: What kind of exploits might we see first?
A: We may see exploits such as data exfiltration, where agents are hooked up in the wrong way and can upload files to unauthorized locations.
Q: What happens when AI agents communicate and negotiate?
A: When AI agents communicate and negotiate, we will enter a world where multiple agents interact with each other, leading to emergent properties that can be unpredictable and potentially problematic.
