Manipulating Autonomous Systems
In practice, exploitation of AI-powered applications requires two key components:
- An adversary must be able to get their data (read: attack) into the system through some mechanism.
- There must be a downstream effect that their malicious data can trigger.
When the AI component of the system is an LLM, this is commonly referred to as either direct prompt injection (the adversary and user are the same person) or indirect prompt injection (the adversary and the user could be different people).
Security and Complexity in AI Autonomy
Even before “agentic” AI became a distinct class of product offerings, the orchestration of AI workloads in sequences was commonplace. Even simple flows, such as an endpoint security product routing a sample to the correct AI powered analysis engine depending on file format, is arguably an example of such a workflow.
| Autonomy level | Description | Example |
| 0 – Inference API | A single user request results in a single inference call to a single model. | An NVIDIA NIM microservice serving a single model |
| 1 – Deterministic system | A single user request triggers more than one inference request, optionally to more than one model, in a predetermined order that does not depend on either user input or inference results. | NVIDIA Generative Virtual Screening for Drug Discovery Blueprint |
| 2 – Weakly autonomous system | A single user request triggers more than one inference request. An AI model can determine if or how to call plugins or perform additional inference at fixed predetermined decision points. | Build an Enterprise RAG Pipeline Blueprint |
| 3 – Fully autonomous system | In response to a user request, the AI model can freely decide if, when, or how to call plugins or other AI models, or to revise its own plan freely, including deciding when to return control to the user. | NVIDIA Vulnerability Analysis for Container Security Blueprint, “BabyAGI”, computer use agents |
Level 1
Level 1 is a linear chain of calls, where the output of one AI call or tool response is conveyed to the next step in an entirely deterministic manner. The complete flow of data through the system is known in advance.
Figure 5. Taints from untrusted sources are difficult to bound and enumerate in Level 3 systems
By classifying an agentic application into the correct level, it becomes simpler to identify the overall level of risk posed by the application and corresponding security requirements.
Recommended Security Controls per Autonomy Level
| Autonomy level | Recommended security controls |
| 0 – Inference API | Use standard API security. |
| 1 – Deterministic system | Manually trace dataflows and order workflow correctly to prevent untrusted data from entering sensitive plugins. |
| 2 – Bounded agentic workflow | Enumerate dataflows, identify ones with potentially untrusted data, explore isolation or sanitization options, and consider time-of-use manual approval of sensitive actions. |
| 3 – Fully autonomous system | Implement taint tracing and mandatory sanitization of potentially untrusted data. Consider time-of-use manual approval of sensitive actions. |
Conclusion
As systems climb the autonomy hierarchy, they become more complex and more difficult to predict. This makes threat modeling and risk assessment more difficult, particularly in the presence of a range of data sources and tools of varying trustworthiness and sensitivity.
Identifying the system autonomy level provides a useful framework for assessing the complexity of the system, as well as the level of effort required for threat modeling and necessary security controls and mitigations. It’s also important to analyze the plugins in the pipeline and classify them depending on their capabilities to provide an accurate risk evaluation based on the autonomy level.
FAQs
Q: What is an agentic workflow?
A: An agentic workflow is a series of AI models that are chained together to perform complex activities, enabling AI models to access additional data or automate user actions, and enable AI models to operate autonomously, analyzing and performing complex tasks with a minimum of human involvement or interaction.
Q: What is the risk associated with agentic workflows?
A: Agentic workflows present an element of risk, as they can be vulnerable to prompt injection attacks if untrusted data is introduced into the system.
Q: What is the purpose of the Agentic Autonomy framework?
A: The Agentic Autonomy framework is used to help assess and mitigate the risks associated with agentic workflows, including understanding the risks associated with increasing complexity, helping to model and mitigate the risks posed by agentic systems, and introducing how to model threats to agentic systems.
