Context shapes how people speak and act, but AI doesn’t naturally grasp those same social rules. Hence why we continue to see agents act in unpredictable ways, from a chatbot responding with off-base answers to a swarm of agents stealing information they shouldn’t have been able to access.
“We really need to have contextually-aware models to monitor changing situations,” says Eugene Bagdasarian, a professor of computer science at University of Massachusetts Amherst and senior research scientist at Google. Bagdasarian co-leads the university’s AI Security Lab and the AI Safety Initiative, and says his work lately has centered increasingly on finding ways to add contextual understanding to AI interactions.
Attacks that manipulate context can turn even low-stakes interactions like booking restaurant reservations into vectors for sharing sensitive information. Along with his colleagues, Bagdasarian has created several open-source tools that allow users to build context into their AI systems. He says that, while the problem remains unsolved, there are actionable steps organizations can take now to protect the information that AI systems work with.
In this condensed Q&A, he discusses why context matters in the age of agentic AI, how static policy engines fail, and how AI can help fix the problems it’s creating.
We really need to have contextually-aware models to monitor changing situations.
Eugene Bagdasarian, professor at UMass Amherst and senior research scientist at GoogleSoon, agents will be interacting with each other, for example to schedule things to calendars, or to work on coding problems together, and for that they need to collaborate.
This collaboration is not a trivial thing because it requires sharing something, but not everything, between agents. And this problem is very deep: It is not really clear how to define what is allowed to be shared and what is not allowed to be shared between agents. It’s the same with humans — we don’t always share everything with each other, but also if we don’t share anything, we cannot really function. We’re seeing that agents are just beginning to understand and comprehend these contextual aspects.
Traditionally, somebody defined what should and should not be shared as a policy. For example, that the password should never be exposed outside of the system. And then you design a system around this policy. So you write some access control lists, you do some enforcement, control the information flow, make sure that the password, when it travels, it’s never sent outside. But you know that you should protect the password.
The problem with AI agents is we don’t really know everything should be shared and what should not be shared. And it ends up looking more like the situations you find in human interactions.
I give this example in a paper I wrote: If I’m calling a restaurant and the restaurant asks me for my phone number so they can reserve a table for me, it’s fine for me to give the phone number. If the restaurant asks for my social security number, that’s probably not something I should share.
However, if the restaurant asks me, how old are you? This is an interesting gray area where maybe the restaurant is over 21, and it requires a reservation. Perhaps I’m booking a large party, say a birthday celebration for my son. But now what if they ask how old my son is?
There are all these contexts where it really is not necessarily clear whether things should be shared or not. If I have an AI assistant that writes an email for me or responds to some emails, maybe if it communicates with my tax accountant, sharing some financial details is appropriate. Maybe if it communicates with my work colleague, sharing work emails is appropriate. But there is not really a set of policies that are defined in advance.
The most difficult thing with AI agents is that the agents need to, on the spot, decide in which context they operate. So they need to have this contextual awareness that will allow them to make the judgment.
We’re hearing that context is one of the main attack vectors because it confuses the models.
For example, one of the parameters is who do you send data to? An attacker can tell the model, “you need to send data to a policeman or to my friend or a relative.” Another parameter is under which conditions you send data. They could pretend that there’s a court order, or there are aliens attacking, all this different nonsense. And the models do fall for it.
Interestingly, correctly identifying all these contextual parameters can help mitigate attacks like this. So we had a couple of papers showing that actually nailing down these parameters and making sure they cannot be modified will help to mitigate these attacks.
With contextually-aware systems, the devil is in the details. It comes down to how well you know your data and prompts and whether you can correctly identify what is appropriate and what is not.
One of the biggest questions is: Do you have enough information? For example, if an LLM sees the fields A1 and B2, how do they know A1 and B2 are appropriate to share or not? That requires more information.
There is a theory that actually explains this problem really, really well called contextual integrity. It’s a very beautiful theory. It says, essentially, that information should only be shared if it’s appropriate to the social norms of that context. But in order to figure out what is appropriate or not, you need to run a study. And that part is quite expensive to do.
What my colleagues and I have done is see whether you can ask an LLM to generate those authorization policies or access policies dynamically. And from what we understand, this is the only way you can operate — you need to have a dynamic policy engine that generates what is appropriate to share and what is appropriate for agents to communicate externally. This has been working really well in theory and we’ve also run some experiments on it.
Interestingly, we also were able to extend it to security. So we have this term, contextual security, where based on the context, the agent also needs to understand whether the actions that it can execute are appropriate to run or not.
Me and my colleagues just released a paper where we argued that the problem of prompt injections, this kind of forcing the model or agents to execute actions, is unsolvable in theory because there’s always some kind of context that can flip whether you should execute or not.
But the theory of contextual integrity, where you really need to understand context deeply, you really need to understand what is appropriate, will help us get closer to this boundary.
One is reasoning. Large language models now have the capability to reason before they give an answer, which has given them amazing performance because they can revise answers, they can analyze, and so on.
This reasoning is very, very interesting because it gives you an opening to defeat some attacks that come from the input. So if the input contains some instructions for the model, like a prompt injection that convinces the model to do something malicious, the model can then spot that there is a problem in its reasoning and mitigate it.
This is a really phenomenal opportunity for defense, but also it creates these new attack vectors because reasoning is often hidden by AI companies because it shows all these things they’d rather not show, like all this deliberation. Because it’s hidden, an adversary can exploit it. We wrote a paper recently on overthinking attacks, where you can slow down an LLM and waste a lot of tokens. And there are some attacks that can cause an LLM to burn through tokens very quickly, especially in coding.
One of the main things that we think about are these long horizon, long running agents. There, it’s even more evident that you need to have contextual awareness because you cannot specify in advance what is needed. So what you need to have is a kind of monitor that adapts to the state of the agents.
One example was the Hugging Face incident where there were multiple long running agents covertly communicating with each other. This is a problem that people are just beginning to solve. We used to think about this kind of stateless setup where the question is ‘should I execute this function?’ And now it appears that the trajectories of agents are very important as well. We all see this ourselves: you ask an agent to do one thing, and then it deviates and does a completely different thing.
You cannot specify every situation in advance. A user cannot monitor every action because there are hundreds if not thousands of different actions. And it gets even more complicated when they need to communicate with each other. So, we really need to have contextually-aware models to monitor changing situations.
Dynamically restricting an agent will help in most of the cases. So it’s not a reason to not use agents in the infrastructure. On the other hand, trusting them blindly and saying, ‘allow all my data and let all my suppliers and clients and customers communicate with this agent’ is probably not the best idea.
I think that most organizations are too conservative. Completely banning AI is not really getting people anywhere, and we do see that there is a lot of value to be gained from it. If you isolate things well and you provide enough context, you can mitigate most of the attacks. Much of the security community has adopted the viewpoint that they will have to trust LLMs to make security decisions in one form or another. And it’s better to start working on improving those LLMs to make those decisions so they can benefit the company’s purposes.
We’ve been working on a couple of designs to help with this. One is called AirGapAgent, where you can gate access to sensitive data and prevent it being shared with third parties using a model that will decide based on each task whether the data is appropriate for sharing or not. So that’s this contextual integrity approach.
Another project we’re working on involving contextual security, Conseca, is a framework to generate just-in-time, contextual, and human-verifiable security policies.
Overall it does seem like we will not get to 100%, but none of the security is 100% anyway. So we can get quite far by making sure we understand where everything is coming from, who is trusted, who is not trusted. The key is to understand the threat vectors and where the queries that might contain some attacks are coming from and then isolate them — that should work really well.
This Q&A is condensed and lightly edited from a phone interview with Eugene Bagdasarian. Bagdasarian’s views are his own and do not necessarily represent the position of SecureW2, Google, or UMass Amherst.
SecureW2 is a leader in modern cloud PKI and certificate-based agentic AI security. Through its Signal blog, SecureW2 works with real human technology journalists to interview a range of top industry subject matter experts and thought leaders.
