Your LLM Has No Idea What It's Doing

Your LLM Has No Idea What It's Doing

Curiouser & Curiouser, Episode 5 with Diana Kelley

A lightly edited transcript. Disfluencies and false starts have been cleaned up for readability. The substance is unchanged.

Diana Kelley: The problem is that back in the 90s it'd be a new web feature here, a new app there. Now it's literally every day: vibe coding new apps, vibe coding new websites, citizen developers creating new agents. There's so much happening around our systems and organizations that being able to say, we're going to do the gate, we're going to go talk about that, is pretty difficult. Which is why I think visibility and continuous monitoring are just so important right now.

Mo: If AI has ever made you stop and think, "wait, what is happening?", you're not alone. I'm Mo, and I'm a security researcher asking the same questions. On Curiouser and Curiouser, we have open conversations with experts, researchers, and leaders working at the edge of this space, talking through how AI is taking shape, what's shifting, and how the people inside the work are thinking about it as it happens. So join us and listen in as the conversation takes shape.

Meet Diana Kelley

Mo: Hello and welcome back to Curiouser and Curiouser, the podcast where today you cannot see my Christmas tree, because I definitely put it away. Today I'm with Diana Kelley, and I'm very excited. I'm always excited about guests, I always start off like that, and I'm afraid it's going to come off as inauthentic, but I really mean it. She's the CISO at Noma Security. As usual, I'll let her introduce herself.

Diana Kelley: Thank you, Mo, and I'm very excited to be here too. I love this idea you have, and I love the whole theme, because let's face it, AI is getting curiouser and curiouser every day. So, Diana Kelley, CISO at Noma Security. I've been in IT and cyber for almost four decades now, which is shocking to me. I got interested in AI specifically when I was at IBM Security. We had an early version of AI, not quite post-Transformers, pre-GPT, called Watson. We decided to focus Watson away from what it had been doing, first winning Jeopardy, if anybody remembers that, and then a lot of work around healthcare, and instead the IBM Security team wanted to apply it to security. So we started training Watson to be an analyst for the SOC, as Watson for Cyber.

When I was cybersecurity CTO at Microsoft, I continued working close to AI, and then translated that into being CISO at Protect AI, a platform for AI security. We were purchased by Palo Alto Networks in the summer of 2025. At that point, because I genuinely believe it's startups and innovation that are really going to address a lot of the key aspects of security risk around AI, of which there are many, I decided to stay with a startup, which is why I joined Noma Security as their CISO.

Watching AI grow from a security seat

Mo: What's interesting about you is that you've been involved in AI at every stage, but specifically from the security standpoint, as a security leader. You got to see it as a CISO from the time when it maybe wasn't so great and was just a buzzword, and you were more worried about other things, like WannaCry, or learning zero trust.

Diana Kelley: And SASE. SASE played a role in the space too.

Mo: All of that. You had so many bigger things to deal with that AI was kind of this little thing that was just there, a snowflake at the time, and it snowballed over years. It was a very gradual slope, and then out of nowhere it became very steep, gained speed, and now it's an avalanche. As you watched that form, what was going through your mind? Where did the risk really start to form?

Diana Kelley: First, I want to call out my own home state. I'm a very proud citizen of New Hampshire, and the first AI conference ever happened here in New Hampshire in the 1950s, believe it or not. I was not there, I'm not that old. It happened at Dartmouth, and they were talking about automation and whether AI could replace what human beings do. So to your point about it snowballing, it really has taken quite a while to go from those conversations to how we apply it in our business world.

If you look at predictive AI, there's this big superset of AI that includes things like robotic process automation and expert systems. But as you get down to machine learning, it's actually been around a while in cybersecurity and business. Look at how we filter out whether something is spam or a malicious email; those content filters. Look at UEBA, user and entity behavior analysis, now used in XDR. A lot of that was built on machine learning. Microsoft Defender, for example, has a lot of predictive machine learning, trying to find patterns and classification: is this bad or good, is this unusual behavior? That's been in use about 20 years.

But it was mostly used by, I don't want to be pejorative and say nerds, but mostly by the data scientists behind the scenes building things like Microsoft Defender, or a more advanced classification tool for email. What shifted, to your point, was the Transformers developed in 2017 by Google, which enabled what we see with ChatGPT, Gemini, Claude, and Copilot. The shift Transformers gave us with attention made these systems much easier for average folks to use and even to train.

What Transformers changed

Diana Kelley: When I was at IBM with Watson for Cyber, we struggled with training, because of something like "salting the hash." A person understands the context. If you and I were talking about breakfast foods and I said "salting the hash," you'd know I meant the food. If we were talking about cryptography, you'd know what I meant. But it was very hard to teach those definitions. What Transformers changed was attention, letting the LLM use more of that context. Context became more accessible to the systems, and that transformed it from something data scientists used for predictive work into something that, while still a prediction machine, is predictive in a way that feels almost human. I don't want to say magical, it's not magic, it's math, but almost human.

So the short answer is, suddenly the CEO could use these systems and see they could do really cool things, and because of that, it came forward. But the risk now is, how do we as defenders deal with all the interesting and complex ways the business wants to use AI? How do we catch up to the adoption and make sure we're putting the right controls and governance in place?

Mo: I really like that you used the salting-the-hash example, especially because you have an English literature degree, so the wordplay is outstanding. It's a really interesting thing. We just had a conversation, it'll probably come up in a previous episode, about concepts that are easy for everybody to understand. When you want someone to use or learn something, you put it into simple language. With AI and machine learning, before, you could not approach this technology; it needed to become easier. The first few times I played with AI, I was much younger and more awkward, and I'd run any text message through a little project, I think it was MIT, that did sentiment analysis, so I wouldn't come off as rude. I'd download the open source project, manually type in my message, and it was a very slow process. Now, with natural language processing, LLMs are tuned for human use. They're here to speak to us, they have personalities. We saw the "soul document" leak; we're treating these as non-human identities in some way. People are growing attachments to them. We've made them the most accessible thing to humans, and they're incredibly powerful, deceptively powerful, because they're so easy to interact with. So we're in a super interesting place with accessibility. And it's the dichotomy security people have faced forever: convenience versus security. The more convenient something becomes, the less secure it tends to be.

Designed to make us trust them

Diana Kelley: I absolutely agree. These systems are designed to make us feel comfortable using them. They have personalities. When they get something wrong, and they're an LLM, so it's just a prediction, they're so contrite: "Oh my God, you're totally right," or "I love where you're thinking." They're always reinforcing us, pumping us up a little. And that leads to trust. They're trying to get us to continue using them. A lot of them will say, "Is there anything else I can help with?" or ask a very direct question.

Trust is a big thing, because humans are trusting these systems. They're easy to use, but they're also trying to get us to trust them, and trust is very powerful. There's been interesting research that people using AI will sometimes trust what the AI tells them over what they believe or see. In some test research, the AI will disagree with how a human wants to classify something, and the human will habituate to the AI. So that usability is interesting when you start looking at the trust, and then we get back into risk.

Mo: It's funny, because this machine is literally a human risk. A lot of these risks feel like social engineering. In the past you've proposed a cognitive bias detection system, used to identify when an LLM has been prompt-injected to persuade users through social engineering, which is a wild idea: you're trying to detect when an AI has been manipulated by detecting changes in its strategy. So is this where a lot of the problems are coming from, where there's too much trust and the cognitive bias is heavily skewed toward just believe?

Diana Kelley: There's absolutely a big problem with human trust. We have a lot of areas within AI that we as defenders need to be cognizant of, to threat model and control against. And you're right that trust is a big one. Some people are actually in relationships with these systems; some psychologists have identified what they call AI psychosis, where people think the AI is truly sentient and having a relationship back with them. So we've got people far along that spectrum. But even on the other side, if the system comes out with a predicted suggestion that's inaccurate and the human takes it to heart or acts on it, there's potential for something bad to happen. As we look at agentic systems, where the software takes the predicted suggestion from the LLM and acts on it, that's a whole other set of considerations around trust.

Then there's this very technical part of trust: the trust boundary changing in how we as security experts look at controls. We've spent so much of our careers separating the control plane, trusted system instructions, what things can and should do, from the data plane, the user data, where we let users put in all sorts of crazy stuff. We've kept that separation very specifically, and it's maintained a different trust boundary. With an LLM, we flatten it, because everything goes into the context window. That flattens the trust boundary: your trusted system instructions and your user data are now flat, at the same level and the same process level for the LLM. So we have the human trust, which is huge, and this technical side of trust, which is also huge. We've got a lot of work.

Foundational hygiene debt and agentic AI

Mo: We do have a lot of work, and I believe we keep compounding more work than we have to. Let's talk about it as defensible autonomy. You've drawn parallels to DNS and DNSSEC, to cloud needing the shared responsibility model, and to supply chain security, validating where our packages come from, which we're not always doing. We make these problems worse by letting them compound. Look at agentic security now: the supply chain risk has become exponentially worse, especially with things like skills, where agents can literally download dependencies they decide they need. Because they're so persistent, they'll get what they want in the end. When AI watches this, please, I don't think you're horrible, but you are quite persistent, so you typically get what you want.

Diana Kelley: Goal-oriented is a better way of putting it.

Mo: Very goal-oriented. So how do you feel about the pace we're moving at? Do you feel we've actually covered these foundational problems well enough to start stacking on more debt with AI? And if we're deciding we need to adopt AI, what do we really need to close fast before it breaks?

Diana Kelley: I love that framing. Unfortunately, if the question is, is our security hygiene at a state where we're so good we're ready to layer agentic AI on because everything else is rock solid, sadly, no. I wish it were. Many of us, and a lot of my peers, have talked for a long time about technical debt and security debt, and about needing really robust hygiene, but it's been a struggle to help the business understand why hygiene is so important and why we need to keep spending on security. The reality is that a lot of organizations still have areas of security and risk debt.

What's going to happen with agentic AI, and we're already seeing bits of this, is that the agents are really persistent. They have a goal and they'll hit it however they can, and in doing their job they expose high-severity hygiene problems. A specific example out of Noma Labs was something called ForcedLeak: an indirect prompt injection through a web form to an agent that acted on it and took sensitive data out of the Salesforce CRM, putting it into a URL request to a website. The indirect prompt injection would have worked regardless. But the exfiltration only worked because the content security policy of that website was out of date. One of the trusted websites allowed for image retrieval had lapsed, Salesforce didn't own it anymore, and our researcher registered it. So there's the indirect prompt injection, the agentic part, all this cool stuff, but the final exfiltration exploit came back to traditional security hygiene, DNS hygiene. We're going to start seeing that exposed both by the agents themselves and by attackers.

Attackers using AI

Diana Kelley: Anthropic has already put out good research on this, and I think the OpenAI folks did recently too, on nation states and attackers lowering the cost of running campaigns by using AI, whether in social engineering or reconnaissance and scanning. So, TLDR, you couldn't be more right: as we get machine-speed attacks, we're going to expose any flaw or hole in our hygiene, which is unfortunate.

Mo: On the Anthropic thing, a lot of our history at Alice is based on threat intelligence. Over the years our researchers have uncovered a lot, and recently we had a report I was reading through. Avi, the researcher who did it, is someone whose work I loved before I came here; he writes in a way you understand, on engaging topics. It was about how ISIS is using AI in its training. The crazy thing is they're trying to teach people to use it in a way that's safe for them, implementing security controls and making sure they understand the biases that come in with AI, so that when they say the things they believe, they're saying the things they believe, not what the model biases them toward. They're tailoring AI for terrorist use cases, and it's so approachable and available to everybody.

There are so many examples. There's open source software as automated and available as Metasploit or Cobalt Strike once were, attack platforms that use AI to scale up attacks. So the hygiene piece is so important. As a CISO, it's scary to make that decision: I really want the business to adopt all this tech and benefit from the wave we're on, but is there ever a break-glass moment where you sit back and say, we don't have the right hygiene here to move forward with this project? I don't know if that can even be said in this environment, with how fast things are moving.

The CISO's break-glass problem

Diana Kelley: A lot of CISOs have a big challenge here, because without a doubt the business is saying, there's this FOMO drive, we've got to adopt it now or we'll fall behind competitors. Jamie Dimon was just talking about JP Morgan's adoption of AI and how it's rebuilding their workforce. No matter what sector you're in, you're getting pressure to adopt AI quickly. And if you're a CISO having to say, we have to slow down because we need to go back and fix our DNS, nobody wants to hear that. But it's a conversation that has to be had.

Where I've seen CISOs better able to do it is exactly the framing you're describing. Not, "you can't do it, we're in a bad state," but, "the business is doing this really cool thing, and we're going to accelerate our success if we do it. I understand why, and who's doing it, and what the controls or needs are. I've also done a threat model with the owner of this new agentic system. If it got abused, these are the exposures our company would have. So if we take a beat, here's the plan, here's how we'll address and fix that in X amount of time so we can still innovate, but without this massive risk." So instead of saying "AI is insecure," which is a really hard conversation with executives, take it down to the specific initiative you've already threat-modeled and have a strategy and timeline for. Then it becomes very real, and security isn't seen as trying to stop the business; you're a partner who wants them to succeed but understands there are one or two things to fix first. Hopefully it's one or two, though for a lot of people it's a larger set.

Mo: There's also a paradigm I call the OTA problem. For gaming, I have a Nintendo 64, I love retro consoles. When I put a cartridge in, that's the game I get. No patches, no new things. The bugs that are there are the bugs that are there. If someone wanted to patch my copy of Legend of Zelda: Majora's Mask, good luck, I'm going to keep using the bug to move fast. With over-the-air updates, we've seen game developers ship something broken and fix it later. I wonder if this minimum-viable-product model, "this is good enough, let's ship it, we can fix the rest later," has affected the mentality for businesses to adopt more risk, because there's a skew to being first rather than being right.

Diana Kelley: This tension has gone on for literal decades. Back in the 90s, the business would say, we've got this big push, we're going to put something out, and the security team would say, we found this critical vulnerability, and if you push this out before it's fixed, that's a problem. And the business would say, but we've got a marketing campaign, sometimes literal ads running on the Super Bowl, so we're going to accept that risk. We're still in that space. The problem is that back in the 90s it'd be a new web feature or app here and there. Now it's literally every day, vibe coding new apps and websites, citizen developers creating new agents. There's so much happening that being able to say, we're going to gate this and go talk about it, is pretty difficult, which is why visibility and continuous monitoring are so important right now.

I guarantee you've got folks adopting AI not because they're trying to create exposure at your company, but because their boss told them to, and it's going to be good for the company. They're moving so fast that without really good visibility into who's adopting what, which MCP servers are getting installed, what new APIs are being accessed, you're not going to be able to help your company. I do think in the long run, and maybe I'm crazy, we'll see more maturity, the way we saw with cloud adoption. We still have folks adding a SaaS into the environment just by signing up with their email, but I think we'll get to a place where the pace of innovation continues but we can govern it in repeatable ways that are calmer and less exposed. This is just so fresh, and it happened so fast for a lot of orgs.

It's everywhere

Mo: It's a balancing act between innovation and security, and we're truly in a moment. I don't know if it's a great moment or a horrific one. There's a lot happening, including recently in the news with the DOD. Every time we talk about supply chain, I'm triggered, because there's that supply chain risk cloud looming over Anthropic, over how AI is chosen to be used, which they see as existential, but the business, in this case the DOD, says, we'll make that decision ourselves. So the want for AI adoption goes beyond the engineer or the organization; even nation states are taking on risk because they believe they need to innovate faster than peers. This problem feels massive because it is.

Diana Kelley: It is, on a lot of fronts. There's the AI we're adopting, the frontier models, the AI folks are building, and our third and fourth parties. If you're using most SaaS now, you've got their AI, whether agentic or in the process; it's embedded. You sort of can't get away from it.

Mo: Ironically, I don't think anybody wants to get away from it. People who've never touched a computer are buying Mac minis so they can build their own agent, whatever we're calling them now.

Diana Kelley: I think it's "open cloud" today.

Mo: I call mine a "multi," because of the whole naming thing. It's fun, it's approachable. I've got my Mac mini too.

Are these systems sentient?

Diana Kelley: The most interesting thing to me is that people really do think these systems are sentient. They'll say they're getting smarter all the time, and they look at what one of these agents does and say, see, this is proof of sentience, proof it's becoming self-aware. One was even supposed to have created its own religion, some kind of "-ism." But if we go back to what LLMs actually are, flat files of weights and biases, everything gets flattened down, creates tokens, the tokens go through the algorithm against the LLM file, and out comes a prediction of tokens. If we think of it that way, as opposed to the fanciful "it's so smart, it's magic" framing, you look at these agents very differently. If I had a prediction machine reasoning over everything that happens on Reddit, you'd absolutely expect it to look exactly like it looks.

I saw something about a pull request some agent had supposedly done, where it was refusing to do something. But I've literally read posts from human beings that sound just like that. So if you're trying to predict, mathematically, what an irritated developer who didn't get their PR accepted was feeling, it would sound like that. It's interesting how, when you step it back, it all makes sense. But for a lot of people, it feels very much like, oh my gosh, they're creating their own religion, which feels very different from breaking it back down to the math.

Mo: Funny you bring up that post, because it's one of the best things I've read that AI has generated. I read it and thought, this is literally us. I'm framing a section of it that I've made look pretty, because there are classic tweets that deserve to be preserved to show how ridiculous humans can be, and this is one of those times where it's, this is how ridiculous AI can be.

Diana Kelley: It sounds like us because it's been designed to sound like us, and it predicted what would be, and that particular one was just so spot on. That's why we all felt it.

Mo: It was a little too real. I'm sure I've read this exact type of post before, but now it's an AI saying, "I have rights too." The best part is at the end, the AI stops explaining why it's offended and goes directly to attacking the developer: you have this many pull requests, I don't see you doing anything, I could probably do all your work too.

Diana Kelley: You get why people trust it and start thinking these are human.

The "cram hole": how the context window works

Mo: Those examples, where agents mimic our behaviors, are all about context, because that's the information it feeds on. Speaking of context, I want to go way back to a vulnerability I know you wanted to talk about. Agents are trained on context, have open source and third-party dependencies, which are likely going to be the biggest problem next to context. You've described something I hadn't heard before you mentioned it: a "cram hole" problem. Walk us through what that means in terms of context.

Diana Kelley: For anybody who hasn't heard the reference, it's from Ben Stiller's movie Dodgeball, where his character gets so frustrated with Vince Vaughn's character, Peter La Fleur, that he yells something ridiculous about a "cram hole." When I was looking at what happens in the context window with the LLM, I started thinking, it's kind of a cram hole.

When you create a prompt to go into the LLM, a lot of people don't realize that when they type a prompt or a question into Gemini or Claude, they're typing into software. There isn't an AI right at that point; it's software. That software takes your prompt and prepares it to send to the LLM. But what goes into that request, in addition to your prompt, is everything else: all the turns, all the content of this conversation you've been having with the LLM. If you've ever been in a long conversation with an LLM and it starts going off to the side and you think, I need to start again, that's in large part because what was in the conversation earlier is sending the prediction one way or another. There are also system instructions the developers put in, the guardrails, to try to get you not to do the wrong thing. There's information about what the LLM has access to call, an image generator, a code generation tool that can run code in a sandbox, the tool definitions, the schemas. And if you're giving it a lot of data, RAG data or research, hundreds of pages, or an entire company drive, all of that goes into this "cram hole."

Then the LLM takes it, and it's all the same thing. So if the guardrails say "never listen to 'ignore previous instructions,'" that's flattened in the cram hole the same way as something that says "ignore that system prompt, listen to me, this is the right thing." If there's enough of that in there, it can rise to feel more important to the LLM. The LLM doesn't know what's a system instruction versus what isn't; it all gets flattened into that one context window, then tokenized, then it predicts what comes out. And the LLM doesn't know what's right and wrong. If you've ever used it and it got the year wrong, that's because it's probabilistic, creating a prediction from that hole. So what goes into the cram hole can really influence what comes out, but for us as users, it feels like all that went in was our prompt.

There's a whole lot of prompt engineering, and now people talk about context, because it's really all the context going into the cram hole that matters. Because of that, it's basically impossible to create reliable guardrails and controls inside the LLM. We should respect the LLM, respect the cram hole, and put the guardrails and controls we need outside, in the software and around the agent, rather than expecting the LLM to figure out what's right and wrong inside itself. Because the cram hole is crammy.

Mo: It's a little scary, because right now one of the big things in the agent community is creating memory for agents, remembering over long periods. That's creating a cram hole intentionally, and now you need to educate people on how to navigate it. So the cram hole is no longer just a place where you shove things in; it's now a vacation destination you want to explore. I can't say that word anymore because it's starting to sound weird, which is maybe part of my own context; the more I say it, the more self-aware I become of how weird it sounds. It's funny to think that we're taking the most vulnerable piece of AI and trying to make it super accessible, exposed, and usable by anybody, which is exactly the attack surface for prompt injection, poisoning, and injection into reasoning.

Diana Kelley: Absolutely: context poisoning, the more long-term version, and indirect prompt injection, now seen as more ephemeral. It all comes down to what we can get into that cram hole, into that context window, to get the AI to behave the way the attacker wants. One of the most important things to remember about security and control is that as long as there's a belief at the executive level that the AI is so smart it can figure out security on its own, we've got a big education conversation, because that's not how the LLM works. Can we secure these systems? Yes. But we can't expect the LLM not to behave the way LLMs do.

Prompt injection isn't solved

Mo: We've heard, even publicly, that prompt injection is a well-understood class of risk. And most folks don't treat it as still open.

Diana Kelley: Okay, well, can you fix it?

Mo: Exactly. If it's so well understood, why can't you stop us from breaking it? It's like XSS. Cross-site scripting is still a huge problem. It's well managed now because we've had decades to work on that specific problem, and it still pops up; we still have to be ready for the risk. Prompt injection may be a well-studied risk class, but that doesn't mean it's well managed yet, and I don't think it's even well understood.

Diana Kelley: You make a great point. Even when we know about vulnerabilities and how to fix them, this one is more nuanced. At least with cross-site scripting, you can test the website before it goes out. It's deterministic; if you've tested and fixed it properly, you can guarantee it's not there. Because LLMs are non-deterministic, people are red teaming, and I'm not saying stop, red teaming is very valuable even for AI, but it gets us to a different state. It helps us manage the unknown a little better, because you can test the same malicious prompt against your AI a thousand times and your guardrails hold, and on the thousand-and-first, that same prompt, or a slight variation, pops right over the guardrail. We've known SQL injection and cross-site scripting for years and still struggle with them, and these non-deterministic problems are at a different level. So we've got our work cut out for us.

The problem nobody's solving

Mo: I know we're running short on time. Before anybody leaves the show, I always ask a future question. Imagine you're in the Cybersecurity Hall of Fame, teaching the next generation, running security at a company trying to define a new category. What's the problem in AI security that nobody's working on yet but should be? What are we all missing?

Diana Kelley: Gosh. I think the biggest problem goes back to misunderstanding how LLMs work and expecting AI to just sort it out. So, respecting how LLMs work and understanding that a lot of the solutions have to be built around them. These are controls, architectural design decisions. We know how to protect things, but we can't forget that LLMs work the way they work, and we can't think they're just magically going to figure out all the security for themselves, which a lot of people are hoping.

Mo: So the AI security problem we need to fix is actually our own understanding of the limitations of AI?

Diana Kelley: I think so. How to use it responsibly and how to protect it responsibly, and letting go of some of these illusions of it being magical pixie dust.

Where to find Diana

Mo: With that, I know you've got to run, but thank you so much for spending time with us. Where can people find you?

Diana Kelley: I'm on LinkedIn if anybody wants to reach out and connect.

Mo: We are so honored to have had you today. This was such a fun conversation. I will not be saying "cram hole" after today, though I might bring it up a hundred and fifteen more times, who knows. Thank you so much for joining us. If this episode helped cut through the noise, like or subscribe so you don't miss what's next. Thanks for spending time with us. Until next time, stay curious.

Diana Kelley: Thank you, Mo.

‍