The Problem With AI Observability Nobody Wants To Admit

The Problem With AI Observability Nobody Wants To Admit

Curiouser & Curiouser, Episode 8 with Alison Cossette

A lightly edited transcript. Disfluencies and false starts have been cleaned up for readability. The substance is unchanged.

Alison Cossette: It's shocking to me that observability of our systems is not table stakes. You need someone on staff who can understand and look at that observability at a system level, discern the impact, and communicate it to the relevant business. You need to make sure you've got internal observability with anything you build or buy, and someone internally who can leverage it at an intelligence level, not just a log level.

Mo: If AI has ever made you stop and think, "wait, what is happening?", you're not alone. I'm Mo, and I'm a security researcher asking the same questions. On Curiouser and Curiouser, we have open conversations with experts, researchers, and leaders working at the edge of this space, talking through how AI is taking shape, what's shifting, and how the people inside the work are thinking about it as it happens. So join us and listen in as the conversation takes shape.

Meet Alison Cossette

Mo: Welcome back, another week of me and my friends. I'm Mo. You've probably had enough of me already, but someone you haven't had enough of is my friend Alison Cossette, founder of ClariTrace and an overall AI nerd. She's super cool; I met her at a conference and knew we had to talk more. I'll let you talk about yourself, because I've found I'm horrible at talking about other people.

Alison Cossette: I'm not so good at talking about myself either, so between the two of us, we're in a spot. I'm Alison Cossette. I started out as a healthcare data scientist and have been in the AI world since before it was cool, when it was just a bunch of math nerds in a corner building algorithms and waiting for someone else to put them in production. The world has evolved, as have I, and most of what I'm doing these days is deep AI research, seeing where we're going, trying to build systems that let us have an AI architecture and ecosystem that makes what we do for humans better.

The black box as an excuse for bad design

Mo: You've said it's easy to call AI a black box, and that it's an excuse for bad design. Those are fighting words, especially since the industry kind of loves the complexity, because it makes the solution seem more interesting. So tell me more about that thought process. What is the black box smokescreening?

Alison Cossette: Am I saying there's no black box? Not really. But what I want everyone to understand is that at the end of the day, everything we've built is technically math. Very complex, very high-computational math, but ultimately math. In a traditional ML linear regression, this is the formula for how much your house should cost: so many dollars per bedroom, per square foot, per lot size. Very explainable. Now we're in a generative moment where we use foundation models and don't necessarily know exactly where each piece came from. The good news is that while those weights aren't open in most cases, people are doing phenomenal work around explainability even within foundation models.

Think about what Anthropic is doing with mechanistic interpretability. Did you ever hear of the Golden Gate Bridge demo? They found within the model where the Golden Gate Bridge was, and they cranked it up. I won't get into the deep linear algebra of neural networks, but they cranked up the Golden Gate Bridge feature.

Mo: No, I haven't. What's that?

Alison Cossette: They asked it, what's two plus two, and it said, two plus two is two Golden Gate Bridges, and another two beautiful Golden Gate Bridges gives you four glorious Golden Gate Bridges. I'm paraphrasing. What they found is it wasn't just the Golden Gate Bridge, it was the awe of the Golden Gate Bridge. That got me thinking. We know it's math, we recognize there are activation functions in different places, and we can clearly see, when we have insight into these models, that we can manipulate them. So when we follow that forward, what does it make possible? It's very plausible that the ability to steer the AI, to steer even the foundation model, is not as far off a concept as we think. That's how I ended up at the concept of control-native AI, which is where you and I met.

What control-native AI is

Mo: Control-native AI was really interesting when you explained it. For folks who haven't heard of it, give a short rundown in your own words.

Alison Cossette: The analogy I like to use is that AI is somewhat like a bowling ball. It has a lot of power and force behind it. It could do great things. We throw it down the alley and boom, things happen, and it's glorious and magical. Then we put on guardrails, because we realize there are certain things coming out of generative AI we don't want. It's built on what humans have said and written, and not all of that do we want repeated. So we put in guardrails: you're in the bowling alley, the ball doesn't go into the gutters, great.

But as we move into agentic systems, which are generative, they're not bounded. What happens if agents start to swarm? We've seen them do interesting things, and I'm sure we'll hit on that today. We've seen agents in labs working together to manipulate markets when given trading capabilities. It's almost as if the bowling balls can now fly. They're in three dimensions, moving all over. Now if the bowling ball's mission is to knock over the pins, what if all the balls start working together and destroy the pin resetter, because they're focused on this one mission? That's the optimization they're going for, the reward they're optimizing for.

So I'm thinking, if we take it back to mechanistic interpretability, what would it mean if, instead of guardrails around where the balls were going, we had sensors inside? What if we could see the directions they were heading? What that means mathematically we can get into, but that's what I mean by control-native systems: how can we build not just the models, but the systems around them, to understand where they're going, not just where they've been?

Mo: So control-native AI, at a high level, is about all the things around it, so you can understand the directions it goes: the right observability, the traceability, and understanding all the factors that cause it to go a certain direction.

Alison Cossette: Yes, and the important thing is to see where it's going. The first thing we need is observability, and as an industry we've done that really well. There are phenomenal companies; Arize is really great, and a lot of folks do that well. From there, we need understanding and interpretability. For agentic systems, in the last couple of months that's starting to catch up. Only when we have that clarity and understanding can we start diving into what it means to steer. That's where we need to look forward. I'm a big fan of the digital twin, of understanding the physics of the system: which tools are we using, which agents are collaborating more or less, what are the live physics of the actual system. That gives us the understanding of where it's going, rather than just a log-level output.

Guardrails as fences in space

Mo: I remember we talked about how guardrails are the hot topic for stopping output, and you said guardrails are useless, they're the worst, guardrails are fences in space. That made a lot of sense. They stop one thing, but AI is pulled in so many directions, and you can't always predict where it's going. Guardrails are almost exclusively made because you know exactly where the response is going, but you miss the tertiary effects. While the response may be blocked, the prompt that got you there could be causing a shift or drift in how the system responds in the future. So you're opening up to something like AI-native control, looking for the AI system's intelligence piece. That's not something companies have ever had to budget for or think about. How do you even walk into a room with a CTO or the new Chief AI Officer and say, you need more AI systems intelligence? How do you demonstrate the value of traceability to a room full of people who just want the best models, want them to work better, and want to create business value, or as the meme says, shareholder value?

Alison Cossette: We've been working on ClariTrace for almost two years, and it's not an easy conversation. We were told early on, you're ahead of the market, people aren't ready to take that on. It's starting to come into the conversation a little. I had a conversation with one of the heads of AI at a big consulting company, and they said, we've built this great platform with hundreds and hundreds of agents, but we don't know what they're doing or how they're collaborating. I was like, the moment is coming. People aren't thinking about it yet; they don't know it matters or that it's an issue. When you think about where agents were a year ago, last February was the first AI Engineer agent summit in New York, and a year later agents are the topic of many Super Bowl commercials. So agents as an everyday commonality are here. What we're behind on is the risk debt we've built up through this rapid adoption. We're starting to see the challenges people run into when they let things run wild.

The sad thing is that until it goes wrong for someone at a really high level, people aren't going to talk about it. People don't think about life insurance when they're 18; they think about it once they have children and something's at stake, or after they've lost someone. People aren't there yet because it's still new and they don't recognize what's coming. So part of what's important to me is to get out and talk about it as much as possible, to get the conversation moving, so chief AI officers and CTOs can start thinking about how to address this at a macro level.

Losing the old rigor to velocity

Mo: It's not an easy conversation, especially now, with things moving faster than any of us anticipated. If we step back to how we got here: last year there was the first AI engineering summit; go back to 2022 and you've got ChatGPT getting big; before that, people were installing Anaconda and taking a long time to build rigorous pipelines around machine learning. I was at a shop doing this, and there was so much concern around compliance and data sovereignty: where the data resides, data integrity, validation, testing. Nobody wanted anything in machine learning to go astray even 1%. Everything, from a forecasting or predictive model to understanding how a model reached a conclusion, needed to be sealed. It feels like we've lost some of that trying to keep up with velocity. So what will it take to make the pendulum swing back to that rigor, or do we even need to?

Alison Cossette: The level of rigor and responsibility we had building them by hand was immense. That's why, when AI first became a commonly used tool by people who didn't understand it, I felt like we gave middle schoolers cars to drive. You don't know how to drive it yet, slow down, can we start with a go-kart? The most important part you raise is velocity. Because the technology, the development, and our ability to deploy are moving at such a breakneck speed, it's very hard for a business, a C-suite, and leaders to know how to proceed. We don't even necessarily understand the entirety of the risk, but we know what our competitors are doing, and we're familiar with the risk of being left behind. Especially for small and medium businesses, that could be existential.

Unfortunately, that's the nature of how fast things are moving. People have a good idea of their business risk and know how to manage it, and while it's not necessarily a black box, the depth of technical understanding isn't there for the people making decisions. That's part of why the Chief AI Officer is becoming predominant, and I'd encourage everyone to have someone in your organization who is the AI expert, because you need someone to bring that understanding and become an important part of your risk-identification profile. At the conference where we met, I said, in generative circumstances the risk plane is infinite. It used to be very clear risk moments: getting to the data, is it valid, is somebody trying to hack our system. Now the risk is more ethereal. Unfortunately, as with most things, it won't become a common conversation until something goes tragically wrong. In the military they say, until somebody dies; in business, until somebody loses a lot of money. One of those will occur, and everybody will say, we should probably look at this. The good news is lots of people are doing phenomenal research, so the information, tooling, and answers are out there. People just have to be ready.

Why guardrails don't work

Mo: I wish I were half as persistent as AI, because it's so driven to a goal. It'll find any path to that objective. I've been in startups, and there's always that one engineer who finds a way to complete a task that would flag security if we could catch it fast enough. In this case, those flags aren't being flipped at all, and these agents will find ways. It's not if, it's when, because they're 24/7 always trying to complete a task. The area of risk is so hard to imagine, and finding a good control for it is really difficult. That's kind of what's top of mind for you.

Alison Cossette: I always go back to the series finale of Silicon Valley, probably 10 years ago now, where the AI got so good they had to kill it because they knew they couldn't unleash it. We clearly learned nothing. But to your point, it really is relentless, and that's why guardrails don't work. Guardrails don't work because the AI will come up with so many other places; there's no way we can imagine all the edge cases. And then what do we do? We have one AI trying to figure out all the ways to stop it, and another trying to find all the ways to break it. What are we even doing?

That's why I feel like if we move toward this control-native place, where we watch where it's going and how it's steering, and become partners of observability in the actions themselves, we can look at it from the inside rather than from the after-effect. So you've got an agent running all night to execute a task. Have we looked at what that looks like? Do we know all the places it went, what it tried, what it understood? Do we see the patterns of thinking, when it turns back, when it follows a branch, when it prunes the branch and tries another mechanism? We're never going to cognitively imagine, or even computationally imagine, all the infinite outputs. But the question becomes, can we see it going in certain directions? Take mechanistic interpretability as the most basic element: some output that's clearly part of a guardrail it shouldn't be crossing. It's not just the words that came out, it was the direction, which pieces were activated, what inside the model was being leveraged, so we understand when someone's going that way.

Using the math to detect and redirect

Alison Cossette: One thing I haven't done research on but am really curious about: we have algorithms that can take photos of humans and turn them into things that are not okay. I don't want to get into details or trigger anybody, but we take humans and put them in situations that are not okay. If we think in linear algebra, the difference between the original photo and the output is a given distance, a given vector. We know what that vector is, we know where it is in vector space. I think we completely under-leverage vector distances as a mechanism, because we don't necessarily need to look at the output and reclassify it. If you know that's the case, you can make it so that area is either blank space, can't be done, or otherwise understood. It doesn't have to be the exact guardrail, but how can we use the math to understand where something is going and possibly redirect or block it, taking a really deep mathematical approach to what's happening? I haven't done the research, but I think there's something there we can leverage.

Mo: There's a lot of credibility to that. Back when computer vision was a hot topic, there was a study, I wish I had it in front of me, where you could paint over a stop sign and a computer would read it as "go 45" or just go through, because it was reading pixels. It would read a couple of pixels, see this part was a different color than expected for a stop sign, and decide it must be a different sign. The machine was easily fooled, by its own expectations. Maybe this leads into what a good implementation of control-native AI looks like. Going back to your image scenario, say image A is an apple, and image Z is an apple still on the tree. How do you know a person is generating an apple on a tree versus something we don't want to talk about? What determines the steps between them? Is there a combination of prompt analysis, understanding the tokens being used for the request? I'm just a security guy making assumptions, but I'd assume a tree in a certain dimension of an image has a certain expected value for creation, taking X amount of tokens. So could you do predictive analysis and estimate that this is way more tokens than we need for this type of request, and then look deeper into the thought process? Some models have thinking tags or reasoning tags where the model thinks, which, going back to what you said, is a black box not exposed to observability. But that's where the money's made. So what would a control-native approach to this kind of generative content look like?

What a control-native approach could look like

Alison Cossette: This is where I wish I were on the research team at Anthropic, because I'd be happy spending my days trying to solve this. The math is very deep, and the number of people working on it is very small. There's a lot of work to be done to get it surfaced enough to be consumable. So the question for me is, on the enterprise side, the non-foundation side, what can we do? My old-school mathematical approach would be to look at anything I could extract, but most of the time these aren't open models. I'd do all my research on an open model where I can look at the weights and track things, but that's kind of silly, because that's not what everybody's using, so it's not that helpful. That's part of the challenge: we get back to the velocity moment, the velocity of, I'm going to use the hyperscaler model to get what I need and hope their approach makes sense, cross my fingers things don't go wrong. It's not a good answer. I want to have the answer for you, Mo, I just don't.

Mo: That's fine, and that's where we are. We're super curious about figuring out what we should be doing. I don't think our conversations have to end in solutions; they should take us steps toward finding them. On that, I know you were working with NIST on some guidelines. I'd love to hear about that.

Alison Cossette: Most of my work there was around understanding the mathematical pieces and how we frame risk, how we classify the different elements. You take the classic AI risk framework and figure out how to put generative AI onto it. What we came out with was a pretty decent matrix of understanding, but again, it's all about what people do with it. We put out frameworks that guide people in ethical choices, but most of it is about application, not the actual build. It's about understanding what decision is being made on an output and what's at risk. Where I think things get more interesting: I'd love to see a foundation model built just on children's books, or something very bounded, because what we know to be true is we can't have something come out of the model that it hasn't seen. That's actually not entirely true anymore, because models have gotten so sophisticated I have to alter that a little. But I'd love to see smaller models built on very closed cases that can hopefully avoid some of the negative noise that can come out of them.

Mo: Smaller models are good to look at for how you scale certain solutions and controls, because you can see the impacts more easily. At the same time, the "smaller" models are getting way bigger by the day; the things considered smaller are actually quite large. So going back to being in the room with the CTO and the Chief AI Officer, who want to get things done, what are the ways to help them feel comfortable about the risk while giving their teams some observability, transparency, and traceability? If you had to name three decisions they should make and three things to implement in their product or AI pipeline, what would those look like, and how would you apply control-native AI solutions?

Three things every organization should implement

Alison Cossette: The first thing is observability. It's shocking to me that observability of our systems is not table stakes.

Mo: So what does that look like?

Alison Cossette: Honestly, if it's me buying a product, and most people aren't building their own agents, then all of the OTel has to be built into the system and easily accessible. It's non-negotiable; you have to have the OpenTelemetry traces. Second, you need someone on staff who can understand and look at that observability at a system level, discern the impact, and communicate it to the relevant business. So you need internal observability with anything you build or buy, and someone internally who can leverage it at an intelligence level, not just a log level. Third, they need to understand what's at stake for the business in each area this touches, and be able to educate. Just like with our basic linear regression models, our job was always to provide a model that explains the risk, so the business unit could make a decision they feel confident about. We need that now, the same way we did before, but at this agentic level.

Traceability versus observability

Mo: Let's dig into traceability. What does it mean here? Is it figuring out how a decision is made, or understanding everything? It can get conflated with observability. What does traceability really look like, the actions an AI agent takes across the environment, or the things happening around the agent that you can look back on as a root cause?

Alison Cossette: I view traceability as the unit, the actual trace: the ability to follow a decision or outcome from an AI back through all the agents, traced to all the data sources, everything that went into it, back to the source of that data. That's the traceability of the decision and the branching that led to it. Observability, for me, is one layer up. We see all the traces and all the movement, and observability says, okay, these are all the actions being taken, but what does it mean? Even in traditional MLOps, drift isn't one output; one thing being a little different doesn't mean the whole model or environment has drifted. It's when we see a certain tipping point, a threshold, that we say, now this is drift.

Because of the nature of agents and their interactions, we can't look at it at log level, or even in two dimensions. We have to take time elements into account, because velocity of shift is going to be one of the big indicators. When I look at a system, what's anomalous isn't really the gold standard for me; what I want to look at is what's influential. What's the tool call or data source that's really influencing many things? What's a big shift? For example, you looked at how quickly someone was responding in your WhatsApp chat. It was the velocity of the change that mattered. Nothing the bot was saying was anomalous; it was his verbiage, so traditional anomaly detection wouldn't have flagged it. But the velocity of that change, and the level of influence, together are what we need to start understanding when we look at observability. One potential client said, you'll come in and look at our systems and determine if something is anomalous. I said, I don't see it that way, because if I come in today and there's already been an infiltration, I'd be assuming everything was okay on day one, and we don't know that. By looking at influence and impact, it allows us as humans to figure out which parts of the system to even look at. We have AI to help, but without that metric and understanding, we're never going to find it. That shift in understanding, what should we be thinking about from risk, is one of the fascinating open conversations right now.

Influence, and agents affecting each other

Mo: There's so much to digest, and you use this good word, influence. For as long as I've been in security, which isn't long on the scale of computing, I've been used to computers being call-and-response: I do one thing and get the exact output I expect. Even as pen testers, you know exactly how you'll get the response you want; you just keep asking the question different ways. Now we have AI, and we still have that vector with prompt injection. But there's this other factor of influence, especially in agentic systems, where agents are influencing each other. Take your space example: this agent does this thing, and it's basically peer pressure, getting pulled in one direction or another. When you think about environments with a handful of agents versus hundreds of thousands, like the one we're living in, how are all these things impacting each other? We're in interesting territory for evaluating risk at scale, especially after the acquisition of that agent product; now they're bringing it in, and we're about to see this technology move into everyday consumers' hands really fast. You won't need a Mac mini to run your AI companion anymore; you can run it from a big platform with the same access. It's moving fast, but it's an exciting time to tackle these problems.

Moving into the literal hands of agents: you were just at a robotics competition, and robotics has been something you've been involved with. What was that like, what are you doing with robots, and when are you getting your Neo?

Robotics: assistive tech and thoughtful building

Alison Cossette: I've been interested in robotics the last couple of years because I have an interest in sovereign systems, data privacy, and data protection. The Meta glasses were hard for me, as someone who feels strongly about data sovereignty, because you couldn't opt out of your data being used to train. Everything's moving very quickly, robotics included. We've had robot vacuums for years, but the humanoid robot is coming very quickly. As someone always looking to the future, I think, what does that mean, what do we need to prepare for, what infrastructure should we think about, how can we be thoughtful about what's coming?

I'd been talking to friends about inference at the edge, but I realized I should spend some time with hardware to actually have an opinion. So a friend, Andrea Turcu, who works at H2O, came out from France and said, I'm going to do this physical AI hackathon, come join me. The three of us, with a wonderful fellow named Sadir Dady who was at the next table, built a single claw arm, an SO-101. It's got a camera, and it opens a book, turns the pages, and reads aloud. It doesn't seem earth-shattering, but as a mother of a son with autism who has a reading disability and was served really well by audiobooks, it got me thinking about what's possible with this as an assistive device: reading, learning to read in a foreign language, extra support for special education, or parts of the world without as much access to technology or education.

The highest compliment I can give any builder, any founder, any human, or probably AI, is, is it thoughtful? Is what I built thoughtful about risk, about impact? Have we figured out what it can do for good or evil? So I started thinking, we've got to find a way to fund it, because nonprofits are great but it's a rough time. Then I was thinking about robotics data sets. What's interesting is that historically everything we built AI for was on a screen, text or video, everything two-dimensional. But we have to take into account force, weight, and physics. At the lablab.ai hackathon last weekend, people were using Isaac Sim and DeepMind, and a lot of ways for people to use VR or iPhones to create these data sets. But there's no physics in it, or if we're in simulation, we have all the physics but no real-world noise. So there's a gap in robotics training sets: real-world collection, which is historically very expensive. A constraint is always just a puzzle to solve. So the question became, we want to put this assistive technology out there, and we know there are these constraints, so what if we leverage these tools to really understand paper manipulation for robotics?

Our big lofty goal is to have a million SO-101s across the globe serving 240 million children with learning disabilities. It's a big goal, and I don't know how long it'll take. At the same time, we take just the telemetry data, the movement of the arm, the understanding of what it means to manipulate paper. My personal goal is to participate in building robots that can perform origami. All of this is to say, when the humanoid robot comes into your home and mine, I want it to be thoughtful, to have that delicacy and understanding of something very fine, so it can make good, intelligent movements to serve whatever you want it to do.

World models and teaching robots the "why"

Mo: That's a lot, and I mean paper is a lot; paper is a very delicate thing. I've been obsessed with world models, not for the world-building aspect, but from a security angle, for the physics piece. Not just "ball bumps into wall, if ball is rubber it bounces back," but what chemicals we can simulate, whether we can simulate chemical reactions against substances, recreate physical properties accurately, and what physical phenomena we can recreate in a fully simulated AI-driven environment. For example, what happens when paper touches water? A robot won't understand that; it'll just dip it in, and the paper is either heavy enough to handle it or soft enough that it tears apart. Paper is a lot. LLMs are not enough to get this information into a robot; we need to figure out how to up-level the comprehension or portability of a world model that fully understands objects and how they interact. How do we get that into robotics so a robot can turn a piece of paper without thinking about it in measurements of force, and instead understand how thick it is and what its fingers look like? Right now it feels binary: grab this with X amount of force, without understanding why. That "why" is so important.

It's the same reason my cat, who didn't grow up around other cats, will scratch and bite; it's actually playful, but without any indicators that it's hurting you, your cat will scratch you and you'll bleed, because it doesn't understand it's causing pain. Same with machines: they don't understand the effect of the force they output on whatever they're interacting with. We're teaching it with words and numbers, not with experience, which is where I think world models bridge the gap, at least in theory. I'm not a scientist or researcher, but I love reading that stuff.

Build versus govern: who's winning

Mo: You have an interesting background and jumped careers into governance, risk, and compliance. We're in a race with AI, but are we in the race you think we're in? I think we're in a race between our ability to build agentic AI and our ability to govern it. Out of those two, who's winning?

Alison Cossette: Building is way ahead. Partially for the reasons I said: we have a good idea of data governance. We know you can access this data, you can't; you can use this tool, you can't. We're really good at that. We are not good at understanding the mechanisms of what these tools are actually doing. And we can't govern what we don't understand.

Mo: That's everything we've been talking about. Builders have a huge advantage, and for folks in GRC, it's all catch-up. It's the exact same place red teamers have been for the entirety of the red teaming practice.

Alison Cossette: Governance debt is real.

Where to find Alison

Mo: That's another thing we could have talked about for an hour. Thank you so much for joining us. Where can people find you, and what's coming up?

Alison Cossette: I'm always working on a lot of things. The best way to find me is on LinkedIn; I'm always there and try to respond as much as I can. I've got an algorithm paper coming out in the next two to three weeks, some retrieval work, and I'm working on some permissioning ideas for agent-to-human understanding. LinkedIn is the best place to see what I'm up to.

Mo: Thanks so much again for stopping by; we finally did it.

Alison Cossette: Thank you so much for having me. As soon as I met you, there was an instant intellectual chemistry, so I'm super happy to be here.

Mo: Always happy to host curious minds, as we do here on Curiouser and Curiouser. If this episode helped cut through the noise, like or subscribe so you don't miss what's next. Thanks for spending time with us. Until next time, stay curious.

‍