AI Governance Needs a Dungeon Master

AI Governance Needs a Dungeon Master

Curiouser & Curiouser, Episode 6 with David Wendt

A lightly edited transcript. Disfluencies and false starts have been cleaned up for readability. The substance is unchanged.

David Wendt: As a data science and technologist community, we have a belief that if we can predict X better, that will lead to more sales, whatever it is. If we can provide a better layout for the warehouses, if we can tell them when they'll have to restock the front bins, that will lead to more sales, or reduced costs, or whatever. It's sometimes drawing the connection between the metrics and the dollars that gets hairy.

Mo: If AI has ever made you stop and think, "wait, what is happening?", you're not alone. I'm Mo, and I'm a security researcher asking the same questions. On Curiouser and Curiouser, we have open conversations with experts, researchers, and leaders working at the edge of this space, talking through how AI is taking shape, what's shifting, and how the people inside the work are thinking about it as it happens. So join us and listen in as the conversation takes shape.

Meet David Wendt

Mo: Hello, hello, and welcome back to Curiouser and Curiouser. I'm Mo, and today we have David Wendt, with a silent D, from Sherwin-Williams, where he's leading AI governance and innovation. But I'm more excited about his history as a Dungeons & Dragons Dungeon Master. I'll let him introduce himself, but I just had to throw that out there.

David Wendt: Thank you, Mo. I've been thinking about the math that is AI governance for about 40 years. That led to a PhD in statistics, which led to 30 years on the keyboard as a lone-wolf data scientist, a mad-scientist data scientist, and a director. But in the last two years, I've stepped entirely away from the keyboard to focus on generative AI and specifically its governance. It's a new, different way of thinking about things, and I find an incredible amount of enjoyment in it. And along the way, during those 30 years, I've also been a Dungeon Master. I've written for a number of gaming books. As I look back over both careers, I think there's a lot each can learn from the other.

What Dungeons & Dragons taught him about governance

Mo: I'd like to hear more about that. In education and psychology, there's a correlation between using stories and using tabletop RPGs to work through trauma. I don't think our field is quite that traumatic, but I know you're working on a D&D governance framework, which is really interesting. I play Call of Cthulhu, but Dungeon Master or Dungeon Keeper, let's be real: at the end of the day, we have final say over the rules. If we need to justify a decision in the moment for the players' experience, we do. When you look at a security program, or implementing policy, it feels like we have that same final say and flexibility, which is interesting in a field where we need deterministic proof. So I'd love to hear how you've applied D&D to your governance framework.

David Wendt: You've got the right starting point. I grew up in a household where we played games: card games, board games. Cards and dice and board games have randomness, but when you played Euchre or Pinochle with regular players, you pretty much knew how things would play out. There wasn't a lot of randomness, unless you were like me and thought, well, what will happen if I try this? Then I discovered Dungeons & Dragons, and it shifted the paradigm from largely deterministic to largely non-deterministic, leaving a lot of randomness and interpretation on the table.

That's the first parallel to generative AI and AI governance: we've had to shift the paradigm. We can put rules down and talk about process, but we're inherently in a non-deterministic world now, so those rules and processes have to be more like guidelines than strict rulings.

The second thing is communication. In 30 years as a Dungeon Master, periodically you get this idea: I'll have this great secret and surprise my players with it. More often than not, that falls flat on its face, because they're not prepared to be in that surprise with you. It's the same thinking here. You must communicate constantly across the organization, not so much that they form the rules with us, but so they understand the rules and feel we've been talking to them. Realizing that coming into this role has been huge.

The third one, and you touched on this, is that in role-playing games there are books after books, a pile as tall as I am, all kinds of rules and stats and numbers. But at this point in my game-master career, I don't look at those books at the table. I might check something afterward or beforehand, but those are the guidelines, the rules written down, so if we have a disagreement or aren't sure how to play something out, we have something to reference. That's what I talk about internally: we record our processes and rules so we can be intentional when we break them.

At my table there's something called the rule of cool: you as a player have just described something so cool that I'm going to give you advantage, because I don't want you to fail. I want that coolness to emerge in play. It's the same here. We're having conversations right now about Replit and who we release it to, and it's a balance: if we give it to too many people, it'll get crazy, but think of the amazing things they could create that we wouldn't think of ourselves.

Helping partners feel smart

Mo: For folks who don't know, Replit started as a smaller company focused on code development, how to deploy and manage software, and they've expanded to full application and site development and deployment with AI, which is the really cool part. You mentioned the rule of cool and surprise. To tie it together: professionally, and as a DM, my players would say I'm out to make sure they lose horribly, in the coolest, most horrific ending possible. But it's the opposite. It's to build an experience true to the world that also feels good for them. So I make sure they understand the constraints of the world and the limits of the magic system, so that when they go to do something, they understand the real implications. That's making them smarter and more aware. We're trying to do the same thing with AI governance, informing people of the uncomfortable reality and the real limitations. But I feel we've lost a piece of awareness that's made our users feel less informed about the decisions they make with AI, because it's so fast and easy to use. You can't always predict what these systems are doing; you're only there for the ending piece. So how has that translation process changed now that your partners are totally different?

David Wendt: They're coming from a different perspective, deterministic to non-deterministic. Part of the challenge, and it may be because they don't feel informed, is that a lot of our partners look to the media and to marketing and say, that's a cool thing, can we do that here? That's okay, but it causes challenges, because marketers tend to talk about things before they're done or ready. Commercials don't talk about all the data structure that needs to go underneath. A lot of the things that have always been true are just as true as before.

But there's a truism I try to teach the junior data scientists I work with. Our partners know we're smart, they know we know what we're doing. So it's not our job to go into a room or a video call and prove how smart we are. Our role is to help them feel smart. That's the really hard part, and that's where the translation comes in. We were talking about this earlier today: people want RAG for everything, and maybe sometimes they just want data extraction from a document. How do we shift the conversation from words they've heard us or others use, to the real use case? Why do you want to pull data out of this? Is it to put it into a database, or to have a conversation with your documents? Then let the professionals figure out a good solution.

That's tough. A lot of egos have to get out of the way, first the ego of the technologist, but also the ego of the senior leaders you're often talking to. Telling someone "you're using that wrong" is not always a career help; sometimes it's an inhibitor. So it's really about building relationships, communicating, and translating from the technical how-it-works to the more touchy-feely what-do-you-need-from-it.

Security as an inhibitor, and giving teams space

Mo: You said security being an inhibitor, slowing things down. A lot of my friends feel that same pressure, and I think it's because of how reactive security teams have been overall, especially GRC teams. I've never worked with a GRC professional who wasn't great at their job, but they always seem one step behind because of what they have to work with, always responding to regulation, incidents, or pressure. I wish there were a way to create space for GRC, or anyone in security, to make these decisions. Imagine a really well-funded security program. What could be built in that kind of environment to create that space?

David Wendt: The one thing I'm working on is a generative-AI-dedicated red team. Red teaming being, essentially, people you bring in to try to break your stuff before bad actors do. The research I've done says a single breach can cost six million dollars, and for a tenth of that you could put in a pretty solid team. The challenge you're facing isn't new. It strikes me back to Dungeons & Dragons and the origins of the game, which started as a war game, a simulation of historical battles. Those battles are now fought on the digital battlefield: somebody gets a new weapon, people respond by building a better defensive wall, which drives a better weapon, and it's happening faster and faster. Mix in Replit and things like it, and it happens even faster. So what you need is a combination, and this will be true for generative AI for a long time: an AI tool running constantly, looking for abnormalities, and a human team actively pushing the boundaries and trying to break things. It can't be one or the other; it has to be both.

You said something about giving our risk teams space, and I'd extend that to a lot of our technology teams. One of the early uses for generative AI was efficiency: write emails faster, write papers faster, now code faster. I had leaders say, but that's not savings for us, we don't want them going home early. For a while I thought that was close-minded, but I actually agree: they shouldn't go home early. Do those things in an accelerated way, and use the time you've saved to take the breather you need to think about what's next and how to be prepared, rather than running like a bull through multiple china shops, just trying to kick things out.

Mo: It's not a position I envy. As a builder, it feels like if we slow down at all, we're already behind. So it's almost like you're either the bull in the china shop or you're not even in the storefront.

David Wendt: A couple of responses. First, I think we all think we're further behind than we are. Every time I compare what we're doing to peers, we're right up in the front grouping, but it doesn't feel like it, and that's the bigger part, the feeling. Things are changing so fast, how could we ever be at the forefront? And I don't think we should be. Maybe some companies should, but companies in manufacturing, sales, so many companies don't need to crack the bleeding edge. They need to say, this function needs help, we need to distribute our products better, or speed up our time to failure in research. I had a conversation this week: we just need someone to act as the intern as we review our tax-law material. That's a great use for generative AI. Ironically, our legal department said they'd never use generative AI, and now they're asking for it, but that's another discussion.

We have to give ourselves grace, to say, I'm confident enough in my ability and my team to take the time to breathe and do it right. Or, as I heard early in my career, slow down to go fast. I thought that was ridiculous at the time, and now I think it's brilliant. I've shifted from bouncing from task to task to stopping and working with my generative AI to identify the next most valuable thing I can be working on. That can apply to a lot of folks: what is the next most important thing I can be doing?

Continuous red teaming

Mo: You touched on something important with the 24/7 intern. In the same frame, your red-teaming idea is one of the killer use cases for testing: continuous testing, continuous governance, 24/7 pen testing to an extent. Imagine a chatbot sitting in a test environment, and all day, every day, when your human red team is sleeping, it's consistently getting harassed by an AI on the other side, giving you a flywheel of attack and response. You look at the telemetry, see how things change, and make it a self-improvement mechanism. As you look at continuous security solutions, including red teaming, what does that look like for you, and how do you see implementing it into your security strategy?

David Wendt: That's a really powerful question. It's one where I know what will work for me, but I don't know if it works for anyone else. Despite 30 years hands-on as a statistician and data scientist, I'm an incredibly visual person. So what I'd look for, and it's how I described it over two years ago when I took this role, is a dashboard I can turn on Monday morning and see where things are yellow, or heaven forbid red, and immediately reach out to the people responsible to make sure they know. I can't control anything, but if I can see, we're getting a lot of weird things around our company chatbot, I can reach out to that team and ask, did we change something, or are we facing a new set of attacks? That's powerful. Back to communication and relationships.

The crazy part is I don't want that just for me. I want it for my team, and all the way up. I want my CEO to have the confidence to go in and look at any point, whether a specific project or in general, and know what's going on. I say that knowing no executive will ever do that more than once, but I want it there, so they have the confidence that if for some reason I got hit by a bus, they could still get the same information.

A quick note from the show: For those of you heading to RSA this March, you know how chaotic it can be, so many vendors, so many booths. With this year's theme focused on community, we decided to slow things down and give the community a space to take a break and maybe join us for a cup of tea or two. Stop by booth S2051 and you'll see what I mean. See you there.

Mo: The good thing is the build-versus-buy argument has only gotten better. When you say you know it works for you but maybe not for someone else, we're in an era where more teams are building than buying, and you can build something well-tailored to your security program. Now let's take it up a notch. You want to see green, yellow, red, really easy signals. But with non-determinism, testing is complicated, debugging is difficult, evaluation is hard. What if you can't reproduce prior model behavior, or can't find the results you got five minutes ago? How do you know your AI systems are actually getting better or worse over time? What does improvement even mean now? So what does green, yellow, red even look like?

What green, yellow, red means under non-determinism

David Wendt: You need to take it out of the moment, because generative AI models are probability models, full stop. Really fancy ones, but in the end, probability models. If you take that into account, then for any given property, accuracy, latency, whatever, if I see consistent behavior that's positive, that's easy, that's green. Then you have a time dimension. You might see one bad spike, but if it's only bad once, it's probably not a red; it might be a yellow depending on the importance of the system. If I see spikes happening regularly, even moderate ones, that pushes me into yellow. And if I start to see consistent high-level spikes, that becomes red. So you're relying on the laws of probability: if it's a fluke, you won't see a repetition right away.

The other part is, how conservative do you want to be? I've spent most of my career in very conservative companies, so any time there's a spike, any time the chatbot gives a slightly wrong answer, we want to be aware of it; that's a yellow, if not a red. Other companies aren't as conservative and can run it with, we've got some spikes but overall it's not too bad, it's getting us what we want, it's predicting who might buy from us. If it hits 80% of the time, that's better than cold calling, so that's okay.

None of these metrics exist in a vacuum, and that's been true forever. People ask, is 57% good? Well, is it better than you used to do? If you used to do 50%, that's good. If you used to do 60%, not so good. Everything has to be dealt with in context, especially with non-determinism, because we're not in raw chaos, as much as some people want to say. Everything is still predictable; it's just a lot more variable than people are used to.

Maturity frameworks in a multifaceted org

Mo: Even if you look at standards like SLSA for open source, and MITRE, especially the MITRE AI maturity model, it's all based in this. They're maturity frameworks where you set where you are and see the steps to the next level. Moving from level one to level two doesn't mean you're bad at level one; it just means this is where your organization is today, the bare minimum, and if that's sufficient today, here's how you move forward. Different organizations have different priorities. Yours has a lot: product development, customer engagement, supply chain, sustainability. These frameworks weren't meant to cover a multifaceted organization that does heavy manufacturing and supply chain but also has a strong technology and customer-experience piece. Are you applying it separately for different parts, or trying to find a one-size-fits-all good-enough and then figuring out the nuances?

David Wendt: It's very easy at the top of the house: we sell paint. The more dollars we bring in and the more gallons we sell, the happier our CEO and the street are. At the most detailed level it's also pretty easy. I sat down with our data scientists and said, I need stuff in these seven categories, start logging it, and I got less grumbling than expected. Someone might say, this accuracy measure doesn't work for what I do, can I use a different one? Yes, there are categories of things. It's the middle ground that's really difficult.

As a data science community, we have a belief that if we can predict X better, that will lead to more sales, or a better warehouse layout, or knowing when to restock the front bins. It's sometimes drawing the line between the metrics and the dollars that gets hairy. It's also hairy because, was it the model that made it better, or the people who reacted to the model, or other features that changed at the time? Everyone wants a piece of that improvement, so that's a real trail.

I haven't quite answered your question, but I'm getting there. The realization I've had over the last couple of months is this: we should not be measuring AI, full stop. We should be measuring the use cases we're changing that happen to leverage AI. So if our big focus is raw-material pricing, that's where we think we'll make our bang for the buck this year, then I don't care what you do, traditional AI, generative AI, a hybrid, throwing darts at the board. I'm looking for more effective pricing, lower prices on our raw materials, which leads to better margin down the line. Nothing in there is AI-specific. That's the shift I'm trying to bring for that middle section: I don't care what model you're using, I care whether you're getting the results you're after. At that point the client, the customer, the partner, can own what's next. We provided a model, we got raw-material prices down 10%, and if you're a good partner, they take you with them. You don't need to say, we own this half of that. That's where I'm pushing my teammates and the younger data scientists: not to think about what we do, but to provide clients what they need and make them look good.

Beyond data literacy: teaching people to think

Mo: What you're describing is traditionally a data literacy problem, because now we're saying we shouldn't judge these models just to say we're using AI. Instead, how much of an outcome can we attribute to the changes we made with AI? Did it make us 10% better, 20%? And there's a cost to AI, which is a whole other episode, how much it costs to run these models 24/7, whether we own the infrastructure. But I wonder if data literacy is even the right metaphor anymore. When I think of literacy, I think of a book, but books don't change words every time you look at the same page. Are we trying to teach people to be literate about something fundamentally changing all the time, where what you knew on day one is different on day two, and the metrics may describe something entirely different?

David Wendt: Funny thing, I have in the back of my mind to use technology to write a book where the words change on the page from read to read. So everything's possible, but your point is well taken. We're fortunate to have an AI literacy team dedicated to this space, and as a former educator and professor, I constantly urge them to think outside traditional teaching methods. This isn't Excel; it isn't "if you type equals-if-parenthesis-this, you get the same answer every time." We're in a space that's constantly changing, where the answer you get will differ from the answer I get with the same input into the same chat engine.

So the things I urge our literacy folks, and I've had the chance to talk to high school students and faculty, is to start thinking about critical thinking, problem solving, puzzle solving, these broader, more cerebral, creativity-driven skills. Because if I'm thinking about the problem holistically, I'll bring in concerns about cost, about this metric. It's just another reason not to worry about measuring the AI, but to measure the thing you're trying to do with it.

That's tough. We're not set up in this country to teach people how to think critically and solve puzzles, and it's going to be a challenge. The good news is that, at least in the greater Cincinnati and Cleveland area, we've got great teachers and fantastic students learning fast and adapting quickly, so I have high hopes. I think people my generation, and the generation or two following, are probably going to struggle the most. Not unlike when robotics came to the auto industry: some jobs were lost, but mostly jobs were transformed. Instead of, I can pull this and do that, you now had to think about how to make the robotic piece do what you needed so the next person was ready for their piece.

To bring it full circle, how do we teach people about this stuff? I think we teach with story, with allegory, with what-ifs, role-playing. I'm noodling on it, it's not quite done, but I'm hoping by the end of the summer to have a role-playing experience I can take to conferences, where half a dozen people sit around a table and play through a governance issue, hopefully a little more fun than the real thing.

The generational problem

Mo: I want to push you deeper. You've spent most of your career helping people make sense of data and technology, and you said the people who'll struggle most aren't the younger generation. I think the younger generation is struggling right now because we've created a world where they don't have opportunity. We've decided entry-level jobs will be replaced with AI, or a couple of agents. As much as we'd like to say we're not replacing humans, we are taking away the bundle of tasks that make up an intern's or IC1's role. Review these basic pull requests? We can do that with AI now, so we're not going to pay an intern another 10 hours to go over pull requests, tickets, and bug tests. We're slowly removing the opportunities for them to learn. Some organizations have caught on and said, we'll eventually need people because of turnover, so we need to keep some of these roles. But on a larger scale we're not preparing for that. So how do we build resilience for the older generation, who I think will need it most? I think the younger generation will just take the ball and build their own thing. Carta released a study last year showing a massive increase in single-founder companies with no funding, which I'll largely attribute to AI. So we need to create resilience in older generations if we're not going to give younger generations the opportunity.

David Wendt: Yes. I apologize, I can't remember the name of the book, but I read one, and heard a speaker, who spoke to this. He looked at dozens of professions, and every time automation came in, the next cadre were not as good. Every field except one. The exception was bomb disposal, because before automation the senior would be out there with the bomb, sure they'd have a mic on, but more focused on not blowing themselves up than explaining. The juniors, back in the truck a good distance away, couldn't really see and could only rely on that. Once you put robotics on it, the junior is driving the joysticks and the robot, and the senior is over their shoulder guiding and explaining. My parents told me video games wouldn't be good for anything; I both agree and disagree, and I also don't want to do bomb disposal for a living.

So there's a real concern. I've worked with some ethics groups at universities, and I think this is an ethical piece we're missing. Not, could people cheat, but what are we doing for the next generation? You're spot on. I do all sorts of things every day now that five or ten years ago I'd have passed down to my next level to figure out. For the junior folks, I think we're ethically bound as organizations and caretakers of our world to keep bringing them in and giving them the opportunity to learn on real things. Maybe that's not quite as cost-effective as going pure AI, but we're going to need it.

For folks my age and older, even a bit younger, I share your concern. I came from a generation where you were a jock or you were unpopular, full stop. We turned that around and said, hey, we tech guys kind of know what we're doing, and we've been fairly successful. But there are still a lot of socially driven roles: sales, marketing. A salesperson, throughout my career, even before generative AI, doesn't want to give up their secret sauce. I'd say, but I know you know these people's favorite sports teams; if you give those to us, marketing can send customized materials based on their teams. They don't want to give it up. This is worse: now I'm going to come in and say, here's your list of AI-approved people to call. They don't want that; they know who to call, and I believe they know who to call.

But the junior folks don't, and they won't learn the same way seniors did. I did an analysis 10 or 15 years ago where you could see generationally how business changed. The oldest salespeople got most of their dollars from face-to-face visits. The next generation down, phone calls. The next, electronic communication. What was fascinating was that when I split it by gender, it was offset by a generation: women of the second-oldest generation behaved like men of the oldest generation, echoing what they saw their mentors doing, and it carried down that way. There are reasons that's not great, but it illustrates that the younger generations will adapt. For the older ones, we just have to frame things, communicate, and build relationships.

At our sales meeting a couple of weeks ago, we announced our internal, sales-focused, product-driven chatbot, targeted toward our younger folks in the stores, where we have a turnover problem. I believe part of it is people not feeling smart: someone asks what stain to put on their deck, they don't know, and they don't want to ask their boss every time. This should help with that, improve retention, and improve profitability down the line. In a company like mine, where the CEO literally started in the stores, and a lot of the C-suite did too, that's important. So I have no great answer other than build the relationships, let them know you're not here to take their jobs, you're here to help them do what they do better and get the stuff they don't do well out of the way.

Disappearing lines between specialties

Mo: That was really well put. A lot of the conversations I've had around this have been skeptical and pessimistic, and to be frank, realistic. But there's another side: what if we focused on enablement? We also haven't talked about what this is doing to senior and principal engineers. It's hard, because now you've got people up top with less support but more responsibility, and if they burn out, who does that get passed to? You said something interesting about one generation echoing another. I've switched jobs a lot and been fortunate to be in organizations that said, Mo, go run with it, giving me flexibility to learn and build. Recently I worked with someone in sales, and I know nothing about sales; my friends would say I could probably sell them on my many hobbies, but I've never made a dollar from any sticker I've tried to sell.

The interesting thing working with the sales team is that they're really forward-looking. Doing competitive analysis, something I had to do recently, the question was how to enable people who don't have all the insight into a particular field, security engineering, to use it at a higher level. It's part me writing something down and telling them, and part using AI to help generate a report in sales language, which I don't know. It's been a great glue between fields. There's another layer where AI introduces inaccuracies: when I write something, I review it, and when I translate it into sales language, they review it, and there's a back and forth where we get better and better. It's one way where, now that jobs can't echo each other as much, we ask how we inform each other and get to know each other. Because in this next generation, the lines between jobs and specialties are really disappearing. Folks who were laser-focused in a single area are now performing in others, where we originally wanted deep specialists. Turns out the jack of all trades has started to shine, especially when AI enables them to level up mastery across areas. So the battlefield has changed, in seniority, specialization, literacy, education, everything. I don't think everything AI is bad or good, but it's been dynamic.

The one place I think AI will never take over, and I know it's already been done, is creating a really good story for a DM. It goes back to the sales analogy: you know the favorite teams of the people you sell to, and you know your players best, exactly what makes them tick. One day it'll be easier to generate scenarios like that, but I don't think we'll exactly get there.

It makes me think of Death of a Salesman, the salesman unable to adapt to new techniques, finding the customers and people were no longer the same, and even though he was the same, he was no longer suitable for the world because of how fast it was changing. I don't think we're seeing the death of literal salesmen, but a Death of a Salesman theme across industries, folks locked in traditional values being forced out of their comfort zones. I don't think there's ever been a time where someone stepped up to the plate, embraced it, and was still left behind. So I don't think we'll see many people left behind; I think we'll see people displaced for a while as they understand what comfort now looks like.

Closing thoughts

Mo: I know we're just about at time. Do you have anything you'd encourage folks to do, or final thoughts on this topic?

David Wendt: I think we have a responsibility as human beings to embrace change and accept humility. If you can do those two things, you can adapt to almost anything. That's part of why I'm in this role now. Going from hands-on-keyboard data scientist to governance might seem insane, but I was open to the change, I was looking for it, and I didn't go in like I knew everything. I was looking for partners, for cohesion and cooperation. When I look back on my career, those two things are where my successes have come from, rather than blindly charging forward the way I've always done it.

Where to find David

Mo: Where can people find you? Anything coming up, any new books or open game sessions?

David Wendt: The easiest place is LinkedIn. I try to do at least one post a week on Mondays, just my ramblings around governance, culture, and change. I believe you have my speaker-page link to share, since it's not a quick one to rattle off. Not a whole lot coming up right now; I'm waiting for the freeze to come out of Cleveland. I think the thing to watch for is to find me at a conference, talk to me about the game I'm designing, and maybe we can throw some dice.

Mo: Thanks so much, and see you all next time.

‍