Threat Modeling In The Age Of Artificial Intelligence With Laura Bell Main

“Laura Bell Main: AI, what could it do for us, right? Well, theoretically, if we can communicate what our systems are, what their inputs are, what they look like, if we can logically and consistently communicate that to a model, then theoretically, and in practice in some cases, now in a limited capacity, they can do the inference for us.”

[INTRODUCTION]

[0:00:27] ANNOUNCER: You are listening to The Secure Developer, where we speak to leaders and experts about DevSecOps, dev and sec collaboration, cloud security, and much more. The podcast is part of the DevSecCon community, found on devseccon.com, where you can find incredible dev and security resources and discuss them with other smart and kind community members.

This podcast is sponsored by Snyk. Snyk's developer security platform helps developers build secure applications without slowing down, fixing vulnerabilities in code, open-source containers, and infrastructure as code. To learn more, visit snyk.io/tsd.

[INTERVIEW]

[0:01:14] Simon Maple: Hello and welcome to another episode of The Secure Developer. In this episode, joining me today is Laura Bell Main, CEO at SafeStack. Welcome to the podcast, Laura.

[0:01:26] Laura Bell Main: Thank you so much for having me. It's a delight to be here.

[0:01:28] Simon Maple: Wonderful. Pleasure to have you. Now, in today's podcast, we're going to have some fun talking a little bit about threat modelling, and some of the advances that we can take with AI in threat modelling. First of all, before we jump in, Laura, I'd love to hear a little bit about yourself, your career, your path into security, and also, tell us a little bit about what SafeStack do.

[0:01:48] Laura Bell Main: Awesome. Well, I'd love to say that I studied the right qualifications, and then I went to the right schools. I didn't. At age 16, I became a COBOL developer for the tax system in the UK, and that was more to do with that I needed a job than it was anything else. I'd never coded before in my life. I thought at that point, I was going to be a lawyer or something, but life happens.

Since then, I've been on a really interesting and wobbly journey. I put myself through my qualifications finally. I went to university when I was a bit older than most and did AI and robotics in Aberystwyth in Wales, and then I worked for places like CERN doing real-time radiation monitoring software, the UK government doing counter-terrorism. Then I was in this weird rut of my career where I was doing enterprise Java, which is a sad time for most of us.

I found that I'm incredibly curious. I'm the person who, if you give me something, I want to go, “Mm, there's three buttons, and I have five fingers, and how many combinations can I push at once and what happens?” I'm quite playful like that. Now, it turns out, as a software developer, that's not very helpful. But my team leader at the time, he was like, “Laura, we really like you and we don't want you to leave, but seriously, please stop. Have you considered security? Because that's what they do all the time.” He introduced me to the security team and I became a penetration tester with time. I [inaudible 0:03:08].

Then I've come full circle now. I live in this weird hybrid space between security, where I specialise in application security in particular, and software development. Yeah, I'm this weird hybrid, and I love it. I'm genuinely excited about the technologies we're building in the world right now.

SafeStack is my ugly baby. If you as an engineer call all your tools and things you build ugly, I definitely do. It's lovely to me anyway. We're on a bit of a mission. I've spent the last 15 years helping teams all around the world go really, really fast and do security as they go. Some of that's tools and some of it is culture and some of it is communication. It's a bit of a wide-ranging thing. Now in COVID, when we all decided to stay home and cook bread for a while, we decided instead to turn what we'd done in person for a long time into a product.

We now work with about 2,000 organisations around the world in 89 countries, ranging from teeny, tiny two-person startups, all the way up to big banks and airlines. Our job is to try and get everyone in software, not just the devs, but the testers, QA product, you name them. If they're somewhere near software, we want to play with them and we want them to do one hour of security every sprint. That's our mission in the world.

We reckon, if there's 30 million software developers in the world right now, that if we all did one hour, the impact overall on the amazing software we're building would be phenomenal. That's what we do. We're an education platform. We have templates, resources, labs, all sorts of great things you can bring into your work flows and make security a little bit easier.

[0:04:47] Simon Maple: That's an amazing philosophy, actually. I think it is from an educational point of view, ensuring teams do actually spend that time and dedicate that time to security. I think not just as a one off, but baking into that sprint. I love that idea of this is something which needs to be almost prescribed into every sprint, as well as various other technical debt styles, things that need to be done with resilience, and other things into that sprint.

[0:05:13] Laura Bell Main: Yeah. I think if we look at it right now, a lot of the way that we've historically talked about security is like this giant monster of a thing. If you look at all the things as an engineer you already balance with performance, scaling, usability, observability, accessibility, all of those things. You do every single day. You do it in minute ways, just as part of what you're doing. We're trying to, instead of security being this huge thing that feels overwhelming, that you don't know where to begin with, that we just bring it in like all those other illities. It's just part of writing quality software. We make that easy. We make it small and achievable, then the net results as an ecosystem are massive.

[0:05:52] Simon Maple: It's interesting actually, because even on the podcast here, we do talk about almost like the destination, ways to get to that destination, but sometimes it's equally, when we think about the implementation, it's about little steps. It’s about not taking a leap to a new destination, but how can we make small changes today to actually get to that destination? What's the next action we need to take? I think that's very fulfilling to see those small changes happen. It's like the 1% change in various habit changing techniques. Yeah. Very, very interesting.

Let's jump into the topic. Threat modelling. Something that we've learned about for a long time. We've heard about for a long time. Why don't you give us a foundation about threat modelling, where we are today, and perhaps even some of the challenges that we, as an industry experience with threat modelling today.

[0:06:39] Laura Bell Main: Yeah. I'm going to ask you a question first. Have you ever robbed a bank?

[0:06:44] Simon Maple: Gosh, you're asking me this live on the podcast.

[0:06:45] Laura Bell Main: I know, right? It’s a trap.

[0:06:46] Simon Maple: I’ll say no. I should say no, right? Yeah. No. No, I haven't.

[0:06:50] Laura Bell Main: No. Well, most of us haven't, right? But weirdly, there's a whole set of skills you as a person have that you have forgotten that you have. When you were little, when you were a kid, you would play imagination games. You would plan a bank robbery. You would have Godzilla and Batman fighting out to protect the dinosaurs from the evil invaders, whatever it was that was your particular jam. One of the things that you did is that you looked at systems and you looked at environments and you went, “All right, here's a way that the bad person could do the bad thing.” You did this naturally.

Then school happened and life happened and you grew up. Now when we build systems, because we're engineers, we're always focused on the end result, the outcome. But we don't take the time to step back and look at those systems and go, “Cool. What are the other ways they could be used? What are the ways they could be misused?” This is where the bank robbery comes in.

If you were the person in charge of a physical brick-and-mortar bank, and imagine that bank had something valuable in the vault, then there are literally hundreds of ways to plan a bank robbery. Our movies tell us about one, two, or three. Somebody comes in with a weapon, somebody tunnels in. But actually, there's so many. Every system we build is like this. Planning a bank robbery is essentially a threat model. Thinking through a system that has controls, like guards and cameras and brick walls, and thinking about it in a way that it could be used to get to an objective.

In software, you would do this with your application. Not just the one little bit of code you're writing today, but the entire architecture of what you're building. The processes through it, the data that goes through it, the inputs, the outputs, the people who interact with it, and the processes. When we look at these, not from an engineer's perspective of, “How do I make this work?” But you take a step backwards and you put yourself in the shoes of an attacker. Now, an attacker is objective-focused, rather than they're not there to go, “Hey, today I want to do SQL injection.” They're here, because today, “I really want more money, or today I want to get political influence, or today, I want to read the CEO's emails.”

We put ourselves in that objective focus seat. We look at why they would want to act, and then we look at our system with a new set of eyes and we go, “Right. If I was in this position, if I was planning this attack to get to my objective, what are the ways I could do it?” By looking at it this way, firstly, you see your system in a completely different way, because you're seeing it not from the intended output for which it was built, but for the ways that somebody else can achieve something that you hadn't had in mind at all to begin with.

Secondly, it’s a very collaborative process. It can show you across a group of people how complex and how diverse the pathway through your systems are. Threat modelling in itself is a whole bunch of tools and techniques and there's free ones and paid ones and everything in between, but the consistent part of all of them is a systematic way to look at what you have built, or what you are about to build from an objective point of view, such that you can look at the pathway through your system to get to those objectives and therefore, identify risks that might be in your design, in your architecture, or in your implementation.

[0:10:11] Simon Maple: I think from a developer point of view, it's not always – I agree with exactly the way you said it whereby, sometimes, we're almost too internal as developers, we look into our system and try and identify what is the objective of my functionality, my feature? I think too much about the golden path, think about that, “Okay, I'm trying to get this to do X, Y, and Z. Once it does that, I now focus on the next thing, where I think about it in that way.” The way you describe that is almost counterintuitive to a developer, because they almost have to stand from the outside in and think about it differently, rather than the way it's supposed to work, how can I almost break this? Well, how can I think about it from the objective of what I as an attacker might want to get, irrespective of what it's supposed to do, or how it does it.

Are developers – when we talk about traditional threat modelling – are developers the best people to do this? Who do you feel needs to be in the room for this kind of exercise?

[0:11:05] Laura Bell Main: I love making these a team sport. Absolutely, we want the engineering team there, because they know how the thing is strung together. Your QA folk, for example. Now QA, no matter whether you are manual testing, or automated, we don't judge, the way you think as a tester is actually closer to that threat modelling naturally than any other part of your team. They are the type of people that go, “Whoo, what if?” If you haven't got those people on your team, you need to find a what if person.

In fact, there are companies in the US in particular who there was a little song and dance once upon time, because they were hiring out-of-work scriptwriters to come into threat modelling sessions with them, just to be that chaos person who would go, “Ooh, yeah. But what if?” You need that voice there.

The other folks you might want to consider, which are completely outside of your software team, are your customer experience, customer success people, because they know how people are using your system because they help them every single day. If you're in a bank, to go back to our example, you'd always want to tell her, you'd want somebody who'd been working physically in that bank for a number of years, because they're going to tell you that, “Oh, yeah. Well, the controls are good, but we know that if you put this chair there, you can't see anything on the camera,” or “We know that every Tuesday, there's a shop over the road that has a buy one, get one free offer on donuts. We will go out for donuts on a Tuesday and we close up the bank for 15 minutes, but don't tell anyone.”

You need the people who are operationally involved in the systems to give you that perspective of not just how you think they are used, but how they are used. The more depth and the more diversity you get in those opinions and those perspectives, the richer your threat model is going to be.

[0:12:48] Simon Maple: Yeah, I think it is a team sport. There is no one individual that knows everything about whether it's the system, the usage, and I think adding someone from the customer – not the customer side, but from the customer success, or someone who is very close to the implementation about how people use it is really, really important.

In terms of who owns the process, though, who should get all those people into the room? Who's responsible to do that? Because the value of threat modelling really is doing it nice and early as things are being designed, so that you can almost make the right decisions about how you architect and design things that later on in the implementation, where you get vulnerabilities potentially being implemented that could be completely avoided by doing things a particular way early on. Who’s the right person to say, “Hey, we should be threat modelling this, or I need to get the right people into a room to do this,” who would you say?

[0:13:35] Laura Bell Main: This is where it all falls apart. This is where we've got a bit of a problem. There's a few schools of thought on this. Those people who are listening, watching at home, you might be able to reflect on this as to what your organisation looks like. Now, we did a survey of about 2,000 organisations. We looked at how many engineers they had, how many security people they had, whether they were joined together, or whether they were siloed, and what percentage of the overall workforce this represented.

Now, the reason we did this is we had some hypothesis. We were like, hmm, if you're a tech company, you're totally definitely going to have a security person. Spoilers, they didn't. That security person is going to be really, really embedded with software teams. The security folk can say, “Hey, we're going to do some threat modelling and they can help champion that, take some of that effort off the dev team, because our dev teams are super, super busy already.” How many of us can actually look at each sprint and go, “Oh, yeah. I've got a few extra hours to spare. That'll be fun.” We're not. We've got way more work than we've ever done before. The pressures on all of our companies and our teams are really high at the moment.

We had this hypothesis that there would be a team. You could ask your security folk. If you're in that environment, you've got a security team and app sec team and they can help you and they can facilitate. Fantastic. Be very grateful for the situation you're in, because it is unusual.

Most companies, if they have a security team, it is quite separate from development. Many of the security folk come from different backgrounds to engineers, and not necessarily people who have built software before. They might be networking folk, or risk and compliance. Now, that's not a problem. It just means that they don't have the understanding needed to really own this process and to work it through. They also may not have the relationships with the dev team to really speak the same language. That's what causes a lot of friction over time. It’s what DevSecOps was trying to fix and didn't.

What we can do here is we need to be really honest with the environment you're in. If you're in an environment where you don't have an app sec or an engineering-savvy security team, firstly, go make friends with whatever you have. You're going to need them in some respect. Secondly, it means it's got to fall into the dev team. Now, what we see commonly happen here is the Batman problem. Now, the Batman problem is one person in your dev team will be super enthusiastic about security. If you're thinking about your team now, you can probably think of that person.

Now, what happens is that person just keeps doing things on their own. They're the person that kicks off the threat models. They're the person who makes sure the tools run. They're the person who triages things. Batman has a real scale problem. If you're in Gotham City and you're Batman and you've got five murders going on at the same time, how do you choose which one you're going to do? Well, you get a backlog and you put them all in Jira and eventually, you choose which murder you’re going to handle and then next sprint, you handle the next one, right? But that doesn't work. That superhero mentality just means you end up with a backlog and eventually, Batman gets really, really tired and just retires. Then you've got nobody. Just this growing backlog of bad things happening.

We need to make sure that it's not just one person in dev. It's actually something that we're sharing amongst our teams. We have to find a way to enable that that's respectful of the constraints of dev teams at the moment. It needs to be something that is easy to do. There needs to be a template. There needs to be a model that we know how to do this. This is how it rolls and we make it super friction-free. We need people on hand who can facilitate. You need somebody with energy in that room who's going to go, “Cool, let's do this.” Doesn't need to be one person across the whole team. It needs to be a group of people who actually care and can come and do this with you.

Then we need to make sure every single team is empowered and also, held accountable for doing this. It's not saying, “Hey, you can do this.” It's saying, “You can do this. We have enabled you to do this and also, we're looking and we notice that you're doing this and that's a good thing. Oh, we're noticing you're not doing this.” You need all of those parts in play. Sadly, it's going to be in the dev team. If you're feeling very tired and stressed out already, take a break, have a cup of tea. You don't need to do it now. But it is something we need to find a way into our world.

[0:17:38] Simon Maple: I love the idea, actually, there of Batman having a Jira backlog. I think that's a – if there's one takeaway from this at all, it's remember, even Batman uses Jira and has a Jira –

[0:17:47] Laura Bell Main: Absolutely. It makes him very sad. That's why he's grumpy.

[0:17:50] Simon Maple: Yeah. Let's move the conversation more into AI now and talk about, obviously, a lot of those problems with even just mentioning AI, we can start to piece together how an amount of automation and AI can play a role in this. Where do you see the future of threat modelling with the addition of AI? I guess, first of all, how do you see AI playing a part into the process of threat modelling?

[0:18:15] Laura Bell Main: There's two sides to this, and I think we need to talk about them both. There's AI and the effect it has on the threat models that we build because how we threat model AI is different to how we would do a linear system. I'll come back to that in just a second. Then secondly, how can we use AI to help with this process? I know there's folks at home who are going, “Oh, my goodness. The snake is eating its tail.” It is going to be a bit like that, so bear with me.

Let's start with the first half of that problem. When you threat model a current system, a non-AI system, it is linear. If I have an algorithm that I've implemented and it's got steps one, two and three, and if I always put an apple in steps one, two, and three apply and out comes an orange, then I know that I can map all of the pathways through that code base, and I can understand and anticipate all of the outcomes of every permutation of the logic I've implemented.

Now, that takes a very long time, but it is achievable. You can map out your systems in advance and you can be predictive of, okay, here's where the bad things can happen and here's how I would know. The problem with AI systems and the increasing use of them, and I'm pro team AI, so I'm not saying don't use AI. I'm just saying, great power, great responsibility, etc., etc., is you've got this box in the middle. Now, that box in the middle is some form of model, and I'm oversimplifying a massive space of forgive me at home, people. I know you have feelings and opinions, but let's call it the big black box in the middle of AI.

Now, either you build that, so you have some understanding of how it works and you trained it, or you're using a third-party model that somebody else has done that work for. Whichever it is, by nature of what they are, they are not finite pathways. They are evolving logic models that can change with time, that can change with what they're trained on, with what data you put into them. As a result, we can't for any more say, “Hey, if I put an apple in and I put it into my big black box, then an orange is always going to come out the other end.” Because you never know that if it's been trained in a certain way, or something influences, suddenly, an elephant's come out.

When you are threat modelling, it's all very well and good for us to say, map out your whole architecture, look for the malicious pathways through it. But if one component of that has a degree of uncertainty, or evolution, then it becomes very difficult for us to be as sure about those predictions, about those threats. We need to be really mindful of that. This gets incredibly complex if it's not your model.

Now, I have no problems at all with people having third-party models and you integrate with them, I get it. I totally do. If you are doing that, so you're integrating with a third-party system, whether it's OpenAI, it’s models themselves, you're using those tools, or you've found an alternative that you're plugging into, you've got to be very, very careful that you're understanding and have an explainability, if you will, over what is happening in that component. Otherwise, threat modelling is going to be very, very difficult to get the value out of. That is the big difference and change that AI is making to the models themselves.

On the other hand, then there's AI, what could it do for us, right? Well, theoretically, if we can communicate what our systems are, what their inputs are, what they look like, if we can logically and consistently communicate that to a model, then theoretically, and in practice in some cases, now in a limited capacity, they can do the inference for us, because they can work the pathways, they can look at those inputs and outputs.

Now, they're not as creative at this stage as humans. There is a degree of this that actually, the creativity we have as a species, that ability to adapt and improvise very quickly from unknown or uncertain data is a real big advantage in security, particularly in threat modelling, but they can do the low-level stuff for us. Now, even before AI was a thing, Microsoft released tools, like Microsoft STRIDE, which is a threat modelling framework. You can go and grab, it’s open source. It stands for spoofing, tampering, repudiation, information disclosure, denial of service, and escalation of privilege. There is a test at the end. Answers on the – you’ll remember those.

Now, they built tools even 15 years ago that you could map out and draw your architecture, and it would come up with a low-level threat model for you. It's all the low-hanging fruit type things. AI systems are now taking this as the next step up. You can communicate your architecture, communicate your data model, but not the data in it. You wouldn't want to do that. It can then start looking for those low-hanging fruit, those common, this might be used in this way. That's the state of play at the moment. Now, there's a lot there. Simon, where would you like to unpick next? Because otherwise, I'll just keep talking, and your audience will be like, “She never stops.”

[0:23:06] Simon Maple: Let's start with that first half. I think, we've got two pieces there that are really, really interesting. Now, that first half, where we're trying to perform threat modelling on your application and feature of your application, that pulls in, maybe it's a new AI model, and potentially even, like you say, models that you don't own, you haven't trained, but potentially, someone else's model you pull in. I'd love to talk about some of the biggest threats that we have there, because I guess, the threats of the model itself, and that could be anything from, we can talk about things like prompt injection that we know is already a big threat, and it's been well documented about the path that attackers can take through the models with using things, like prompt injection.

Other things, for example, about the way these types of models are implemented, like you say, whether they're next to our data, whether they're using our data and things like that, what would you say, I guess, are some of the newer things that we have to think about that we can pick up from a threat modelling point of view? What are the most dangerous aspects of using an LLM in your application from a threat modelling point?

[0:24:10] Laura Bell Main: I can actually point you at something, you can go home and read as well. The nice bit about this section is it's not just going to be me talking, you also get homework. Good luck, team. [Inaudible 0:24:20] and a whole bunch of very smart people in the OWASP space, open web application security project, have released the first OWASP Top 10 for LLMs. As well as what I'm about to say, I highly recommend you go and have a look at that OWASP Top 10. This looks very different from SQL injection, cross-site scripting. It is a different way of thinking. It can be a really great way to start thinking through the layers, the challenges with, I almost said problems, but the challenges with LLMs.

If I was going to break it down and give you the high level, so some cliff notes, if you will, and there's a few things you want to look at. Where did the model come from itself? What is it built from? Now, there's sometimes you're going to have a lot of visibility on this. If it's open source, you can really dig into these things. If it's not, you're going to have to have a conversation with people. What is underneath this? What is the algorithm was being used? Sometimes they'll share, sometimes they won't.

You have a responsibility, if you're using software, whether it's a picking an open-source library to use, or using somebody else's model, we have to do our due diligence and go, “Hey, what is inside this magic box? Is it okay to use?” First tip for you is to start thinking about what are the questions I'm going to ask of every model that I start working with? Because then, you can turn that into a little checklist that you can do and make it a consistent process.

Outside of the model, you've also got the training data. Training data is fundamental. You need it and you need really high-quality training data to have a really good-quality model, because essentially, a model is regurgitating that training data, predictably, it's looking for patterns and combinations of those bits of data. Now, you don't always get to know where the data has come from, but you should be able to ask things like, is this still training? Is the data that's being put into this being used to change the algorithm, to change the data model? If it is, then you know how quickly that's going to be changing.

If it was trained once statically, but is no longer training, that's different. In those circumstances, you can say, “Well, all right. Although I can't predict all of the outcomes here, there shouldn't be any big surprises, because there isn't anything new coming in.” It also talks about a risk there. One of the risks with the model is how it's trained. If you can put malicious data into a model to train it, that model will reflect whatever was put in. Much like in any software, garbage in means garbage out, evil in is evil out. You want to be making sure that you're understanding how data is vetted before it is used, and what controls are there to make sure that malicious data isn't entering.

Now, as well as malicious data, you probably want to consider bias as well, because biased data can also give you really unpredictable and unsafe results, just not in a traditional security way.

[0:27:03] Simon Maple: Yeah, and I think everything there from ethics, as well as morality and biases and things like that are really, really important.

[0:27:09] Laura Bell Main: Absolutely. We all end up having a software developer's existential crisis at this point. We've got our data, we've got our model, and then we've got what we're doing with it. Now, what you're doing with it, there's technical vulnerabilities, things like prompt injection that were just mentioned here that the OWASP Top 10 for LLMs does covering quite a lot of depth. There's quite a lot of good primers out there on that. I would recommend going and having a look at some of those.

The other side is, how are you using the output of this model? Now, you have to treat every piece of data in a system as untrusted. That's core rule of software security is trust nobody, trust nothing. Now, we're not that great at that even with non-AI systems. There's a lot of systems out there that if the data comes from an external user like me, I’m not going to trust it. Going to do lots of validation. But if the data comes from another micro system in my architecture, oh, that's fine. It's come from internal. I won't do so much. That's a really dangerous assumption.

Now, the foundations of zero trust are that we don't do this. Everything that comes into us, we verify, and that we do our analysis on to check that it's suitable to use. Our AI components are no different. We need to make sure that whatever comes out of them, that we don't just take it at face value. Are the checks and balances we can put in place to ensure that that outcome, that analysis, that result, whatever it is your thing is suitable to the context, that it's safe to use, that it's trustworthy.

Whatever your context is, it's going to look different. If you're in a financial system, that's going to be a very different thing. For example, it could be a credit score, or can I lend on this loan? If you're in a CRM, it could be enriching data. It could be saying, “Hey, you've got this customer record. I'm going to pull you extra information and give you some insights into your customer.” There's lots of applications. For each one, you need to look at, right, what is the data that this model is giving me? How can I verify it? How can I flag whether it's safe or not? What is the impact? What is the result if this data is not trustworthy?

In a financial system, for example, if the decision to lend, or not lend is based on an AI engine, we would probably need to put some scrutiny in that took a sample set of those decisions over a period of time, and still ran them through a manual verification process, just to double check that the logic rules we hope are being applied are in fact being applied and that the decision is fair.

You can see how that would apply to an immigration system, for example, if you were using AI to determine if somebody was valid for a Visa or not. In your company side, sorry, I'll just finish that little bit up. On that company side, although it seems less important because you're just pulling company data to enrich something, if you're basing your sales workflow off those enriched data pieces, or doing automation, you can cause a heck of a lot of disruption in a company and damage their reputation by getting that wrong.

[0:30:11] Simon Maple: No, absolutely. I think the OWASP Top 10, I think referred as part of that as over-reliance, right, in terms of leaving that day. Actually, one of the things, which I love the way they say it, is pre-input even, as well as output from in-tune an LLM, as you would use a data, which I think is a core rule to think about.

When we do think about threats like prompt injection, which I believe the OWASP Top 10 puts as number one, it's exactly what you said. It's unpredictable in the sense of we don't know necessarily what a user is going to provide to it. There's an infinite number of inputs that a user can provide that. Again, there's going to be an untrackable number of outputs that the LLM can provide us with. Is there a foolproof prompt injection mitigation that we can use? If not, I think you alluded to it there, guards and things like that being able to validate and verify output. Is this the technique that we should be using for these kinds of eventualities?

[0:31:05] Laura Bell Main: Yeah. I could be really facetious and say, you could just turn it off. That's the best way to protect against prompt injection. No, there isn't. There's no foolproof way to get past any vulnerability in software. We've had SQL injection vulnerabilities known about for now 20 years. We still see them. That's because there is no guaranteed solution. That's okay. Part of being good at security is accepting that there is no magic box that is going to do this for you. There's no, I missed a button, turn on the button and you're fine. It just gives us that responsibility. You're right, it's guardrails.

It's having shared patterns in our teams. A lot of this is new. You can't teach this at university. It's genuinely going too fast. Most of us, even those of us who are personally passionate about this area, there's so much happening every day that it is incomprehensible how much change you're working with. The best thing you can do in your teams is make this a known and shared problem. Make it a fun thing that you do once a month. You get together for lunch, or whatever, try and get your boss to buy pizza, if that's your thing. Come together and go, “Cool. What changed in AI this month? What does it mean?”

Really start collaborating and sharing, because there is not one of us that is going to be able to come up with the perfect set of guardrails, or solutions, or fixes for this. Some of the best people positioned to come up with anything right now are not security folk. They are engineers. This is probably one of the hardest security engineering problems we have ever seen emerging. It's not going to be someone who has a networking background who fixes it. It's going to be people who are knee-deep in these systems and code and really understand how they work and can collaborate in a way that is going to show us, okay, this works in these circumstances and these are the limitations. You need all three pieces. No magic boxes. Just, okay, we can do this. It will work to this degree. Be wary of this. The more we can share those things, the stronger we're all going to be.

[0:33:11] Simon Maple: Yeah, it's really good advice. I also like, with things like that, with almost assuming something like this will and can and will happen in terms of like, yeah, maybe we will have an attack on prompt injection, maybe an attacker will be able to get through. It’s risk reduction. Ultimately, do we have everything else correct behind the scenes in terms of principal, at least privilege in terms of data access and things like that? So that if something like that, maybe we think when something like that happens, are we best protected internally to the potential damage that someone could do? I think that's almost where we are, right? For those risks at times.

[0:33:48] Laura Bell Main: I think we're going to start seeing a change in the way we talk about how we measure success in security as well. For a very long time, we've been focused on, we had 10 bugs in our software this month, and the month before, we had 12 and therefore, we’re making progress. When you're looking at systems holistically as you have to do in AI, that doesn't work anymore. They are rapidly evolving, such that if everything stayed the same, and you could say, there are more or less vulnerabilities each time, that would be a valid metric.

When the system itself evolves exponentially quickly, then you're not comparing the same thing each month. What we're going to start seeing come through is more focus on leading indicators. Looking at things like, how many issues did you identify in securities this month, versus how many did you change? Looking for actions that are consistently applied throughout the whole SDLC in ways that you can say, “Right. My whole team is participating. Every part of my system has been involved in this process this month. We know that we're not just identifying, but we're addressing.” It's looking at that much more like a continual process and a culture than it is just a, is it safe, or is it not? Is it secure, or is it not?

[0:35:02] Simon Maple: Yeah. Move on to that second piece, actually, that you mentioned earlier in terms of using AI to almost determine various threats in the application and being able to identify that to almost like, new to some of those metrics that you were talking about there. When we do assess the threat analysis, basically, of your application, you mentioned you would tell the modeller about your application. Do you ever see, for example, an LLM-based threat model, or whereby it learns about the application and it tries to identify itself? Or do you see it more, almost like a tool that learns from our teachings about the application, about the architecture, about the usage to then almost like, trying to extract the output from it as to where the various risks are? I'd love to hear your view of how that all –

[0:35:50] Laura Bell Main: Awesome. Please note that all views you hear on the Internet, and there are many and they vary. I think we're going to see generations in this. I think in the first stage, what we'll be seeing is AI tools that can facilitate processes. It's very difficult at the moment for you to give enough information to a model, even if it's model you've built yourself, that could coherently understand what your application and your system is.

What artefacts would you need to feed into a model to do that? Yes, the source code is one thing, but then what about your architecture? What about, how it's being used, your logs? Then there's a heck of a lot of risk about putting all of that information into a model, unless it's a model that you've trained yourself. That's a lot of work. I foresee only really the top end of town doing that, the very big organisations doing that level if they chose to.

What we can do that's much easier is that we have lots of proven models and thought processes and question sets that we have used for threat analysis before. Whether it's attack trees, or whether it's STRIDE, so by teaching those frameworks to a model, the model could in its own way begin to coach us through thinking through these processes. I think it's not as glossy and exciting. It's not a magic bullet, but it's a facilitation.

I think this early generation of AI assistance in threat assessment is going to be efficiency. It's going to be productivity tools in a way. How do I make this easier? How do I make the results of this easier to get into the systems I need to get them into? How do I make sure that I can do this in a way that there's been 15 pull requests on my system and it touched that authentication bit. Maybe we need to do a threat assessment now. It decides whether, actually, enough has changed that we need to think about this again. Those kind of enablements are going to be really, I think that's very short-term. We'll see those in the next six months to a year.

Longer term, we still have a lot of questions about the safety of the amount of data you would need to put into a system to truly threat model it. Like I said, if you can do that internally, fantastic. If you're Google, I'm sure that's completely achievable. If you are 500-person high-growth company somewhere, this is well outside of the reins of what you can do. If someone does offer you a tool that can do it, then you've got some really big challenging due diligence questions. How is your data used? Is it being used to train something? Who else can see it? How long do they keep it for? There are not proven models right now for safely having those conversations, those negotiations. Sadly, I think we're not there yet.

[0:38:20] Simon Maple: All those wonderful, fun, privacy conversations that we've been having most recently as well, right?

[0:38:26] Laura Bell Main: Absolutely.

[0:38:27] Simon Maple: Yeah. At the start of our discussion, we talked about that room, where you have developers in it, you have some security folks in it, you have some folks who are close to the end user, the customer who can all provide their point of view. I guess, now when we pull up another chair for the AI tool who can help us threat modelling, is this a new chair? Are they taking someone else's chair? Who is most to interact with this new wonderful AI tool that we have?

[0:38:54] Laura Bell Main: Oh, I don't know. It's really hard, isn't it? I think inputs, I think the AI from an input perspective is going to be very tightly entwined with both the systems that we're building, so source code architecture, that kind of thing, but also, our customer support and success. Thinking of things like semantics and trouble tickets, all of those things are going to be really useful and log ingestion. I think there's a variety of players that need to be involved in the feeding and watering of the system and making it as useful as possible.

I think at this stage, it's most likely that the dev teams themselves will have the most interaction, but I would query whether it was those who are writing code, or whether there's an opportunity here to give some superpowers to some members of our software team, who we often overlook the influence they have.

For example, our product people. Product people are really, really early in the process. The business analysts, the people who are trying to understand that problem and articulate it in a way, such that it can be built. I would say that, yup, absolutely, our developers right now, but with time, wouldn't it be amazing if actually, we bought it even earlier in the lifecycle and that our product people, our business analysts, our UX designers could all use this as a tool to simplify security by design, so that by the time it got to implementation and that grooming phase there, a lot of the things that already been considered already taken into account.

[0:40:23] Simon Maple: A lot of the time, we actually don't ask ourselves enough when we do threat modelling, do I have the right information? Do I have enough information here to actually make a reasonable decision about the outcome of this? That's actually one thing that AI does reasonably well at times. It says, well, there are various opinions on this, and sometimes stutters before trying to give you an outcome.

I did that just today, where we were talking about whether Die Hard was a Christmas movie. I had to ask it three or four times before it said it was. Maybe that's a bone of contention. It's interesting that I can see an AI system like this being able to say, “Well, actually, I don't have enough information about this piece of the system.” Where we as humans might say, “Well, actually, I'm almost not asking that question as to whether I have enough information to make a good call. Maybe I'll just make that call, because it's my gut feel,” or you make it with too many assumptions, perhaps. It's almost that nice piece, where it can determine itself in an automated way. Ask that question of, do I have enough input here to make a reasonable enough judgment?

[0:41:26] Laura Bell Main: I'm going to go deep, though, because the second we get into this, we get into the same problem you see in medicine. If your AI system is going to say, “Hey, I can't do that, Dave. We don't have enough information,” but your lead engineer can still press the button and go, “Well, I'm going to do it anyway, because my gut instinct says it’s fine.” Now, we've got a bit of a danger. In medicine, what happens is we have a lot of med tech systems that can say, “Hey, do you really want to prescribe that? It has a counterintuitive situation. It's going to have effects with this other thing and people might get hurt.”

What can happen is the more senior you get as a medical professional, you can override these things and say, “No, I know better.” Now, in some cases, that's the right choice to make. In some cases, it's not. We're going to have to be really mindful of humans that we take a little bit of stock that we have some self-awareness of when to trust our gut and override the hang on, we need more information. When we need to go, “All right then, computer. Sure. Who do I need to talk to? What information do I need?”

I know from my own perspective, I've been guilty at times of going, “Oh, evidence, frevidence. It's fine. How hard could it be?” ust pushing the button. I think we're going to all need to be our best version of ourselves if we're going to take advantage of the fact that the model might be able to spot gaps that we haven't spent the time to look at ourselves.

[0:42:46] Simon Maple: Yeah. I love that extra step of effective validation and verification, really, about that judgment call, whether things are right or not. It's been wonderful. It's lovely to chat with you, Laura. I'd love to ask one final question, staying on the topic of AI. That's really, if AI were to take a piece of your job away, whether for good or bad, what piece would you like that to be?

[0:43:11] Laura Bell Main: Actually, AI is doing a really good job of some of the boring bits of my life. I'm quite pleased. I'm going to pick something from my personal life and break your rules a bit. I have two daughters. I have a five-year-old and 11-year-old. Between them, they're obsessed with Harry Potter, Paw Patrol, and a number of awful TV shows that they think are the best thing in the whole wide world.

For me, writing coherent crossover fan fiction that I can deliver in five minutes at bedtime without having to – Often at bedtime, I will get asked, “Hey, I need a story about Sky from the Paw Patrol who meets a dragon and Harry Potter and they eat a burger with a cat.” Sometimes I'm okay to do that, but sometimes I'm just too tired. The magic AI storytelling bot would make me and my children a lot happier, I think sometimes.

[0:43:56] Simon Maple: That resonates so strongly with me. Right now, I'm telling stories to my kids called the Great British Bark Off, which is across that I'm making between our two dogs on the Great British Bake Off. That resonates so strongly with me. Yeah, wonderful.

[0:44:09] Laura Bell Main: That's amazing.

[0:44:11] Simon Maple: Well, Laura, it's been a real pleasure chatting with you. Thank you for all your insights. I'm sure people will have absolutely learned many, many things on this and take many, many actions as well for their team. Thank you very, very much for joining us on The Secure Developer.

[0:44:26] Laura Bell Main: Yeah, thanks for having me. It was a lot of fun.

[0:44:28] Simon Maple: Wonderful. Thanks for everyone for listening. We look forward to our next episode on The Secure Developer. Thanks for listening. Speak to you soon. 

[END OF INTERVIEW]

[0:44:40] ANNOUNCER: Thank you for listening to The Secure Developer. You will find other episodes and full transcripts on devseccon.com. We hope you enjoyed the episode. Don't forget to leave us a review on Apple, iTunes, or Spotify and share the episode with others who may enjoy it and gain value from it. If you would like to recommend a guest, or topic, or share some feedback, you can find us on Twitter @DevSecCon and LinkedIn at The Secure Developer. See you in the next episode.