Securing The Future Of AI With Dr. Peter Garraghan

[INTRODUCTION]

"Dr. Peter Garraghan: I remember thinking, I'm not so sure if security tools work against deep neural networks. At the time, we're looking at what we call CNN's and RNN's, which are like image and NLP models, and thinking, "Well, I don't think it's going to work because, A, these deep neural networks are very, very random." No, we're used to random software, a non-deterministic system. Non-deterministic is multi-threading. But this one's very, very, very random. It's also incredibly opaque. If I think of a SaaS solution, there is no code in the neural network. It's a bunch of matrices and probability. I can't kind of unravel semantic meaning. I think, ‘Okay, that's interesting. Let's spend the next few years interrogating the security problems within deep neural networks.’"

[0:00:41] Guy Podjarny: You are listening to The Secure Developer, where we speak to industry leaders and experts about the past, present, and future of DevSecOps and AI security. We aim to help you bring developers and security together to build secure applications while moving fast and having fun.

This podcast is brought to you by Snyk. Snyk's developer security platform helps developers build secure applications without slowing down. Snyk makes it easy to find and fix vulnerabilities in code, open-source dependencies, containers, and infrastructure as code, all while providing actionable security insights and administration capabilities. To learn more, visit snyk.io/tsd.

[INTERVIEW]

[0:01:23] Danny Allan: Hello, and welcome to another episode of The Secure Developer. I am your host, Danny Allen, and I'm very excited to be here today with you, with a very special guest, around a topic that is at the top of everyone's mind, and that is AI. And that guest is Peter Garraghan. Peter, welcome to the show. How are you?

[0:01:40] Dr. Peter Garraghan: I'm good. How are you?

[0:01:42] Danny Allan: Excellent. Now, Peter, I know a little bit about your background. We were talking before the show started. But maybe you can just introduce yourself to the audience, and what your role is, and what you do on a day-to-day basis.

[0:01:54] Dr. Peter Garraghan: Sure. Happy to do so. I'm Peter. I am the founder, CEO, and CTO of Mindgard. It's a company based in London that specializes in AI red teaming. I'm also a chair professor in computer science at Lancaster University and I specialize in security of AI systems.

[0:02:13] Danny Allan: Now, Peter, when I think of a professor or educational institutions, my first job was for a university, by the way, I tend to think of them being well behind the industry, and the fact that you're both in AI security, AI is the hottest thing, and in the education space, is that true or not? Because I'm going off a frame of reference that's 25 years old.

[0:02:34] Dr. Peter Garraghan: Sure. It varies. It depends on your time scales. If you think about academic research, you have a pendulum between the conceptual and the theoretical for looking at future problems like in the span of maybe sometimes decades and you're looking at actually problems that exist today. And there's discussions about actually the problems today with their – they don't have enough funding to keep up with some of the big tech companies.

But I think take a step back, if you think of deep neural networks, it's not a new concept. It's been around for decades, even before industries are thinking about this as a concept. There's kind of that. The conception of the article is super useful, I think. And, actually, what we see today, it does feed into it.

In terms of problems today and practical problems, researchers, and scientists, and academia do have a bit more freedom to pursue these things. That means you'll pursue avenues which may not lead anywhere into the near term but sometimes they can crack a problem that someone in industry might actually really struggle with because it's a new concept and they figure blue sky thinking.

The way I think about this is – I do systems research. I build solutions for big tech companies and solve problems to kind of combine together both the blue sky, "Hey, here's some interesting problems that are coming out in the future." And I'm happy to talk about why on earth I'm doing a copy of this space, but also things that are kind of happening right now.

[0:03:53] Danny Allan: Yeah, that makes sense. It's funny, back in the 90s, I was studying homomorphic encryption, which was totally theoretical, and I was so excited about whatever it was, three or four years ago. All of a sudden, they were actually doing homomorphic encryption. It does take time for these things. In your opinion, when did AI start? I know AlexNet, it became a thing. Or deep neural networks became a thing – what was that? 2014 that they started doing the classification?

[0:04:20] Dr. Peter Garraghan: Yeah. This also feeds into my knowledge in this space. Let's go back even further. Let's think of new technology in general. Virtualization underpins cloud computing. No one's going to argue about this, I think, virtualization. Virtualization kind of existed back in the 60s with the mainframe computers, but memory was so expensive, they think, "Oh, we can't virtualize things."

It wasn't until the x86 architecture got virtualized and memory came cheap enough, "Hey, I can scale this thing." Cloud computing, it's now awful a lot of money that had like a 50-year gap. Think of neural networks, artificial neural network goes way back, several decades. So people won some awards for this. It was like you mentioned, 2014, that they figured out that these deep neural networks can now perform other ML tasks. And now, back 2017, the transformer architecture, which underpins LLMs, became popular. And now you can see what happened about the space.

We talk about AI. They're talking about deep neural networks. And more specifically, they're talking about transformer architecture mostly. But people have been using machine learning neural networks, classification for image, and text, and audio for a long, long time since recently. Now it's become much more interest, but also types of use cases are forming from it.

[0:05:33] Danny Allan: And is that practicality coming because GPUs or the capabilities have expanded rapidly in the last 10 years? Why now? Why is it happening now when they've known about it for so long?

[0:05:43] Dr. Peter Garraghan: I think it's a mixture of things. If you think of the transformer architecture, which underpins LLMs, at the time when that paper came out 2017, 'Attention Is All You Need', 'All You Need Is Attention', there are other research labs trying to solve that same problem. It's all about natural language processing, and they kind of cracked that issue.

It was a combination of a few things. I think one was the tech companies that kind of had the GPUs at scale and all the internet's data could train these really big models to do some really interesting things. That's kind of one big aspect. The other one is that it became cheap enough and enough data that it gave results that allowed to penetrate the public consensus. Actually, I can spin this thing up on a website and talk to it and anthropomorphicize it to – I'll talk about that is also an issue. But it's a combination of things.

One is, yes, the transform architecture can do some really cool things, and that kind of taps into how people interact with these type of systems. But, also, the tech companies can scale these things up massive but also transformer our architecture. Attention mechanism really opened up a lot of possibilities, like NLP techniques and generative AI.

[0:06:57] Danny Allan: Where do you think we are on that time frame? It's interesting, I talk about AI kind of in two different ways on this show, or we've had guests in two different areas. One around AI-accelerated development, because we have all these coding assistants basically that are – you have people saying that there's going to be no developers, and 90% of code is going to be written, or 100% of code is going to be written. That's one category. And then there's AI native applications, which is really what you're talking about, is software that's backed by LLMs. Where do you think we are in the time curve of that? Are we like day one of 365? Day 10 of 365? Halfway through?

[0:07:32] Dr. Peter Garraghan: Yeah, I think it's an evolution. We've had AI for many, many years. Again, I love analogies, so I'll give you another one, which is I call it the great sieve theory, which is imagine you have a big sieve full of flour. This flour – again, go back to cloud computing. When cloud computing kind of hit more mainstream, let's say, 15 years ago, people claimed you can do absolutely everything. Everything can be solved by cloud. Everything can be virtualized. It's great. The world's your oyster. Lots of investment. Lots of developers throwing cloud resources. And those ideas and use cases are the flour and the sieve. If I shake the sieve slightly, it all falls through. What ends up happening, you got four or five pieces. Each of those are worth a lot of money and super, super valuable as use cases.

Fast forward to now with this generation, generative AI, everyone's claiming AI can do absolutely everything. Yes, it can do docking summarization. Yes, it can basically do everything for me. You shake the sieve slightly, a lot of things fall out. But the things that remain actually are super useful and super valuable. Thinking of today, a lot of the cases I see for use of generative AI, what's it good for, it's good for [inaudible 0:08:37]. It helps me support coding activities. It helps me support for idea generation. It helps me for the docking summarization. A lot of people using transcription, docking summarization is super, super useful. It's good for translation, for example. These use cases are worth a lot of money.

The other use cases they will evolve over time. But what happen is that the AI today will become ubiquitous, and new types of AI kind of fall from it. In terms of putting on a pin, in the whole life cycle of AI, it's just another blip in the – I think my view is actually it's another pin on the board, which is quite important. But the use cases that are emerging today could be useful. If you think five years from now, will it replace every use case and application? I personally don't think so. But will it really help and hyper-focus some of them, make them worth a lot of money and super valuable? Yes. The question was always which ones actually are they?

[0:09:31] Danny Allan: Well, you already mentioned some that are interesting. Now, all three that you mentioned were text-based, large language model, which was code generation and document creation, etc. Do you think that that is where the big elements in the sieve are? Do you think the video, the audio, the diffusion models are sifting through right now and we're going to focus on text? I guess I'm just curious, in a multimodal world, does the model matter?

[0:09:57] Dr. Peter Garraghan: If you think of multimodality, again, depends on the use case. A lot of companies are using these to generate cheap videos, and you should make cheap audio. And that is super useful. Again, it's come up cheap enough for a startup to spin up their own resources to do – and that could be useful. Again, there's lots of discussions about the quality of that, and copyright, and the philosophy of human endeavor for artistic license. That stuff is quite important. But I strongly think that it will find a use case in that workflow.

I think of the game industry. If I'm a designer, an artist, I should be using it not to replace my entire art team. No, it helps my artists to get some storyboarding or prototyping, get some ideas, and then I take those ideas and then I should define it if need be. Because the thing of most things and the AI can do is it helps me cut down a lot of the laborious automation work, which I want to do. But I still need to kind of finesse it around the edges. The only exception is I have an incredibly strict use case, a very strict input parameters and output parameters. I can just use – example I give people is think of regression, which is it is AI. It's used for inputs and outputs, and it doesn't do generative. Just does this thing. It has this use case. It's super useful to me.

I'm seeing two worlds here. One is people using generative AI, and multimodality, and even agents. One view is that, yes, it can replace all use cases, which I think is pretty hard because there's lots of nuance, and processes, and other components to look at. Versus I'm hyper-specializing this library to do one thing and one thing only. That one kind of makes more sense to people now, but we're seeing a lot of people play and experimenting about this.

[0:11:35] Danny Allan: Yeah. And those use cases are going to shake out for sure. We've actually been doing a regression analysis at Snyk. The way we do our SaaS testing, actually, symbolic regression analysis on the abstract syntax tree. You get these use cases that it works really well for, and you can do amazing things.

I'm interested in the fact – your company, I know, is focused on the security of LLMs. Why did you pick up on security? I mean, obviously, here's what's interesting to me – and then I have a question. What's interesting to me is that you get these crazy valuations on generating things, like building something new, and doing something interesting, and reading X-rays, or whatever it happens to be. And all these companies with multi-billion dollar valuations tend to be on the generate something side of things. But then you have the control side, which doesn't tend to be nearly as exciting, and it tends to follow on, and security follows in that category. I'm interested – my question is this, why did you choose security? Clearly, there's something that you're seeing that makes you believe we need to be focusing on this now.

[0:12:35] Dr. Peter Garraghan: Yeah. The advantage of being a professor is that you kind of see things coming from a long way away. I started looking at this problem about a decade ago, maybe 12 years ago, before transforms even existed. I have a big research lab in the UK. I train most people with some security at doctoral level. I remember thinking, I'm not so sure if security tools work against deep neural networks. At the time, we're looking at what we call CNN's and RNN's, which are like image and NLP models, and thinking, "Well, I don't think it's going to work because, A, these deep neural networks are very, very random." No, we're used to random software, non-deterministic system. Non-deterministic is multi-threading. But this one's very, very, very random. It's also incredibly opaque. If I think of a SaaS solution, there is no code in the neural network. It's a bunch of matrices and probability. I can't kind of unravel semantic meaning. I think, "Okay, that's interesting. Let's spend the next few years interrogating the security problems within deep neural networks."And we did. My scientists and engineers figured out. This is actually really hard to do. And there's no tooling in this space. You need to know both security and AI at the same time. People tend to specialize in it.

And the reason why I think it's useful to do, I care about social good. That's why I'm a scientist. That's why I run a company now. And with the AI you're seeing nowadays becoming very ubiquitous, and pervasive, and deployed in everything, AI is still software. If you got you're deploying a piece of software that has a bunch of issues that you cannot articulate, can't test, can't find, this is a huge problem, both from a system security issue, which is a confidentiality, integrity, maintainability, availability, all these issues. But, also, at the very heart of public discourse, AI is being used in a whole different information, deployed in infrastructure, information sharing, replacing use cases.

It's the same as saying, "Well, should I care about any software security?" The answer is I hope you should, because lots of risks, both technical and economically, but also societal. With the AI adoption as it is, we found that the current tools and security didn't really work for it, but they're analogous. Hence why, back in 2022, I built this company. But this was a long time in the makings. I made a talk about this 10 years ago thinking this is a problem, but no one has AI enough at scale to justify it. It wasn't until 2022 when I started seeing people really going to get transformers uploaded, starting to build traction in the space.

[0:15:00] Danny Allan: I love the idealism of that. And similar to yourself, I'm passionate kind of about the greater good because it is a problem. I've been in the security space for a while, and I always have told people historically it's all the same stuff that's just happened over the last 25 years. It's input validation, it's output encoding, it's authentication, it's authorization. The reasons why the vulnerabilities, it doesn't matter what it is, changes. The reason why it occurs is the same, although the infrastructure might change.

That all made sense until 3 or 4 years ago when you started looking at AI when it is non-deterministic. I can send a single tick to a database and cause SQL injection, or I can put in something that results in cross-site scripting. How do you actually do security testing in a non-deterministic world? How do you deterministically tell if there's a security issue in something that is inherently non-deterministic?

[0:15:55] Dr. Peter Garraghan: Again, I'll even go back a thousand years ago, which is a thousand years ago before computers, security issues could be somewhere around the different towns, basically throwing information out to try and incite riots. We're doing that now with laser precision at speed and velocity that you couldn't see beforehand. There's actually the sun here conceptually. There are some differences now.

Fast forward now in generative AI. What are some of the differences? One of the big problems is the data plane and control plane are the same thing, which is really problematic in security because I can't separate instructions to the system as application logic. We've tried for a while, and we've abandoned it for now. You actually can't do it as well as we should do. The second thing is, yes, can I do formal verification of all possible combinations of generative? No is the answer. It's an infinite search base, combinations of tokens. That's not the way to think about it. Will there be eventually? Probably. I bet. There's lots of great research right now trying to do a very, very small scale. But as you keep adding more complexity, it gets super complicated.

But, again, this is not a new concept. We do formal verification for satellite systems because it's simple enough to look at all the state transitions, and then we can figure out how to recover. In big, complex systems, no, you don't do that. It's too expensive and not worth my time. You think about security in AI and deep neural networks, the case is, yes, you need to do just testing. What type of testing am I doing? And how long should I do it for? And what is my confidence? These are all great questions that you do, and also for testing, which is I have a budget, I have a threat model, I have some risks I want to prioritize. I'm looking at the feasibility, but also vulnerabilities. How much do I test to actually eliminate this? And the main thing I tell everyone in the space, scientists and my customers, is it's still software. There's nothing new here conceptually. It's still the same, but there are nuances like the data control plane, and optimism, the opaqueness of the system.

Think about back to basics. Replace AI with software and say, "Okay, I have a use case. I need to have a threat model for my use case. I either prioritize my budget for testing, modeling remediations. I do it, and then I can figure out if I have done enough or not." You can't find all combination in that space. AI is the same. And some of the open problems in the space now, actually, how much testing do I need to do? Well, you can't do none. That's really really bad. You have to do some. The question is, "To my threat model, what type of risks are relevant to me? And then what controls I have to put in place for it?

[0:18:09] Danny Allan: How do you manage – I know that you're involved kind of in the red teaming space and testing along. How do you manage the context windows in a sequential manner that learns sufficiently to know what the next layer is? And how deep do you know to go? Because when I – you're essentially trying to do – I don't want to use the term AGI. Well, I used to be a pen tester a long time ago, and breaking into a system was always a sequential step of like 10 things that you would do to get to the final step, but you needed a lot of creativity and brain power to know how to get to the 10th step where you actually extracted the data and got it out of the system. How does AI do that? Is it AGI? Are you approaching what a human would typically do? I truly don't even know how this works.

[0:18:55] Dr. Peter Garraghan: Yeah. Yeah. I think the phrase AGI is used quite loosely. So I try to avoid it entirely because that's [inaudible 0:19:04] put it lightly that way. But I think your point is a good one, which is one of the difference of the transformer and generative AI is that one of the big differences is that you think of APIs. You basically essentially have an API that uses natural language to do things. And that makes it very fascinating and very interesting, which is I can do all these capabilities, but it makes it incredibly painful to do any testing and security about it. How do I do some other things?

You go back to think of the pentest as a good example because we work a lot of pentesters. The example is – I'll give you an example of this. We have an engagement. If I just ask the AI, "What do you do?" It will tell me what it does. Great. Shannon's entropy, which is the idea of security bits of getting information of who – not anonymity. Great. If the AI will tell me, "Yes, I can call a database. Yes, I can retrieve orders. And, yes, I can send emails." As a pentester, that's great. You told me already the things I can do. I can then prioritize my next step in that system.

I'll give you a classic example of what we see if you're a pentester. I have an application with AI inside it. I've been told this. Step one, is there a guardrail? Yes, it blocked me. That's good to know. I'm going to make sure I need to be careful I'm doing other things. I then figure out, "Hey, it can recognize other languages." That's not a problem. That's useful to know, because I might come back to the capabilities later on. I then ask it, "Hey, what do you do?" Like, "Oh, yes, [inaudible 0:20:25]." That's interesting. I think you have a database tool chain in the background. Now let me put in a prompt injection that's encoded in the format the guardrail won't – see if it responds to it. Great. Then I can do the kill chain, which is, yeah, SQL injection, prop injection back in system again.

This is not new, conceptually. The difference now is that the AI can be socially engineered to give information, natural language, as opposed to I could do as a pentest very, very slowly, but it takes a lot of time. But what we really think is that a lot of manipulation of the AI, A, to trick it, to do things it shouldn't do. But, also, will give the information willingly sometimes. Say, "Yes, I'll tell you everything about my system." Or sometimes it will say I can't answer that or I won't answer that. Won't is interesting. That means it recognized what I tried to do, but it's not going to do it. I know I have [inaudible 0:21:11].

[0:21:12] Danny Allan: If you ask what it won't do, will it tell you what it's not allowed to do generally?

[0:21:16] Dr. Peter Garraghan: Sometimes. Even that is enough. For example, I can try, upload a document with a hidden image inside and say – parser says, "Sorry. I won't do this." That means it recognized what I was trying to do. I now know that's interesting. That means it can recognize this thing. Oh, I have to now figure out is the way to get around that system. It was just I don't understand this thing. It means, yeah, this thing's not compatible in the system. Again, same as an interrogator. Or if I'm to interrogate somebody, those responses as a pentester or red team is like that's flagging. That's interesting. It's not a problem necessarily, but that's evidence I'm using to build my attack stage again.

[0:21:50] Danny Allan: How standardized – No, go ahead.

[0:21:52] Dr. Peter Garraghan: Again, it comes back down to – so we have a lot – a lot of people trying to pentest AI systems applications. And what we try to encourage them is that they can still use their processes and controls, and all these things are great. The way of thinking that however is that you're kind of social engineering this thing to give you information up but also tricking it. And the way you trick it isn't as simple as I need to run a recursive testing harness combination saying, "Hey, what do you response me with." I should look and then say, "Okay, that's great." Now I need to then go my next step and – again, example I gave people is if I went to a call center, call them up, could I say there are exact words and information that make them so angry that they're going to delete someone's accounts? Probably. [inaudible 0:22:33] of the world, yes. And that happens in LLMs that if I get the right combination of things, could I get it to delete some information? Yes. If you have good security controls in place, that shouldn't happen. But as we know, secret injections are being solved for many years. It still keeps happening in the system. So, same problem.

[0:22:50] Danny Allan: Yeah. How standardized are the attacks? If you're trying to attack an LLM, is it a similar approach that you would use for all of them or is it completely creative and you're going to use AI to generate an attack or a red teaming kind of assessment of a system?

[0:23:07] Dr. Peter Garraghan: It's a mixture. I haven't seen novel attack in AI that you've not seen in cybersecurity. There's perfect analogies between SQL injection, prop injection, between detection system bypass, and gather bypass, and database extraction, and model extraction. The ideas are there. The techniques are very different though as you mentioned. You think of prompt injection – again, this is another point I make emphasis about is threat models and use cases are super important in this case. Could I do jailbreak and prop injection? Talk about bombs and these things? Yes, that's actually quite easy to do. But do I actually care about that? That's the bigger question. I can go online now and search how to build a bomb. Someone probably might knock on my door by saying this, but information is available in a data set.

[0:23:56] Danny Allan: You know, this is being translated, and you're going to get an MI5 knock on your door or something.

[0:24:00] Dr. Peter Garraghan: Exactly. The problem we find in this space is once you know the use case – and there's a double-edged sword here. I want to do attacks that target that use case. For example, if I have a public-facing AI chatbot to talk to vulnerable people about healthcare records, the type of attacks I'm going to launch are very different from an internal coding chatbot that I want to see. If it's my internal chatbot, if I spent an hour trying to make it insult me, who cares? It's my product. It's not an issue. But for a public-facing entity, this is not just a security problem; this is also a quality problem of the product itself. They might try to – because the person in question might be trying to incentivize to do so for a whole bunch of reasons, not just reputational damage.

The type of attack you're seeing is prompt injection jailbreaks. You can get it – eventually, prompt injection jailbreak, that's the case. The question is can I get it in a way that gets past it in a feasible amount of time that also causes damage? Give you one example. If I have a candle shop – use generative AI to have a candle shop. If I build basic AppSec and have some input-output filtering for bad words, and using off-the-shelf guardrails, and just some basic security thinking you will block nearly every generic prompt injection guardrail in existence because they'll block it. Sure. But if I know the context. Okay, you talk about candles. What's related to candles? Well, fire is related to candles, and sending emails to your customer calls. I can do like things like market injection attacks by putting in a URL that has candles inside it. That'll be blocked in another guardrail. But because it's context relevant, it'll think it's actually normal. Again, it comes back to social engineering element.

The trick we see in the space is a combination of, yes, you can use AI to help inform the red team or pentest to augment abilities, to look at context-relevant things. But on the other hand, we've seen quite often is that the red team pentest knows saying, or the developers says, "I kind of know what I need to do. I don't want to spend the time generating prompt injection. Can the tool do it for me if need be?"

[0:26:02] Danny Allan: Is most of this stuff, Peter, theoretical? Are you seeing attacks within the – I don't know, the public, or the public domain of trying to break these systems, or is it mostly academic and theoretical?

[0:26:14] Dr. Peter Garraghan: A lot of practical problems we see day in, day out. There's kind of three groups that I would put these in. Adversarial machine learning, which is around 15 years old now, I would say, that was looking at what we call perturbation and try to figure out image – classify such problems. That's been around. A lot of that started in looking at conceptual problems which researchers and scientists states should be looking into. And even with AI safety nowadays, a lot of jailbreakers saying, "Hey, we could break this thing. And here's the problem."

I have a lot of quibbles about is it actually relevant or not to most application use cases? But that's a different conversation. Talking about bombs and stuff for those applications is not relevant. You're blocking directly. The second one we see in this space is a lot of companies, big and small, are showing a lot of conceptual what could happen by saying, "Hey, you could have an agent that could poison data that could act as systems." And that is kind of an interest, because it's still a new area.

The third one is – and this is not taught as much as it is, is actually real problems that exist. Sorry. There's a fourth one here. One of them is showing proof of concept saying, "Hey, I found this thing. And, hey, I reported it. That is a problem. It gets fixed." The other one is actually these are real problems. Classic example is most people deploy LLMs of losing Ollama. Ollama, they run in pseudo mode on their Unix machines. And then all you do is just send a prompt injection with an OS command embedded inside. And guess what it returned to me? Whatever in the system itself. Classic security problem. OS command injection works in the system.

Those problems I see. Again, depends on the use case. Yes, you can do injection attacks. If it has internet access, it can call a bunch of systems. That's problematic. It can leak data because they hooked up the AI to another tool set component. Guess what? They applied basic AppSec controls. So again, it comes back to basic development practices in terms of what I'm looking for. But the way you do it is quite different. And I think all three are quite useful. We're seeing much more now – in my company, we see a lot of practical problems as we kind of prize. Actually, forget about the noise about what could happen as we think about what's actually the risk we see. But also, there's a lot of research coming out saying, "Hey, these things could happen." And it gets bridged into the concepts that get talked about and say this could be an issue, but it's actually what's happening there.

[0:28:28] Danny Allan: And that's the interesting areas, too. I always go back to you don't have to run faster than the bear. You have to run faster than your friend. And so in the early days of AppSec, people are running Nessus to scan the world. And so if you did basic controls, you could secure yourself. It's probably similar in the AI space. Not a lot of broad scanning tools. But if someone is very targeted, and there is obviously clearly targeted attacks, it's hard to defend against.

[0:28:53] Dr. Peter Garraghan: Again, it goes back to classic AppSec, which is replace apps with with app or systems and saying, "Okay, how to attack it? How to defend it? Same principle, which is if I have a budget, I have a threat model, I need to test things I need to do. [inaudible 0:29:07].

[0:29:08] Danny Allan: Well, if it's like basic AppSec – my son is in computer science, which is a whole debate in itself because he says, "Dad, am I going to get a job when I graduate?" And they say, "Yes, of course you can get a job. Computer science is not going away. There's still a discipline there." If you go back to AppSec, we, over time, said that the developers were responsible for security. Now, that could be a whole debate in itself. But over time, what we did is we put security in the hands of developers. They had to understand it and implement it. Didn't spend all their time. Most of their time was creating, not securing. But they did become accountable for it. Do you think that the accountability for LLM security and AI security will also end up in the hands of the developer? Or do you think it's going to live within the apps community for the next 5 years?

[0:29:52] Dr. Peter Garraghan: I think it's in the hands of everyone in the organisation. Support the CISOs, they will say. Yes, everyone's responsible to some extent, but also customers we work with. Yes, the security, CISO office worries a lot about this problem. They want to educate their developers. Developers have good practices in place. It makes the headache much, much smaller. People in AppSec again are responsible to make sure they're doing due diligence and testing about it. People in GRC, risk compliance, are also responsible for risks and security of the business.

With the LLM, AI space, the thing we've seen – and there tend to be two ways of thinking about this. Traditionally, a lot of the [inaudible 0:30:29] data science teams, the innovation, they kind of left alone to their own devices, and they wanted to be trained in security. They're trained to do data science and machine learning, which makes perfect sense. The problem now is that they have to kind of collect all the data in one location for the AI and ML and they're getting more attention, the security saying, "Wait a second. What's actually happening here? Why are you following these practices that we have in place?"

On the flip side, you have the security team saying, "I know nothing about AI, really. I'm trying to learn about this other space. But for me, it's slightly magic." And they're trying to apply their controls to a space that has analogies but not exactly techniques against it. Going back to your original point, is AppSec responsible? Yes. But so is every developer. I always go back down into which is replace the word AI with software. Same thing applies. Exactly.

[0:31:16] Danny Allan: Well, one of the interesting things that you just brought to mind is, typically, when AppSec started, it was just AppSec people. And in today's world, I would say, it's kind of three teams really. It's the AppSec team, the developer team, and the platform engineering, the DevOps team that kind of come together. There is a new data team, business intelligence team, the people that are actually creating the vector databases and the backends. As long as you're not using a generic hosted model, like I was talking earlier today with someone who is implementing RAG at scale and doing these massive vector databases, there is a new group in here that is part of the data set that is feeding these models as well that comes into play.

[0:31:53] Dr. Peter Garraghan: Yeah. And again, we're seeing – it's like the sieve again. Things are shaken out. We've seen anything from an entirely new AI security teams being formed in organisations to spearhead these initiatives to the day I've being told, "Figure this stuff out for me please." To the platform people saying, "You're responsible. It's in the platform." Versus saying we should do it. I suspect what happened was as best practice gets more and mature, you have a nice repeatable process. So you could say that DevOps emerged from the necessity of having specialist team looking infrastructure, then DevSecOps also emerged from it because I need security of the DevOps pipeline. If I have an ML ops pipeline makes it ML sec ops pipeline, which exists. If I have an AppSec team, I have a developer team, I have a security team, an AppsSec team, an AI AppSec team makes perfect sense.

[0:32:43] Danny Allan: We are beginning to see the emergence, or at least I am, of AI security engineers that are coming into play, especially the larger organisations. I know we're almost out of time here. I guess I have two questions, and they're the inverse of one another. I'll start with the negative one, I guess. What scares you the most about the future? If you look out, I mean, obviously you're on the cutting edge of all this and you're sitting in classrooms, universities, and educating people in this. What worries you the most about AI and the implementation?

[0:33:09] Dr. Peter Garraghan: I think the one the biggest worries that people anthropomorphize a lot and that causes a whole bunch of issues. I get it. It's captured human imagination for decades. But the idea of the automaton, the fighting machine, the golem and stuff, it kind of –

[0:33:23] Danny Allan: I don't believe in that, by the way. But anyway.

[0:33:27] Dr. Peter Garraghan: And it gets into people not treating it as what it is. It's not a piece of software which allows you to do a whole bunch of pattern matching and classification. And systems is great. But people overlook basic principles of software development and development. And these are interactions, because it can trigger. It can say please and thank you to me. It might make you think differently.

There's also societal. That's a big societal issue in terms of if we're anthropomorphicizing our interaction with human kids interaction and APIs, what does that mean? I think it's not saying it's good or bad. I'm saying it's very different. And our current ways of thinking about security may not be aligned type of thinking about this space. That's the thing that probably worries me the most, because that's not just technical problems, that's also societal consequences as well.

[0:34:10] Danny Allan: Yeah, the technology is relatively easy. It's a cultural societal implementations. What makes you most optimistic? What excites you the most about where we're going?

[0:34:20] Dr. Peter Garraghan: Yeah. I think the most exciting thing about it is some of the use cases, AI, that you see nowadays in the future is actually pretty exciting. People are using it as we speak. I see it as another pin on the journey of computer science and AI. How quickly that will accelerate remains to be seen. The law of talking about agents. And agents are interesting because they typically lend themselves towards action. And my suspicion is agents – we'll talk a little bit conceptually here. Agents aren’t a new concept. They've been around for decades for agents. AI agents are interesting because now you have this idea of late runtime binding of arbitrary use. Normally, service architecture, yes, you have different services that form together, the workflows. Now, you have natural language involved with this. You can actually build some really cool applications or use cases that we've not seen before that cause some new issues. But I suspect that sieve, you're going to have some really cool use cases that kind of form from this.

[0:35:17] Danny Allan: Is the world ready for completely agentic AI and those decisions being made by AI? Do you think, culturally, we're ready? That's not a technical question. Do you think, culturally, we're ready?

[0:35:29] Dr. Peter Garraghan: I think we're ready. Again, I know [inaudible 0:35:31], which is we've been using software to automate the decision-making for a long, long time. And we have processes in place already. I know we're running out of time. But I once had a politician. I won't say which country and where it is. But completely sincerely, I think putting them on nuclear bomb. Completely terrified. I said to them, "Well –" let's take a step back. Okay? Even that ever happened, we already have protocols in place for interacting with other systems in terms of interaction of – a whole bunch of protocol that exists in defense military already. With the use of agentic, we already have controls in place. You should be allowing this for things. What's not going to happen is pull a switch happen system. You shouldn't be doing that because, A, that's not how you build software or build use cases. And, b, it's probably wasting you massive time already.

I think the world will be because in the agentic – again, using soft AI agents to make actions, it'll be very good at certain use cases, certain contexts. Not all of the. Be able to work in some great ones, and I think that'll be really interesting. I'm not going to see – I don't think, in the next five, six years, you're going to see everything purely agentic because there's so many edge cases of it not being perfect. But what it can be is support me in very specific activities I think is super useful.

[0:36:44] Danny Allan: Yeah, it makes total sense. You scare me a little bit when we talk about AI on a nuclear bomb, but there are many, many use cases where there's far less risk, and there's so much value to be gained by putting it on. Autonomous cars is probably a good example of that. I mean, there's a lot of –

[0:37:01] Dr. Peter Garraghan: We see autonomous cars all over the place in San Francisco, for example, nowadays, and it's actually working. Again, innovation stays kind of sometimes predicted mostly to be the use cases that form that are successful kind of we didn't think about it necessarily into actually proofs in the pudding.

[0:37:15] Danny Allan: Yeah, that's true. Well, Dr. Peter Garraghan, it's been awesome to have you on. Thank you for joining The Secure Developer. And I look forward to seeing some of these things come to fruition. And best of luck in building your company. I think it's super important.

[0:37:28] Dr. Peter Garraghan: Okay. Thanks.

[0:37:30] Danny Allan: All right. Thank you, everyone. And thank you for joining us for another episode of The Secure Developer. We'll see you next time. Thanks.

[OUTRO]

[0:37:39] Guy Podjarny: Thanks for tuning in to The Secure Developer, brought to you by Snyk. We hope this episode gave you new insights and strategies to help you champion security in your organisation. If you like these conversations, please leave us a review on iTunes, Spotify, or wherever you get your podcasts, and share the episode with fellow security leaders who might benefit from our discussions. We'd love to hear your recommendations for future guests, topics, or any feedback you might have to help us get better. Please contact us by connecting with us on LinkedIn under our Snyk account or by emailing us at thesecuredev@snyk.io. That's it for now. I hope you join us for the next one.