Build, Break, And Defend With Mohan Yelnadu

Mohan Yelnadu: “Security is everybody's responsibility. Unless everyone is aware of this, things are very challenging otherwise. The culture should enable being challenged in its DNA. One should not focus on the end result, but if you are able to focus on the process, you are definitely going to end up with a better result. Irrespective of which nation that you come from, what domain, you can actually achieve continuous security. The only thing is you need to take baby steps, but you need to take it continuously.”

[INTRODUCTION]

[0:00:33] Guy Podjarny: Hi, I'm Guy Podjarny, CEO and Co-Founder of Snyk. You're listening to The Secure Developer, a podcast about security for developers covering security tools and practices you can and should adopt into your development workflow. It is a part of the Secure Developer community. Check out thesecuredeveloper.com for great talks and content about developer security, and to ask questions, and share your knowledge. The Secure Developer is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market. For more information, visit heavybit.com.

[EPISODE]

[0:01:07] Guy Podjarny: Hello, everybody. Thanks for tuning back into The Secure Developer. Today we have a great guest from Singapore, back in the Asian side of the world, Mohan Yelnadu runs the application security part of Prudential. Mohan, thanks for coming on the show.

[0:01:23] Mohan Yelnadu: Pleasure's mine.

[0:01:23] Guy Podjarny: So, Mohan, before we dig into a whole bunch of topics of conversation, tell us a little bit about yourself. What do you do, and how did you get into security in the first place?

[0:01:33] Mohan Yelnadu: It started with building. I was primarily working at a university teaching application development, primarily for what are postgraduate engineers. Then I joined a startup company that I actually started from bottom layer and reached the application layer, where I defined kernels, I developed drivers, and then I went the application layer. That's how I started working in mobile applications in the beginning and on the web application. I've been a builder first, then I moved to testing. I became a “breaker,” I would say, and then eventually I moved into the pentesting world where I was a “true breaker.”

Eventually, I realised being in pentesting, the sphere of influence is very limited to that application that I was working with. But if I have to make an impact, I have to move slightly towards the DevOps side, and that's how I moved into DevSecOps. Now, I'm a defender, so builder, breaker, and defender now.

[0:02:47] Guy Podjarny: That's a great journey actually, because you come from – oftentimes we try to draw analogy in security to the world of quality, and you've literally done that journey. Going from assessing quality to assessing security to now building security. What is it that you do today? Tell us a little bit about AppSec and what's your role today?

[0:03:08] Mohan Yelnadu: Okay. At Prudential at the group level, we have different local business units and each business unit because they're based in different countries, there are local security compliances that we need to follow apart from the regional, at group level. So, we basically have recently kickstarted the DevSecOps program and we have deployed the fourth primary area from the software security through SAST, static application security testing. Then we have open-source software security, and third is the container security, and the final is they do DAST, dynamic application security.

We have chosen different tools and working with development teams to onboard their applications. We have just kickstarted the onboarding process, so eventually we plan to go to different local business units and give them workshops on how to use these tools. Then at the same time, we are slowly building something called an ecosystem because we are very few in number in the AppSec team. We basically plan to create this ecosystem which basically fills the gap between what developers want and what we can actually support, being small in number.

We are planning to put how to use different tools. One of the frequently asked questions in terms of vulnerabilities and how to fix those vulnerabilities, then some small five to seven-minute videos about how to use the tools, how to fix those tools, and primarily one or two major technologies, and then eventually the questions can come from different areas. We will basically try to help them with some with some Jira to track and close and guide them.

[0:04:54] Guy Podjarny: Got it. That sounds like you're thinking scale, so you're building the AppSec team to build supporting – do you call the team the AppSec group or is it the DevSecOps? What's the identity here? How do you call yourself?

[0:05:12] Mohan Yelnadu: Yes. We have a family. Within AppSec, two teams. One is the engineering team, another is the operations team. The engineering team basically does all of the R&D part, and the reviews, and latest test tools, scripts, and any process improvements. They do it in a limited environment, and then the operations team basically does the rollouts and education and tracking different issues around day-to-day life. The daily activities.

[0:05:39] Guy Podjarny: But both of these groups support the engineering teams in doing it, or do you mean – so, the engineering side is the one writing code or reviewing code, but really a lot of it is about working with – it's not about the codes they build, it's about the codes the engineers build and keeping it secure? Is that a safe assessment?

[0:05:57] Mohan Yelnadu: Yes.

[0:05:58] Guy Podjarny: Or are they actually building solutions and maintaining services?

[0:06:00] Mohan Yelnadu: Yes. We in the application security team, we don't develop any applications, but we do have integration code. Writing playbooks for installing automated installation of various scripts, two sets of terms of, you know, maybe develop that part of integration code, but otherwise there are no applications that business into this. That's a complete development activity. We basically support that.

[0:06:28] Guy Podjarny: Got it. No, that makes a lot of sense. You talked about the local businesses, but then it sounds like the initiative, this DevSecOps initiative is not local. It's a global initiative led by a local group, and then expanding into the other different Prudential subsidiaries or local businesses around the world?

[0:06:46] Mohan Yelnadu: Yes. That's right. At the regional level, we basically have developed the whole concept and set up different tools at a central level in the cloud. We primarily use cloud, and then we have different local business units, which use this cloud through their own tenant subscriptions. These development teams actually, primarily, the earlier focus was on the infrastructure security, and the very less focus was given on the application. Although, the real pentesting as part of compliance and other important tests exist. But AppSec, as such was not a priority, basically. So, we started with this DevSecOps programs, more focused attention towards the application security.

[0:07:34] Guy Podjarny: Then you drive. You build that expertise locally and then you expanded and permeated out to others.

[0:07:40] Mohan Yelnadu: Yes, exactly. We have our extended hands in the local business units through IT SMs, IT security managers. Some of the IT SMs are really very good. Some are good in other security areas, so we basically tried to balance this out, and we helped them build additional skillsets and we work with local government teams.

[0:08:04] Guy Podjarny: Got it. Cool. It's always probably both a blessing and a curse to be the trendsetter, to be the one driving and mobilising the organisation. You got to shape a new future, but you probably have to battle some blockers. Do you find – as you engage, it sounds like you're in an interesting position, where you're thinking ahead, you’re thinking about security needs to look over time, today and over time. You have these two groups you need to convince or bring along for the journey.

On one side you have the development teams that need to be signed up to this, but on the other side you also have your peer or other security teams in different regions that need to do it. Do you find people are resistant to this change or they embrace it? What's your view of the overall –

[0:08:50] Mohan Yelnadu: Interesting question. What I've read and what I've come across is slightly different. Normally, when I read articles and blogs, I get a feeling that developers are trying to register the security part of it. But in here I found developers do appreciate the importance of security. Only thing is if it is transparent to them, they love it, because that's not slowing down the process to production. But if it is slowing down, they want to find better ways to manage this. I've read another story that people tried to bypass it, but somehow, I don't see that being here.

In a way, I would say I'm lucky that the developers do understand the importance of deploying secure software. But sometimes a business’ timeline to take their products to market forces them to do it as quickly as possible, but that time you have to balance out how much you can put in production, and how much security in depth that you can reach.

Sometimes it can go in a tricky way, I would say. But again, it's not the developer’s fault. It's basically the whole ecosystem around that is pushing them to deliver faster. But as an AppSec consultant, my job is basically to simplify this friction of taking quicker and doing it securely. My job is basically to simplify this and DevSecOps program basically, tries to address this. So, if I'm able to give the developers feedback early in the lifecycle and a constant and transparent one. These three things if I am able to give to the DevSecOps program, I think developers would love to invest in security and have a different feeling that we are not enablers, but blockers.

[0:10:52] Guy Podjarny: Yes. I love that. Spoken like a true partner. I think, really, at the end of the day I think it all starts with people, and I think probably the worst way to approach it is with an adversarial, it's us versus them. Really, you have to understand the context of a developer. They're being demanded to deliver function and speed and the likes, and you need to roll with that and you need to adapt to their surroundings. So maybe, let's get a little bit deeper into detail. How is it that you engage with the developers? I know there are security champion programs. You mentioned a little bit about automations. How do you work your team with the dev side?

[0:11:31] Mohan Yelnadu: So, there are multiple areas in which we need to focus. It comes to onboarding developers to do secure development. One is definitely the tools and early feedback. If you have automated pipelines to whatever orchestration tool or platform that you use, early feedback gives them a sense of where they stand in terms of the application security posture. But at the same time, that is not enough. You need to have a process around supporting them, as you rightly said, being partners actually.

So, that's where we need to build a good relationship across different development teams from different regions, and that's what I'm trying to target. My main challenge in this process is basically the people. If we had enough number of AppSec folks on the team, it actually helps building that relationship faster and we can deliver the requests earlier. But having said that, we know what our challenge is in terms of getting a good number of people in the AppSec domain. We need to go for the third option, basically, the ecosystem. So first, tools. Second, the process, and then the third part is the ecosystem. That's how I prefer to build that whole relationship.

Ecosystem basically, helps the progress without the AppSec consultant’s availability, because that's the CISOs these days. Since we have hundreds of developers spread across different local business units, building an ecosystem basically helps them reach their destination faster. Say, for example, look at the static application security testing tool. Take any tool in fact. I have, in fact, designed and developed part of the, one of the leading secure code analysis to compiler. I know that domain quite closely. I know that this domain, although it has been there since three years or more. But inherently, the core problem of this type of code analysis, either for quality or for security, is the false positives. Because you cannot develop models to understand the technology, because there's a variety of ways in which developers can use the technology to deliver business value.

But in such cases, a false positive is an obvious by-product, and more than a number of false positives, less the faith the developers have in their tools. It's very important to iron out this part very quickly and in a smooth manner. As part of the ecosystem, we plan to build this stuff sure, a 100% false positive list coming from a particular application for a particular tool. Then, I want to say, “Look at your result and look at this table.” If you see these issues in your application straight away, go ahead, and monitor the false positives. You don't have to worry about fixing it. We will not break your build in any case.

So, that's one level up, resolving or reducing the number of issues to be resolved. Secondary, is basically, sometimes there are possibilities with a particular situation. For example, there's a CVE reported on a library, for example, an open-source library. Then there's a caveat saying, “Hey, if you use this API in this particular library, then only the CVEs are applicable. Otherwise, you're not.” If I have a dependency and then there is a transitive dependency on it. If I'm not using this API from that transitive dependency, I am not required to fix this issue. Basically, I will not believe it.

In such a case, we need to build that confidence of the development teams that, “We have to analyse this CVE. This doesn't apply to you. Don't worry on this.” But at the same time, because it's a robust organisation, there will be another thing which would be affected by the same CVE, but they are vulnerable, their application is vulnerable. So, we need to balance this out, and this has to be done on a quick basis because the more longer you take time to resolve this clarification, you are actually going away from the development teams.

This is very crucial. This analysis for any new CVE, because imagine the severity of these vulnerabilities on the higher side to just drop and leave aside everything and then focus on this analysis, and clarify to both types of development teams. “Hey, for you, this doesn't apply. For you it does apply. Okay, you better fix it.” Or you have some other problems in placement, or some kind of monitoring.

[0:16:25] Guy Podjarny: Right, another mitigation technique.

[0:16:27] Mohan Yelnadu: Exactly, yes. So, that's what we mean by creating the ecosystem around them, so that there's less friction, basically.

[0:16:35] Guy Podjarny: Yes. It makes a lot of sense. I think there is a lot of conversations in the security world today that goes from maybe a perfectionist perspective, that says, “Well, I want to report and I want to tell new developers about all the problems, and you should fix all the problems,” to being much more diligent and understanding that there is only so much time and so much attention that you can demand of developers for security issues. You have to use that time wisely and ensure that you really invest in using their time well in avoiding issues. Sounds like a really good thinking.

[0:17:12] Mohan Yelnadu: Absolutely. Yes. The extension of DevSecOps program through some kind of mechanism where they get feedback, and at the same time there is something which controls the whole delivery to ensure there is security in production, which is to build better programs. So, build better programs, I think you test up on that part. It has to be implemented in a manner in which it is pragmatic, so every tool should give different types of vulnerabilities, like some may give critical, high, medium, low. Some will give threat level 10, nine, eight, seven and so on.

When we target these breaking-the-build program, once they're comfortable with the whole ecosystem, we need to be considerate, as you rightly say, of their available time to fix these issues. Developers are not there just to fix issues. They are there to deliver business features. So, we have to balance this act very carefully. The way we plan to deliver this is maybe in zero to six months, allow them to scan, get the feedback, and have a clear understanding.

But at the end of the third month, you basically would like to tell them, “Your build is going to be broken for, if not all the criticals, maybe some core criticals.” For example, a SQL injections. Maybe process the thing or any other data injection issues. That gives them enough time, basically, to look at a subset and then focus on fixing it actually. Then eventually, after maybe nine months from the start, you basically come down and say, “Hey, no criticals allowed now. You had this much time to fix it. Now, no criticals.”

Then, eventually in the next step you would say, “Now that you're focused on critical issues, now that you're done. This was down, and say no high issues.” And so on. We give them time enough to get comfortable with the whole ecosystem of DevSecOps tools and scanning, et cetera, and then we give them, “Hey, you start from the top code,” and then slowly increase it so that they have enough time for both doing to delivering business features as well as fixing the issues.

[0:19:23] Guy Podjarny: Yes. Sounds like a good way to roll out. You have to stagger it. You can't just come in and be a bull in a china shop and just break everything. You have to be wise about it. I do feel in the world of open-source defences and the CVEs, that another practice that I'm fond of is this separation between a new vulnerability that comes along and a change. This notion of establishing a gaze that is more forward-looking, so you have all these vulnerabilities right now, but can I break the build in whatever fashion that, whether it's in a pull request or in a build, or whatever it is. Only on a new vulnerability that you've added, because you've modified the co-changes, so you don't necessarily just go backwards. But it's also a stop the bleeding and draw a line over here, and then onwards.

I think, Mohan, you've already touched on a bunch of those. But I want to make sure we don't miss them because I like that framework. You've written about this notion of the six principles of continuous security, and I liked the way it was organised. Maybe what we can do is we can go through them and see – I think some of these we already elaborated and talked through them. Does that sound good?

[0:20:33] Mohan Yelnadu: Yes. I would love to do that, in fact. So, this was done for one of the conferences in Europe that I visited. These six principles primarily focus was one clear message, that irrespective of the passion or the budget that you have, you can have continuous security. Basically, it's achievable for all the organisation, irrespective of the site, et cetera. You can still achieve continuous security.

The first principle basically talks about, “I code I predict.” Which basically means, “I'm developing something as the developer. It's my duty to have some security protection and develop that as part of the whole module development.” When I say that at the same time I'm saying, “As a developer I should not expect another layer, or a higher layer to protect my application.” For example, DAST. For example, real-time application self-protection or RASP. I shouldn't expect these controls to protect me. As a developer, I should have any code that I put in production should have both the business value, as well as the security built around that.

That's the first principle. The second one is basically, security is everybody's responsibility. So, be it a developer, it's not just the developer's responsibility. It is the responsibility of the scrum master. It's the responsibility of the product owner. It's the responsibility of the people in the leadership as well as people in the marketing area as well. Unless everyone is aware of this, things are very challenging otherwise. Leaving security only to development teams, it's a walking disaster. Because people, I mean, the hackers have found different ways to get into the network, through maybe receptions of e-mail, or [inaudible 0:22:36], actually. Or through some other ways within the ecosystem, either through the application layer, or the [inaudible 0:22:46], or the perimeters. So, there are multiple ways. Unless everyone takes responsibility of not having enough awareness about security, things will get really difficult.

[0:22:58] Guy Podjarny: I like how that one very much complements the first one. Our VP of sales has a phrase that says, “Accidents happen at intersections,” which, I kind of like that phrase for the context of security, just as we discussed. Everybody has to take this, and just like developers shouldn't expect somebody else to do it. That someone else also shouldn't expect them to just be perfect at what they do. It’s everyone's responsibility. Love it. So, we've got one and two, let's go on to number three.

[0:23:28] Mohan Yelnadu: Sure. Security through automation, simplified security through automation. I love this principle because these days, everyone expects to deliver whatever in the shortest time. We came from waterfall background to DevOps, CI/CD, et cetera. The prime factor here is everyone is trying to reduce what is going to production. Instead of 10 features, I'm actually developing the MVP and then adding one or two features, whatever they lead, or enhancing those features in the subsequent releases.

When they are simplifying from the business feature side, I think it is our duty on the security side to simplify whatever security process or tooling, or any other procedure through automation. Automation can be brought out at different levels. I see primarily three areas. One is automation through onboarding your application in your DevSecOps program. If it is smoother, people start adopting it quickly.

The second thing, the false positive analysis, or the tries process. If you are able to automate this through either configuring the tool itself or having the mechanism where they can address these issues. The third is the metric submission. So, metric submission, if you are able to automate, that gives them a very good feedback in terms of where they stand from different dimensions of security. The static analysis of the code that you develop, the code that you use, then the containers that you use, the ecosystem that you use. Kubernetes or so, as well as, a number of the inputs coming from manual pentesting.

So, if you are able to put these together, in one single pane of glass. The point is if you have security dashboard, the developers in one shot get very quick feedback and then they can analyse, “Hey, last series we had this much vulnerabilities in these areas, now we are progressing way ahead because it shows the graph is not going downwards.” But if by chance, it is going up, there could be multiple reasons. There’s increased code, new features, multiple new features. In that case, they can get a sense of how they have to act upon, if the trend is going in an upward direction.

They can think of all this only when they are aware of this, and to be aware, you need to have a single dashboard given to them, in a manner in which they can consume it. The automation in this area is something that I really feel boost the overall security posture of the applications across the group.

Then the fourth one, build ecosystem and build confidence. This I always press upon. You can build an ecosystem around many different areas, because the number of AppSec consultants in their field is not proportionate to what you'd expect, basically. When you are not there, the offline developers should be able to get their issues resolved and their issues addressed. If you are building an ecosystem of documentation to smaller videos about how to use the tool, or how to address a particular vulnerability, or frequently asked questions about the possible list of issues coming from different tools.

This is what I mean by the ecosystem. The moment I have found personally that say, for example, in the previous organisation I started, when I started building the ecosystem and all the people I observed, for almost two years. Developers started using the whole ecosystem and then the feedback coming from tools, eventually they feel more comfortable with delivering security inside the business features, and they feel more confident about their deliveries.

[0:27:35] Guy Podjarny: They grew, right? You taught them. Also, they’re better at their craft right now. They're able to do it. They're happy. You're giving them something back, right?

[0:27:45] Mohan Yelnadu: Absolutely. In fact, that's quite gratifying, by the way. When you see that confidence in the developers' eyes, more than anything, I would love to see that.

Moving on to the next one, get feedback as soon as possible from all sources. We step on it as well. Coming from those four areas which I talked about, the static, dynamic area, container security, and the OS scanning. In fact, I like to try to integrate other areas, and I'm planning to do that. One is the back rooms. If you are able to get a walkthrough set for applications, you get more visibility in terms of feedback about that and what attacks are happening on your application. The vast consultation, the SOC, the security office centre. SOC people would be able to give us some insight in terms of, “Hey, in this period of time, the attacks are primarily focused on a small injection.” Or, “In this period of time, the attack was primarily to the network and not on the application layer.”

These kinds of insights, basically, ought to be there to grow the understanding of how the hackers work and how your application should be secure. The feedback coming from different dimensions actually helps developers mature for the future.

[0:29:12] Guy Podjarny: Yes. Knowing if what you're doing is any good, you have to be able to measure. If you can't measure it you can't optimise it. You have to improve it.

[0:29:20] Mohan Yelnadu: Absolutely. The last one is the focus on process improvement. That is a continuous process. It's not like you have a DevSecOps program and then you're done. It's never like that. The process is very crucial because I have seen, if you focus towards product, the end product, you would be able to achieve some significant value, but you will not be able to reproduce successfully in a longer period in a timely manner. But if you focus on improving the process itself, that is making it light and making it adaptable, making room to improve by itself through some other feedback or other mechanisms.

If you focus on the process, the quality of that end product becomes a secondary factor. You can predictably make it successful with every delivery if you focus on the process. I lacked one tool from this lady, Dr. Carol Dweck, maybe you should have a session with her on growth mindset. Really, although her domain was completely different when she talked about the growth mindset, but I love that idea and I tend to apply that in the process improvement. The process can live in our domain. It can be applied to data science, it can be applied to security, it can be applied to business processes.

So, what she says is, “You should embrace failure in the journey.” That means don't point fingers when something bad happens. Instead you say, “Hey, this is an opportunity to learn.” Embrace failures and think it as a stepping stone and improvise the whole thing.

Then inculcate the desire to be challenged. Now, this is very important. There is a limit of culture involved in this, but this culture can be worked upon or repeated over time. In certain cultures, hierarchy is very important, where you cannot challenge your superior.

[0:31:34] Guy Podjarny: Yes. Your boss or your managers.

[0:31:35] Mohan Yelnadu: Exactly, yes. But I've found – because I was lucky to work in different cultures so far in the US, Australia, Europe, and now in Asia. I found some of the European cultures are really nice in this respect. Where they say, “I may be the CEO, but if you are a developer, and you have some suggestions, come and share it with me.” And then that feedback goes into improving the process. The culture should enable being challenged as a feature in its DNA.

Irrespective of where you stand, if you're able to give good feedback by challenging whatever existing, I think, that's the best thing that can happen to improve the process. She says, “Believe in process." That was what I was talking about. So, one should not focus on the end result, but if you are able to focus on the process, you are definitely going to end up with a better result.

[0:32:38] Guy Podjarny: Really good principles. I love how, again, they touch one another but a lot of them are about ownership. A lot of it is around simplification and making it easy. If security is too hard, people won't do it. You have to make it easy. Then, I think with the whole last few principles in general, in particular, but also the broader set. A lot of conversation around continuous improvement, “You're never done and that's a good thing.” Embrace that you're never done versus being – but you also need to have the mindset and the perspective for it, but you also need to staff accordingly. It's not a temporary project that you're going to hire some people for. You have to understand that the people here that need to be constant service providers and enablers, and be on top of the next technologies, threats, and the next evolutions, and just changes in your own work, and what doesn't, doesn’t work for it, and adapting that.

[0:33:30] Mohan Yelnadu: That's true.

[0:33:30] Guy Podjarny: I love the principles, Mohan. Before I let you go here, I like to ask every guest coming on the show. If you have – this is a wealth of information, but if you have one bit of advice, one either pet peeve about things you're seeing people do, that you think they should just stop doing. Or one thing that if you embrace this, you're levelling up your security, whether it's on the dev side or on the security side or organisational. What would that bit of advice be?

[0:33:59] Mohan Yelnadu: I have two advice. Not one, but two.

[0:34:01] Guy Podjarny: That's okay.

[0:34:04] Mohan Yelnadu: One thing, the top leaders, maybe can be the CIO or CISO or CTO, from them, if the security message goes down, it is easy to percolate. Because you get funding, you get processes resolved, you get mandates. So, all these things are very conditional if it comes from the top. The second aspect, the second tip is basically irrespective of which nation that you come from, what domain, you're not the banking or financial institution to work on security very seriously. You can be coming from any different domain of industry. You can actually achieve continuous security, only thing is you need to take baby steps, but you need to take it continuously and you will definitely achieve continuous security, for sure.

[0:34:59] Guy Podjarny: That's great advice. Get going. Get started. It matters to everybody. Everybody can do it and you don't have to start from switching from nothing to perfect, just make a step in the right direction. Mohan, this was great. Thanks a lot for coming on the show.

[0:35:15] Mohan Yelnadu: My pleasure.

[0:35:17] Guy Podjarny: Thanks everybody, for tuning in. I hope you join us for the next one.

[OUTRO]

[0:35:23] Guy Podjarny: That's all we have time for today. If you'd like to come on as a guest on this show, or get involved in this community, find us at the thesecuredeveloper.com, or on Twitter, @thesecuredev. Visit heavybit.com to find additional episodes, full transcriptions, and other great podcasts. See you next time.