Level Up Your Security Champions With Yashvier Kosaraju

[0:01:12.9] Guy Podjarny: Hello everyone, welcome back to The Secure Developer. Today, we have with us, a guest, who is really leading with one of the some of the more advanced technology companies and sort of forerunners I would say in the world of kind of the modern unicorns in Twilio, which is Yashvier Kosaraju who heads the product security group at Twilio. Welcome to the show Yash, thanks for coming on.

[0:01:33.5] Yashvier Kosaraju: Thank you.

[0:01:33.7] Guy Podjarny: Yash, before we dig in, you know, I talked about Twilio but really, your security journey goes further back and has those — tell us a little bit about how you got into security and kind of what is it that you do, the path and what is it that you do today.

[0:01:46.0] Yashvier Kosaraju: My path is pretty unconventional as well, I guess we can security a lot of people have unconventional paths into it. So I did my bachelors in electronics, never really liked semiconductors that much, but we had a couple of electives and I got in to like crypto and that choice sort of spoke to me so after that, I managed to get into Johns Hopkins for a masters in cyber security and that’s sort of what kicked off my security career. After that I did some some consulting for a while and then worked in the product security team at Box and then now here I am at Twilio.

[0:02:21.3] Guy Podjarny: Throughout that time, when you were working on security engineering before Box, you were sort of a bit more, was that more of a consulting type engagement from different companies before you went to work within a company.

[0:02:33.1] Yashvier Kosaraju: Yeah, I was consulting from a little over a year with iSEC Partners. Mostly in the Bay Area, a lot of good companies, a lot of good exposures seeing how different people did things, mostly penetration testing and model sort of engagement. But we’re all pretty good exposure straight off college, consulting and then getting to work with many companies.

[0:02:52.9] Guy Podjarny: What made you take the plunge and move from a consulting to a full-time job at box?

[0:02:57.8] Yashvier Kosaraju: Finally it was the commute in the Bay Area. If you know what I mean, it's a horrendous commute, it was taking me two hours every day, back and forth and at some time, I didn’t want to do that anymore.

[0:03:11.3] Guy Podjarny: Yeah, that’s actually a very good reason, there was all these studies that you’ll how – one of the things that correlates best sort of happiness in life is the length of your commute. Well I guess we’re all not suffering from that problem specifically at the moment. There’s something of a ‘glass half full’ aspect to it. At Twilio, did you sort of jump straight into the management role, sort of the head or did you start off more of as an individual contributor?

[0:03:35.2] Yashvier Kosaraju: I started off as an IC and before joining, I was all this sort of debating, do I want to go down the management route, do I want to stay as an IC. So when I joined Twilio, I asked my boss if there would be potential management opportunities in the future and would I be considered and he sort of said, "Yes, we’re growing at a good rate, if we keep growing, there will be opportunities." And again, it was a luck factor as well where I joined as an IC and then an opportunity came, I was considered and being fit enough to take on that role. Here I am after that journey.

[0:04:08.7] Guy Podjarny: Do you like management better than the individual – what would you say the pros and cons of making that shift, you know? If we’re talking about the previous plunge.

[0:04:16.0] Yashvier Kosaraju: I actually like it, when you’re an IC, you kind of get happiness from your contribution to a project, but when you're managing a team you get the opportunity to help them grow and I get happiness in the projects that my whole team completes, so it’s my project versus all the projects that five people are doing. So it’s 5x the work that’s been done and it’s oddly satisfying to just look at a team and see where they’re going from day one to maybe end of the year and see how much can be achieved and how you can help them sort of mitigate roadblocks and achieve their goals they set up and accomplish. 

[0:04:56.8] Guy Podjarny: Spoken like a great manager, you know, that’s a great attitude for someone in a management position. I guess let’s expand a little bit before we dig in to your space, you run the product security group, can you give us a bit of context of the broader security organization, at Twilio and a little bit of how responsibilities are divided?

[0:05:14.6] Yashvier Kosaraju: Sure, I run the product security team then we have a cloud security team, we have an enterprise security team as well then there’s the whole trip management group which has certain diligence and role management, then there is other team which is security engineering that builds sort of large scale platforms which will be used by multiple security teams.

Individual teams do a bunch of automation too but this team is a core group of security developers, building large scale platforms. That is one side of the team then there’s compliance, customer trust, physical security and a whole bunch of sort of other teams within the security and trust organization.

[0:05:56.4] Guy Podjarny: That’s an interesting, this was a couple of divisions I’d love to dig in to that’s kind of everybody’s struggling I guess with that split of responsibilities. One is maybe first of all, cloud security. You do product security then there’s cloud security. How do you divide responsibilities between those two groups?

[0:06:12.0] Yashvier Kosaraju: We work pretty closely together but if you had to draw a line in the sand and say this was product security and this is cloud security, the way we try to define it is the code, you look at it from like the code deployment perspective, starting from writing the code to building an artifact is product security and then the infrastructure on which these artifacts run is cloud security.

[0:06:37.3] Guy Podjarny: So containers and I guess that would fall in the gray area or like, how much gray is there between those two? Which you know, you did say you collaborated a lot but – 

[0:06:49.1] Yashvier Kosaraju: That’s sort of spot on where we collaborate a lot like containers and Kubernetes and sort of a hybrid of product and cloud and especially if you're talking about EKS where it’s Kubernetes but it’s backed by AWS permission modeling. That’s a super grey area so that’s one big space that we work very closely together.

[0:07:09.8] Guy Podjarny: Yeah, I think that’s indeed one of the primary grey areas these days. The other split that I think is an interesting one is this sort of product security or other security aspects versus a dedicated security engineering team. Do you already know how that split works?

[0:07:24.5] Yashvier Kosaraju: That’s a great question, it’s a fairly new team that we are forming and we haven’t figured out the complete nuances of it yet.

[0:07:31.9] Guy Podjarny: Okay, it’s interesting, you know, would love to check back in and sort of you know, hear what worked and it doesn’t over there, those fronts. You know, this is the security organization and that’s sort of within I guess reporting to the CISO, I’d imagine. But you also run a security champion’s program, right? Tell us a little bit about that, you know, what’s involved there.

[0:07:51.3] Yashvier Kosaraju: Sure, product security specifically works very closely with engineering. We look at it as a partnership of sorts that we really don’t want security to be a roadblock or sort of a hammer where we say, "Hey do this". It's more of a partnership where we work with engineering and sort of come to collaborative decisions, saying, "This is the right way forward for the company." 

The Security Champions program is something that helps us do that on a large scale, the idea of the program is an engineer on every engineering team is their dedicated security champion and then there’s one security engineer who is their dedicated security partner. This security partner and champion meet on a regular basis and then talk through what the engineering team is doing, what the security team is doing, what reviews they need, what help security can provide, do things securely and make sure that they’re building robust and secure solutions. 

What helps in this sort of format is all the reviews are scheduled on a timely basis. Otherwise, the other sort of side of it is engineers build the whole product and once they’re ready to deploy, somebody says, "Hey, let’s do a security review," and they are blocked for you don’t know how many days or weeks it can be based on the project size. But the champions program, these reviews sort of happen in tandem as the engineering team is building and you can get real time feedback and it’s easy for them to make changes, there’s way less friction and we don’t affect their timelines. At the same time, we help them sort of do things the right way.

[0:09:22.2] Guy Podjarny: Yeah, and secure. I’m curious a little bit about some ratios here. You mentioned that the security champion is aligned to someone in the product security team, what’s the rough ration you aim for in terms of product security? How many security champions does every product security person work with?

[0:09:37.9] Yashvier Kosaraju: That’s something that I’ve been debating with myself for a long time, like what’s property or the ratio of your security engineers to – engineers within the company. I don’t have a good answer but what we’re trying to do is instead of talk about the numbers, talk about capabilities within the product security team and also in terms of champions, look at how big or significant the product or product growth that these ensuring teams belong to and then sort of divide it based on that. It would be like an engineer has five products under them or four products under them based on the size of the products. 

[0:10:16.1] Guy Podjarny: Yeah, that makes a lot of sense you know, it’s still the mission orientation that product security person is still more mindful and aware of the substance and the surrounding context, you know, when a request comes in, indeed some products are big, some are small, some require – some are more hairy, you know, from a security perspective and others.

The security champions within the teams is this an official role like do their managers expect them to spend some 20% of their time on security or is it more than just general interest and how do they manage that extra responsibility? 

[0:10:47.2] Yashvier Kosaraju: It is an official role, the percent of time that they spend on security is team specific, we don’t like mandate a percentage of time that champions spend doing security so what we expect them to do is sort of meet with their security partners regularly, talk about their team security posture and sort of work with their partner to sort of improve their product security posture over time.

[0:11:13.3] Guy Podjarny: Yeah, that’s awesome, are the security champions themselves a community within Twilio, do they also engage with one another or is it like specific education for them? Is it more about the one to one relationships?

[0:11:24.2] Yashvier Kosaraju: It’s both, if you do have one to one relations and like this year, we wanted to expand on that security champions as a community do happy hours and a lot of other social events but you know, COVID-19 happened and every one is stuck at home so that isn’t happening. The other thing you mentioned was training.

Within the champions program, we have a self-program called the advanced champions program, where we have red, blue and purple categories, which are like challenges, requirements to do ‘XYZ’ things which are aimed at offensive for the red offensive for blue and purple, we kind of modeled it as like a cloud security subspace within the program.

As they do these, we kind of give them more perks and responsibilities that otherwise they wouldn’t haven't had and the ideology behind it is you take these trainings, you do these challenges, you level up your security knowledge and then we as a security team will trust you to make these decisions which now you have enough knowledge to make the right decision for.

[0:12:25.5] Guy Podjarny: Yeah, that sounds awesome. That’s a great thing to do and I must imagine that these things are sort of acknowledged as a skillset and a learning by different management teams and might help them as they sort of move up the ranks or as they might kind of move into security, right? In some cases.

[0:12:39.4] Yashvier Kosaraju: Yep that is true and this ties back into how much time a manager expects their champion to spend doing security work. Once they start their journey on these different tracks, they generally put it into their sprints and work on it as a story ticket, something of general work item rather than on the side and most of the managers are happy that people on their team are interested in this and spend time learning security.

[0:13:08.0] Guy Podjarny: So we now understand I think the world around you and that digging a little bit into the work of product security itself so this team, what are the types of projects or initiatives that typically get covered by the team? 

[0:13:21.6] Yashvier Kosaraju: Sure, so we cover most of the SSDLC from secure developer training all the way to putting tools in the CICD pipeline. A good chunk of my team’s time is working with the champions to see what they’re working on, what reviews they need and making sure they get done in a timely manner. The other part they do is a lot of these projects and we are an automation-first team and most of the tools that we work on the end goal is to enable them in the build pipeline. 

And also something that we have been discussing internally is to have a self-service capability for most of these tools. The idea of being an engineer should not have to release a build if they want to see what security vulnerabilities are present in their code. So there should be a self-service capability and the novel way of approaching this is sort of having a CLI, which fronts all of our tools, because we cannot expect developers to learn the UI for 10 different tools we have. If it is easy enough for them to use say security CLI, this is my code, tell me everything I need to fix. That is the simplest form of user experience we can work towards and that is what we are trying to do. 

[0:14:38.8] Guy Podjarny: That is really interesting. So it is a specific kit I guess the sort of CLI that is attuned to Twilio and that the developers know how to use and behind the scenes it might invoke different type solutions and those I’d imagine some of those are home grown and some of those are commercial, off the shelf. 

[0:14:55.0] Yashvier Kosaraju: Yes. 

[0:14:55.9] Guy Podjarny: Or open source projects. 

[0:14:57.4] Yashvier Kosaraju: Yes. So when we look at integrating a tool both commercial and open source, we make sure they have API support like extensive API support that we can use to enable the CLI to talk on the back end. 

[0:15:09.2] Guy Podjarny: That sounds very smart and a way to lower the effort I guess on behalf of the developer, so that it is easy for them to more things. How do you think about developer experience for this? How do you see it as a product organization? How do you improve that you’re suddenly a tools provider into that organization, how do you assess yourself? How do you design in the first place a tool that developers would actually want? 

[0:15:33.2] Yashvier Kosaraju: I think that goes back to a very basic question of what would make an engineer use this tool for close security vulnerabilities. The traditional way has been more like create tickets that go into the backlog and you have your well-management team, which has an SLA that talks to the team, but there is friction in that, right? The easiest you can make it for someone to do security than we have noticed that they do it. 

When we think about the tools that we want to provide to developers, the first question we ask ourselves how can we make this easy for them to work to sort of use. How can we make it easy for them to fix the findings. Your tickets are nice to track stuff but they don’t make it easy for them. What makes it easy for them is comments and PRs or a PR to fix the issue itself or a very easy way, which tells them what they need to do and that sort of philosophy we have been using — maybe internally build a tool or get an external open source or commercial tool to integrate within pipelines. 

[0:16:36.4] Guy Podjarny: I think that is a great approach and I love the elements of philosophy, right? As we go through when we talk about sort of partnering with developers. You talked about automation first. We talk now a fair amount of a clear theme as this make it easy for developers to do the right thing. 

I guess when you think about shifting a little bit to the side of this, when you think about the profile of a person on your team, do they need to be a developer? What is the profile that you would hire for when you hire someone into your team? 

[0:17:03.5] Yashvier Kosaraju: I’ve been thinking about that over time, like even when I wasn’t a manager. We had these internal discussions of who do we hire next and at this point in time, I look more for a passion for security rather than a specific skillset, but the broader sort of requirements of you need to be able to write code. It doesn’t have to be production level code but enough to automate a security process in a robust way but the main sort of criteria that we hire is empathy to work with developers. 

I would want someone who can partner with them, versus give them a list of requirements to accomplish. So that is the main thing, the different skillset we need it is always a broad thing, right? We need some cloud experience, coding, threat modeling, kind of patient testing, all of these. We can up-level kind of letter individuals on these skills but we cannot really up-level them on the empathy factor. So that comes first in my books. 

[0:17:59.9] Guy Podjarny: Yeah that makes a lot of sense and very aligned with what you have built but yeah, unfortunately not necessarily the primary hiring criteria that many security managers end up opting for. 

So you now have this team, you know they know some element of coding. They have this empathy for developers, there must be a whole bunch of cool projects that you have taken on. Give us an example maybe of one of the projects that the team has done that is interesting. 

[0:18:21.3] Yashvier Kosaraju: Sure, as we talked about we’re building self-service capabilities and the CLI, they are integrating container scanning where a developer can build his real container and say, security CLI and give the full tag of what they’ve done and then that gives some of the results. On the same line, something that we deployed a couple of months ago was an internal tool that looks at the latest solution of the base image and creates PRs in all of our Docker files. 

To update their prompt statement and the Docker file to use the latest version of the base image and the idea being to monitor the base images closely. We work with the team that maintains them to create patches and fix critical vulnerabilities and then there is subsequent automation that creates PRs to make sure that that change trickles down to all the other docker files within the organization. 

[0:19:17.3] Guy Podjarny: That’s very cool, so you are automating because we have a team that is building this base images and maintaining them but then you have the problem of how do you pass that information on, how do you make it easy for developers to consume those newer base images. So you actually open poll requests with these updates when the central team has chosen to release the base image. How has the take up been, all of it, like the people, do developers embrace it? 

[0:19:39.5] Yashvier Kosaraju: Surprisingly yes. So what happened was we developed it, we deployed it and then I went on paternity leave for a while and then COVID happened, we didn’t really sort of advertised this or pushed people to use it but what we found was there was roughly 60% [inaudible] rate where around 65% of the PRs that we created so far have been merged and closed, which is a significant number if you look at the fact that we hadn’t sent any communications out to developers regarding this. Which kind of tells you that everyone wants to do the right thing but maybe it doesn’t have time to do it. So when you make it as simple as possible, they do the right thing. 

[0:20:19.9] Guy Podjarny: Yeah, absolutely I think that’s a great stat and I guess shows you on one hand touch a nerve of a need because I guess they would have had the responsibility to take those base images anyway, so you make it easy for them but also shows that you’ve – you didn’t need to email them about this because you came into their natural surroundings, which opened the port. How do you reward this type of behavior? 

So you have these teams, you have the champions. Do you think about rewarding or sort of highlighting somehow sort of celebrating good security behavior amidst the dev team? 

[0:20:52.1] Yashvier Kosaraju: Yes we definitely do. So the champion program in the advanced sort of version of but we have a lot of folks that go within and they get the capability to do a few actions and approvals that were restricted to your own leader security team with one to sort of level up within the champions program like we give those — I look at them as responsibilities but on the other side, there are perks that we give the developers saying, “Now you can do this.” 

[0:21:19.1] Guy Podjarny: Yeah you don’t need to wait. When you think about sort of these activities and when you think about more security debt, you know like some problems, maybe some vulnerabilities you have in the pipe, things that rightfully you didn’t fix on the spot for the elements, is the management of those types of the backlog of issues something that you manage, that the champions managed that the teams manage, how do they handle the less exciting kind of debt and technical debt is never exciting, security debt is no different. 

[0:21:48.5] Yashvier Kosaraju: We are still figuring it out but I look at it as a combinational effort between the security team, security champions and the engineering team, going forward what I would like to do is sort of have a cache book for every team, which lists out like you have 10 vulnerabilities here. If you want to do this here and these are our feature request from security to that product or engineering team and that can become a discussion topic between the champions and the partners whenever they meet and they can work together to prioritize which ones go first and which ones don’t. 

[0:22:21.2] Guy Podjarny: These leaderboards I think have sort of have been demonstrated to mobilize through actions so yeah, definitely a fan of those. I have been asking it feels like you have these very methodical answers to every one of these topics. I mean it is great, we talked about a great security champions program with the progression inside of it. We talked about key guidelines in terms of how to approach the product, the security and engaging with development from the partnership mentality and empathy to the automation first to making it easy for developers and some great specific ideas like the CLI approach and the poor request for internal base images that you are promoting. 

So Yash, this all sounds like already a whole pile of great advice but still, I cannot finish with asking you if you still had one specific bit of advice not necessarily related to what you are doing in the day to day that you want to give a team that is looking to level up their security fu, what would that be? 

[0:23:16.2] Yashvier Kosaraju: I would say listen to your colleagues within the company, listen to engineers, listen to your colleagues in the industry in general. A lot of these ideas to be honest are not mine. These are some things that have come up in conversations with my peers in the industry, people who manage other product security teams and also in conversations with engineers when I ask them how can we make these easy for you and those conversations lead to really great ideas that extend to a bunch of high value projects for us. 

[0:23:46.3] Guy Podjarny: Yeah, well that is great advice and I think you can build on the shoulders of giants. Yash, I didn’t even noticed this time fly. You know there is all of these great bits of advice. I am sure this is – thanks for sharing this pile of great information and learnings with the rest of us, so we can basically follow your latest advice, which is listen to it. So thanks a lot for coming onto the show. 

[0:24:05.9] Yashvier Kosaraju: Thank you. Thanks for having me. 

[0:24:07.1] Guy Podjarny: Thanks everybody for tuning in and I hope you join us for the next one. 

[END OF INTERVIEW]