Beyond The Security Team With Julien Vehent

EPISODE 46

[0:01:19.2] Guy Podjarny: Hello everyone, welcome back to the show, today we have a special edition of the secure developer podcast, this sort of DevSecCon edition and with us, we have one of the keynote speakers here at DevSecCon. Julien Vehent. Welcome to the show Julien.

[0:01:31.4] Julien Vehent: Thanks for having me.

[0:01:32.8] Guy Podjarny: Julien, before we dig in to sort of the talk and details, tell us a little bit about yourself, what is it that you do?

[0:01:37.9] Julien Vehent: Sure, I am a security engineer at heart, I run the Firefox operations security team at Mozilla and our job is to protect the Firefox backend infrastructure. So all of the services that you don’t see when you use browser. Some of them that you see and all of the build infrastructure that compiles Firefox. I’m also a DevSecOps advocate and the author of Securing DevOps.

[0:02:01.3] Guy Podjarny: Very cool. Already some initial homework for our audience to go check out the book. Okay cool, you sort of build security operations and grow them within Mozilla. What is your talk about here at the event?

[0:02:14.0] Julien Vehent: My talk is really a mix of a retrospective of how we built the Firefox operations security team at Mozilla, where we came from. And where I’d like to see it go and the talk is —initially I wanted to call it, "There is No Security Team," but I thought that might scare people off a little bit so I said "Beyond the Security Team", which is really the goal where the organization is so mature at improving and adopting security that it pretty much no longer needs to have a dedicated security team.

It’s really was the end state is for a lot of us is making all of the improvements we can to have developers and operator, control their own fate basically and control their own security. My talk is a discussion of which techniques we use at Mozilla to grow the maturity of the organization, which one worked, which ones didn’t and where we see it go in the future.

[0:03:07.1] Guy Podjarny: Do you think that’s the future so do you see kind of the end result being a place in which there is no security team? Do you see it as a different type of role?

[0:03:17.3] Julien Vehent: No, I don’t think so. I think we’ll always need a security team but I really wish it were so, I think the future shape of the security maybe quite different. We’ve already de-invested heavily in infrastructure security because we mostly use the cloud now and we outsource the cost of infrastructure security to cloud operators.

The traditional shape of a security team has really changed over the last five or six years, where we’re starting to see more and more teams adopt a software engineering culture developing tools and integrating their tooling and their controls inside of the CICD pipelines, et cetera. I think what’s going to happen over the next few years, that this trend is going to continue that teams are going to be less involved into the day to day because we have this notion of paved roads.

Because we have these services that are secure by default that are going out that don’t need to be really managed by security engineers constantly. Instead, security teams will move towards being more of a strategic team that builds tools that can be used globally in the organization or set the long term strategies, security strategies of the organization. Collect metrics, discuss threat models and risks, that’s really where I think what would end up being a team of expert consultants and not down in the trenches anymore.

[0:04:32.7] Guy Podjarny: Kind of operators. Is it sort of a fair analogy to compare it to like the system reliability engineers or, you know, some of these that sort of the modern roles of ops experts in kind of the era of DevOps, if you will?

[0:04:48.5] Julien Vehent: Perhaps, yes. I think what we’ll see is really these ideas that we’re using more and more standardized tools that are secure by default. The security team is kind of writing the security standard and improving the security standard and not necessarily worrying about how the standard gets implemented day to day, right? 

If you look at very mature organizations, they essentially have one way of deploying things. One way of writing applications and that one way is secure by default, right? The security team focuses on that one paved road and improves that one path. It frees up a tremendous amount of time for security people to go work on other things. So that they don’t have to worry about checking every single deployment or checking every single instance of a service.

[0:05:37.6] Guy Podjarny: You created that paved road which has sort of seem indeed, secure by default, it might have you know, some relevant tests and tools in the pipeline, whether it’s in the practice you do at Mozilla or just a general recommendation, you know, do you think it’s a paved road but, you know, a developer should be allowed to kind of walk off to the side or there’s more two walls to the side of that paved road?

[0:06:02.1] Julien Vehent: No, I think we need to embrace chaos and we need to allow people to do crazy things. I mean, that’s part of allowing for success that no one else can really foresee in advance, right? You may have a developer who has a crazy idea in the corner of the organization, and allowing that developer to try out that idea, may prove to be immensely successful, it may fail as well but you need to leave some room for that. 

The paved road is really about if you have a standard service, if you're not doing anything interesting, you don’t want to spend a whole lot of time rebuilding everything, then just do it on the paved road. It’s easy, right? If you’re spinning up a new API that just solves a business case and you want to design a couple of days, the paved road is great for you. If you want to try something new, then you should have the support from the security team and the rest of the organization to help you cover the risks that people may not know about yet, right?

But freeing up that time on the paved road gives security teams a time to go look at these new projects, to embrace the chaos and go help the snowflakes, kind of do things and avoid the risks. The way we do this at Mozilla, we have a lot of innovators, I like to think of Mozilla as more of an incubator of startups and not just a corporation and all of the small groups that like to innovate will come up with new frameworks, new infrastructure, new models we’ve never seen before.

We give them checklists of a very specific type of risks and threats that we want them to think about and cover when they engineer their applications and then it’s kind of up to them to figure out how they implement that and how they take security into account. We help them of course but we give them that set of requirements and most developers actually really like having those clear expectations written down for them to work through and think about. They feel like they own the aspect of security and they don’t just offload it over to the security team.

[0:07:52.4] Guy Podjarny: Yeah, I fully relate to that and I think the idea would be, make it easy to be secure and also make, choose that path. But then you give someone the freedom to — a developer and then part developer to veer off the path and to choose their own aspects but they need to be intentional about it and in that case those are the cases where you need to equip such developers with the ability to ask the right questions and to maybe piece and parcel the different elements. 

Where do you think we are on this journey? I don’t know it is as an industry, as Mozilla, how far down that path do you think we are? 

[0:08:28.9] Julien Vehent: Well not everyone moves at the same speed even within the single organization, you always have groups that adopt new things faster than other groups that are slower to move. I think we are already very much in a situation where the security team in most dev-ops organizations, the security team is no longer involved with the low-level infrastructure security operations, right? They really let the operators take care of that and they build the tools to assess. 

So we kind of moved away from that down in the trenches really model that we had a few years ago. Most security teams are adopting this tool building software engineering approach. So we’re probably half way through, in that sense. The next big step is really to have the security teams really work at the paved-road level, provide those secure by default controls at every level of building services securely. 

So there is still a whole lot of work to do and another thing that I think as a security industry we need to make progress on, is improving our security strategy and then on-off cases, we are down at the tactical level. We look very specifically at a specific control, a specific version of TLS that needs to be adopted, dependencies that need to be updated, et cetera. To be honest, those problems are kind of boring like update your systems, have the tools — like I am not going to say that to you, have the tools to tell that a system is running on insecure updates and then just do the thing. 

You don’t need a security team to be involved with that, maybe the security team builds the tooling but it doesn’t need to tell developers things that this should just know and they already know in most cases. And so we’re still thinking in those tactical terms, when in fact we should just let the organization take care of those things and move up. Just think about what it is, what it means to run a secure service tomorrow, right? Does it mean that we need to think about data retention and data security? 

We are still leaking data all over the internet every time someone uses a service and trusting some S3 buckets that may be publicly accessible, etcetera. We need to provide better guarantees that our mostly at an end user and business level and stop worrying about the day to day as much. 

[0:10:44.8] Guy Podjarny: Got it. And how maybe kind of one last question is, what does this do to the skillset within the security teams? Do you think security teams as they stand today, you the that people employ it with sort of the experience and skills that they have, they are also the right people to sort of level that up? Should we expect some shake up in the makeup of the people and the skills and the talents within the security orgs? 

[0:11:12.9] Julien Vehent: I think it already happened — not everywhere obviously. You still see a lot of traditional organization that employ traditional security people or the [inaudible], the network security engineer who manages firewalls. But in a lot of more webby, dev-ops-y organizations, we already see security teams being composed of software engineers, mostly, of people who have an architecture background, who are more concerned about how we build services globally than they are about this data firewall rules. 

Or the shape of the VLAN, right? So the next challenge for the security industry is how to level up its own people to make sure that everyone understands software engineering, services engineering and can help organizations adopt secure controls at the software engineering level. We are moving up the stack basically. We are moving up the stack. The security industry is already a lot more of an knapsack industry and it was 10 years ago right? 

And it is going to continue going up. Because framework subtract more and more of the knapsack level and so that is why I think to be successful in the next 20 years, a security engineer should know how to program, how to implement services but also have a keen understanding of risks, threat models and how to talk to business.

[0:12:33.7] Guy Podjarny: Yeah that makes perfect sense and I think the thing that happened in some places and there is a lot of places that it hasn’t — but I agree with you on that trend and I guess overall a positive. 

So before we let you go any other sort of key highlights you want to or plugs for the talks? We are going to post the YouTube video as well and a link to it and the other sort of key sound bytes to lure people into watching. 

[0:12:57.5] Julien Vehent: Well there are hopefully quite a few interesting stories. I guess the one thing that I mentioned and maybe that will make people want to watch the keynotes, I used to run a site affiliated with “Who Wants to Be a Millionaire?” And I used to crash that site almost every day. So that was the fun side of fronting ops before we had dev ops. So what I am going to share tomorrow is some of the lessons learned from doing shitty ops and how to prevent that. 

[0:13:23.2] Guy Podjarny: Excellent that is a great plug as well. Julien, thanks a lot for coming on the show. 

[0:13:27.3] Julien Vehent: Absolutely, thanks for having me. 

[0:13:28.9] Guy Podjarny: And I hope you join us for the next one if you have enjoyed this DevSecCon edition of the Secure Developer Podcast. 

[END OF INTERVIEW]