Prioritizing Secure Development With Kyle Randolph

“Kyle Randolph: The struggle with security is keeping it prioritized.

Guy Podjarny: If you're going to build it, build it right, and that includes building it secure.

Kyle Randolph: How does my security stack up against everybody else in the industry? What are all the easy wins, high-security impact with very little effort, very little disruption? Trying to work with engineers when you're not part of the engineering organisation is much more difficult. It's much better, if you can, to get a security feature owned by a product manager. If I'm doing something myself, I'm probably doing it wrong, because there's only one of me and there's dozens of engineers.”

[INTRODUCTION]

[0:00:37] Guy Podjarny: Hi, I'm Guy Podjarny, CEO and Co-Founder of Snyk. You're listening to The Secure Developer, a podcast about security for developers, covering security tools and practices you can and should adopt into your development workflow.

The Secure Developer is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market. For more information, visit heavybit.com. If you're interested in being a guest on this show, or if you would like to suggest a topic for us to discuss, find us on Twitter @thesecuredev.

[INTERVIEW]

[0:01:08] Guy Podjarny: Hi, everybody. I'm here with Kyle Randolph from Optimizely, who handles AppSec there, who we've had several conversations in the past. I'm really happy to have him here on the show to share some of his experience at Optimizely. Kyle, do you want to introduce yourself a little bit, talk about what you do, your history?

[0:01:22] Kyle Randolph: Yeah. Hey, I'm Kyle. I've been doing security for about 15 years. Citrix, cleaning up the mess of Adobe, building security at Twitter, protecting free speech there, and then now over to Optimizely, where we're building security from scratch at a startup.

[0:01:36] Guy Podjarny: Oh, very nice. Yeah, that's cleaning up the mess at Adobe was – I guess, this was at the time of the Adobe, the repeated Adobe PDF reader vulnerabilities and Flash vulnerabilities?

[0:01:47] Kyle Randolph: Yeah. If you want to get malware on somebody’s computer, it was headlines in the news. Send them a PDF or a Flash file and take control of their computer.

[0:01:54] Guy Podjarny: Yeah. I guess, there's the security aspect, and they’re the victim of their own success in the sense of having Flash at what? Some 90-high percent of machines everywhere.

[0:02:03] Kyle Randolph: Yup. Yeah. Yeah. There were more computers with Flash at one point than there were Windows machines.

[0:02:09] Guy Podjarny: Yeah. You deal with, you say you're at Optimizely, you're building security. What's your role these days?

[0:02:15] Kyle Randolph: Yeah. My role is, throughout engineering, all of our products that we build, ensuring that we have security built into them, making sure both that all the different features we're building are secure, as well as having the right security features in place that enterprise customers would expect in a product that they would use across their organisation.

[0:02:32] Guy Podjarny: Interesting. You're working closely with the product itself. You don't just deal with the security of it after the fact, or the audits. It's more about building it in?

[0:02:40] Kyle Randolph: Yeah. We've built our program to be tied very closely to engineering. We sit with engineers, interact with them every day from design reviews to code reviews, to advising on libraries, design patterns, tools, and automation that they can use, working very closely with DevOps. Then from the beginning of an idea to when we ship that out the door, all the security that we need to have built in, making sure that it gets built in.

[0:03:05] Guy Podjarny: Cool. Is it the same team? This is an AppSec team, as opposed to an InfoSec team?

[0:03:12] Kyle Randolph: Yeah. We call this an AppSec team, because it's within engineering. We work with the product management and engineering teams to build security in there. We partner with the InfoSec team, which is more focused on protect the company from traditional IT security risks, so making sure our Wi-Fi is secure, our actually, directory domain controllers are isolated and have good access control.

[0:03:38] Guy Podjarny: Okay. You and your group are in the development organisation. You work with Dev?

[0:03:42] Kyle Randolph: Right. Yeah. Yeah.

[0:03:44] Guy Podjarny: Cool.

[0:03:44] Kyle Randolph: I've been in organisations both where the security team is inside and outside of engineering, and trying to work with engineers when you're not part of the engineering organisation is much more difficult. It's harder to get traction with them, or you may not seem as approachable, but being tied in engineering, you know what's being built, you know what's on engineers' minds. It's much easier to gain influence over engineers and also have them come to you.

[0:04:08] Guy Podjarny: Yeah, it makes perfect sense. Just for context, how does the ops side of the fence work in Optimizely? Is that also a part of that same entity?

[0:04:17] Kyle Randolph: Yeah. There's a DevOps team, we call it E-Squared, or engineers for engineers. Building tools and automation to help engineers get their job done easier. We partner with this team very closely. For example, with Amazon Web Services, a lot of our infrastructure is deployed there. In E-Squared, they want to automate a lot of that, so developers don't have to think about how do I get my code into production. Then we want to bake a lot of the security in during that deploy process.

In AWS, you have security groups and VPCs and cloud trail logs, and so on. There's dozens of things that we don't want developers to have to think about in order to make their features secure, so we build that into the automation when you're pushing this thing into production, you're also baking all the security into what they're shipping.

[0:05:01] Guy Podjarny: Right. Yeah, it's all about, one, lowering the bar, just making it easy for you to consume it. The other is avoiding all the specific manual laborious tasks and make it just happen. It's all about scale, I guess, which is amazing how that type of concept has really become natural in the world of quality testing, where QA – it's not that QA is decreasing really, but in many more startups, for instance, in Snyk, we don't really have a QA team, and there's really no plan for it. The understanding is that you own it and you own the quality. You build tests. You build components, and they grow there. Yet, somehow, it's a little bit more novel to do that in security. That's cool.

You joined Optimizely. When we first chatted, you still had a lot to build up. You joined in to create this ecosystem. Can you share a little bit about how that transpired? What did you see when you came in and what did you do to improve it?

[0:06:03] Kyle Randolph: Yeah. I joined Optimizely, October 2014. It was a startup. They'd been around for five years. Built primarily in Python, on Google App Engine. A large monolithic web app in Python, so you didn't have to worry about buffer overflows, because it's an interpreted language. Then because it was built on Google App Engine, you got a lot of your security for free, because Google has taken care of all the security lower in the stack and they know what they're doing. That helped Optimizely get along for a very long time without a formal security program in place.

Then the developers were security savvy and had the right idea for how to build some security in based on reading OWASP, being familiar with the OWASP Top 10 and common vulnerabilities. As this monolithic application grew and they got more customers and larger enterprise customers, the security requests from the customers began to grow. Asking for two-factor authentication, single sign-on, audit trail, and so on.

[0:07:03] Guy Podjarny: The security features, not just security guarantees, but straight-out security features.

[0:07:06] Kyle Randolph: Yeah. Then the security risks started to become more sophisticated as the product gained traction. Now on a lot of the largest websites in the world, the most popular ones, they have Optimizely on them. As an attacker, before a little startup, you may not even notice it, but now you're like, “Hey, I could compromise all these sites at once.” If I compromise Optimizely, that raises the stakes, and that's where more advanced security was needed to be baked in.

[0:07:33] Guy Podjarny: Yeah, that makes sense. Today, Optimizely has its scripts on also, many pages. You can get a lot of access to data and you can actually manipulate the content. I guess, Optimizely modifies that content, eventually, as its core being to optimise the site's performance. It feels like a rich target. I'm sure you have your hands full. There was no formal program. There were a bunch of developers doing the OWASP thing. How did you approach it? What did you do first?

[0:08:04] Kyle Randolph: Yeah. This was interesting coming into a company. I would say, the engineering team, there was no security engineer there ever, but the engineers wanted to do the right thing. They just wanted to know what is the right thing to do. There's thousands of security things they could do, but what are the most important ones they should be doing first? As the security engineer coming in, I wanted to make sure I built up a good relationship with engineering than just bog them down with tons of security requirements and keep them from building new innovative features.

Some things I wanted to do from the start are like, find what are all the easy wins to begin with. Let's find all the low-hanging fruit that will give us high-security impact with very little effort and very little disruption in engineers. Building out a backlog of those. Just going around, talking to engineers, buying them beers, buying them coffee, whatever's needed to talk about what's on their mind. Not even saying anything that they should be doing for the first month or six weeks, but just listening. Everybody brings security issues and concerns to you, and you discuss them, but without – don't make them defensive and tell them what they did right, or wrong, or whatever, but just you collect a lot of information that way. Then from there, you can start building out your plan and you know where the hotspots are that you need to focus on, or put some process around, or build some defences in.

[0:09:24] Guy Podjarny: Yeah. I guess, it comes back to the fact that everybody wants to be secure, and then it's more about dealing with either the lack of expertise, or just the time crunch. The fact that if you've invested a whole chunk of time in making something more secure, oftentimes that's invisible. That's just not seen by anybody. Yet, the fact that you have not built this feature, this functionality, that's painfully visible. It's something hard. I guess, to an extent, you can be there for the things that they want to do, you can be their excuse, their support person, and also, their expert to say, “Well, is that really worth doing, or not worth doing?”

[0:10:00] Kyle Randolph: Yeah. What should you do first? Security vulnerability, like severity ratings, those help somewhat. But what helps more is telling compelling stories of like, hey, here's a real-world example of when this vulnerability led to a company actually being compromised. Something very similar to Optimizely, a company called Gigya, they also load JavaScript on a lot of different companies' websites. What happened to Gigya is the Syrian Electronic Army said, “Hey, let's compromise Gigya, compromise their JavaScript on all these different sites, Forbes, all these different sites, all at the same time,” and said, “You've been hacked by the Syrian Electronic Army.” That was a great way to drive home. What would a Optimizely Security vulnerability look like in the real world? We don't want that to happen, so here's the things we need to do to prevent that.

[0:10:46] Guy Podjarny: Yeah, that definitely sounds like a good example of it. I guess, as close an example as you can get to the actual business, makes people relate to this could happen to us. Those were some of the tools that you used to be able to educate, or promote maybe you give them these real-world examples. I guess, you also used the, so these conversations both to build relationships, which at the end of the day, it's all about people, right? Also, to understand and get the lay of the land. What did you do? At some point, did you do need to start tackling those? You've identified your priorities, you understand them. You're buddies with different people, so they would trust your judgment and listen to you. What are the first steps that you introduced that are still process-oriented or tooling-oriented?

[0:11:33] Kyle Randolph: Yeah. A guiding principle was, if I'm doing something myself, I'm probably doing it wrong because there's only one of me and there’s dozens of engineers. It's all about enabling engineers to be able to do these things. Setting up many different things at once. Showing off tools at lunch-and-learns, like, “Hey, look at this really cool security tool that can help us find cross-site scripting.” Just show it off. Show how cool it is. Show a success story, and then cross your fingers that at least one engineer in there will think it's so cool that they'll go run it themselves.

That's like fishing, not PH phishing, but just fishing for people who are interested. Sometimes that works. Sometimes it doesn't. but it's a good way to get interest. Getting a security group put together, we call it the security cabal at Optimizely. This is like, any engineer interested in security, we meet each week, just talk about security stuff or cool security tools. We also have a Slack room, where we just talk about these things daily. Just by having a constant conversation, a lot of engineers will pick up something, play around with it for the afternoon, and start trying these things. That helps a lot.

[0:12:40] Guy Podjarny: Sorry, these are just the people that are interested in security naturally. It's not their job. They’re engineers, and they just, alongside their day job, that just is one of their passions?

[0:12:52] Kyle Randolph: Yeah. Yeah. In the early days, there were so many things that need to be done for security. It was like, okay, we have this. Let's say we have a hundred things to do. We have 20 things we know we need to do now. Let's put those 20 out there and see how many people are interested in it. Chances are if they're interested in and passionate about it, they're going to be much more productive working on it than if it's assigned to them. Let's get those out of the way first.

Other ones kept on the shelf and waited until like, let's say, there was a quality improvement that need to be done, where you’re going to split this feature out into a microservice. That's a pretty good time to add in some security that we had sitting on the sidelines that we knew need to be done. Or another quarter, we had initiative to speed up the delivery of our CDNs. That's a great time to say, “Hey, now it's time to look at TLS optimisations,” for example. Let's clean up the unsecure cipher suites that we have in TLS. Satisfy the security guy, making it secure. Then also, we get some speed out of it when we start turning on TLS optimisation.

[0:13:55] Guy Podjarny: Right. Yeah, basically bundled it in as a, I guess, maybe as a natural aspect of quality for those components and just, yeah, just make it be around. If you're going to build it, build it right, and that includes building it secure. How did you find out? As you say, you're one too many in terms of a number of people. Absolutely awesome. I think that you're not a gate. I’m a firm believer that it's just it would never scale. It has to be built in and it has to be spread out between the people in this continuous fashion. Also, you can stop any process and have them be done manually. But you still need to know, like you need to know that a microservice is being built. You need to maybe advise a little bit about what are the opportunities. Did those end up just permeating to you because relationships? I mean, how does that work?

[0:14:43] Kyle Randolph: Yeah. That one is a challenge. How do you scale up with the organisation? Classic waterfall times, you go through this big requirements phase to see all the requirements. One PRD document and you could just mark the ones that you have concerns about. Now in more agile development, everybody's building on their own schedule, they may or may not use common, or agile, or whatever. There's no one place where you can find everything.

With that, you – I found you just have to spread yourself everywhere that the engineers are to have a lot of ears. Hanging out in Slack rooms, where the action is. Watching the email list, joining the team’s email list, going to their meetings, going to their special interest groups, speaking every week about security and different meetings where engineers are going to be. Just make yourself as available as possible. Then you have a lot of engineers coming to you, which is great, especially after they see you're going to help them and not slow them down. Then also, having your ears open in all the places that you need, so you get wind of things. Sooner or later, you'll find out about them one way or another.

It may not be in the design phase, but you want to find out about them as soon as possible. A great place to find out about things is source code. Our code review tool, setting up rules, like if anybody touches these files, like opening up a new route in our monolithic web app, then we know that's something. That's a new feature. That's something that is a good place to flag for security.

[0:16:15] Guy Podjarny: You've got little spies there, right? Not people spies, but just sensors, I guess, spread out to tell you something interesting might be happening. You do this, and I think it's awesome, and again, I believe in the approach of permeating it, making it a natural part of the culture. I've seen a lot of that success also in performance, in web performance, because I've been oscillating from security to web performance. You see a lot of that also in the notion of you don't want to be a performance janitor. You want to permeate that and make it a natural part of your company, and security needs to be the same.

A lot of the conversations we had there was about celebrating successes, which is always tricky in security, because not getting hacked is not sufficient, because hopefully, that's the normal course of operation, otherwise, you've got other problems. Did you consider that? I mean, are there ways in which you somehow celebrate either an investment or a success in security?

[0:17:10] Kyle Randolph: Yeah. The company has different ways to recognise success, and we tap all those for security and make up some of our own as well. Company-wide, there's a way for one employee to thank another employee for something, and that's very public and recognised each week at the company all-hands meeting. Using those thanks to thank people, who even just fixed one security bug. Anything was secure. They had a good security question to give them thanks there. Stepping that up a bit more, in quarterly retrospectives, what went well, what didn't go well. Security guy, you could always just say, well, security wasn't as good as it could have been, but more pragmatically.

Talking about like, who did do awesome security. This team is great. They fixed all the security bugs they said they would. Or everything they made this quarter, they had a design review before they built it, and so we knew security was baked in at that very early phase. Going further than that, giving out T-shirts. We actually give out T-shirts that say, “Security Hero” on them. This is more exclusive, so it makes people want to step it up and really go above and beyond to make a security contribution. Maybe you eliminated cross-site scripting, you built it into the framework your team uses, then that would warrant a Security Hero T-shirt, then you'd recognise from the whole company.

[0:18:26] Guy Podjarny: Nice. That one you need a slightly higher bar, because if you start giving them up too much, then you actually lose the prestige of having that shirt.

[0:18:35] Kyle Randolph: Yeah. Many different tiers of reward there and recognition.

[0:18:39] Guy Podjarny: I guess, are there specific examples of, well, you gave a little bit with the cross-scripting framework, or with the team that is fixing this, but do you have a specific story that you feel manifests, really demonstrates how this succeeded converts, or the least?

[0:18:57] Kyle Randolph: Yeah. Cross-site scripting and cross-site requests forgery in both of those cases, we had an engineer who they may have had a bug or two assigned, and they're like, “I've seen this before. I don't want to see this again.” They just went out of their way to really say, “Here's how we're going to fix this comprehensively.” In one case, this one team with cross-site scripting, they didn't even talk to me. They were just within their own team, the guy made a presentation like, “Here's why cross-site scripting is bad. Here's how we're going to make sure our team never has it anymore.” Then they're like, “By the way, we did this. We've been cross-site scripting.” That was awesome, and not even have to get involved. They just took it upon themselves to do it. That's even better than roping the security guy in to tell you what to do, but to decide on your own to do it.

[0:19:44] Guy Podjarny: Yeah, that's great. That's definitely reaping the fruits of your labour, right? You planted those seeds and they grow inside. This is all process and people, and how do you celebrate them. Are there specific tools that you'd throw out that you find especially useful?

[0:20:01] Kyle Randolph: Yeah. We've been focused in AWS. How do we get our heads around that? It's not so much security tools that help there, but just latching on to the automation that's going in there. DevOps team and engineering teams, they want to manage just their networks and VPCs through cloud formation, but, hey, you could also manage security groups in there by bundling clean-based way in a nice tool called Demeter, which manages their security groups, nice and simple.

We're using a lot of Spinnaker for our deploy automation, which is not a security tool, but that's just the place that you can bundle in all the other security configuration that you want to have happen. Those have done very well for us. The third-party libraries, like we know we need to address that, and Snyk looks very promising. Looking forward to trying that one out. That's crazy to think how much third-party code we have, and we don't know what's going on if we don't have a tool, they can actually do all that analysis.

I remember being in Twitter and wondering like, how's this monolithic Ruby on Rails app pulling in code from all over the web, the Twitter scale not have some compromise and it's really scary to say. That's looking very interesting. Then we have a web app security scanning tool. I've never found one that works. I love to know about one there. I've built two, and I can point out many, many flaws in either one. Yeah, so those are the tools now. Definitely looking more to automation and open-sourcing things as we go.

[0:21:37] Guy Podjarny: Cool. I guess, on the – and if it's dark side, or a necessary evil, in security, it's good and well, that, and in fact, what you need to do is permeate and build it, let it come organically. Sometimes you do have hard lines, right? We talked a little bit about PCI just before this started. You do have some other regulations. There are security questionnaires, these joyful hundreds and hundreds of questions and diagrams that you need to provide. How do those get handled?

[0:22:09] Kyle Randolph: Yeah. With those, it's like customer stories. That's putting on your product manager hat and telling these stories very frequently. Unfortunately, you have to repeat yourself very, very often when you're being this security feature evangelist and reminding people constantly like, these are – our customers want PCI. Then PCI wants to preserve the integrity of our JavaScript, and then we need X, Y, and Z to preserve the integrity of JavaScript. That's our contract with the customer. Tie it all back to that. Or it could be ISO, or something else, data privacy in Europe. Like, tying it back to a story and then just telling that customer story over and over that keeps it in people's heads on why they need to do that.

[0:22:53] Guy Podjarny: I guess, when you come to think about it, that's indeed, oftentimes, how you promote, how you manage to get a customer story around a feature, or a need, or a capability comes in, so as well, this is a core necessity. I imagine, those do eventually get captured by more the product managers and put into a prioritised backlog, or?

[0:23:10] Kyle Randolph: Yeah. Typically, it's security feeding them into the product management backlog. Then once they're accepted in there, then you have a – it's much better, if you can, to get a security feature owned by a product manager and then they can – that's their specialty, and they're very good at seeing those things actually get built and built properly with the right documentation, marketing, whatever you need, so that you can focus on doing the engineering side of things.

[0:23:35] Guy Podjarny: I guess, stepping out of Optimizely for a moment, you have this experience across different companies, right? You've been big and small and, I guess, different phases. Do you feel these techniques that you have in Optimizely, how many of those are, this is just the way you would have approached it, almost regardless of company, versus things that you think you need something there to be able to apply this. How broadly applicable are they?

[0:24:02] Kyle Randolph: Yeah. With this one, I'd like to look at the BSMM, the building security and maturity model. This is where Sigital did interviews with over 100 software companies and asked them just like, what are all the things you do to build secure software? It was across different industries, types of companies, and so on. The interesting thing there is a lot of the activities are common across different types of software companies. It could be an insurance company. It could be a SaaS company. They may do them in different ways, or for different reasons.

It's not one size fits all, but across different domains of security. I believe they have 12. There are very common activities, about 120 of them. Some of them make sense for Optimizely, others don't. Some of them make sense for Microsoft, others don't. That's a good place to start and use it as a measuring stick like, how does my security stack up against everybody else in the industry?

[0:24:58] Guy Podjarny: It's always an endless set of tasks that you could invest in it. how do you know that you're doing sufficiently well? I guess, that's a million-dollar question. I guess, within this, do you have a favourite? This is all broad and this, I've asked for the anecdote as well, but what's your pet peeve? What's the thing that you find you really, really either like that people do, or you really get annoyed when people do not do and surfaces? Is there one?

[0:25:26] Kyle Randolph: Yeah. I guess, the struggle with security is keeping it on, keeping it prioritized. A lot of times people want to do the right thing, but then when you get to a resource crunch, it features often when out. A thing to be careful with here that have been a struggle throughout my career is just getting teams to commit to building things, like we agree this is a security issue that needs to be addressed, but when the horizon goes out more than like a sprint, okay, then it's like, it'll be done next sprint, next sprint. Next thing you know, you're a quarter out and something has fallen off the radar, even a year out.

With this one, I found it is unfortunately very proportional to how much energy the security team puts into evangelising those things and pushing for them. You have to be very selective and identify those at-risk things that may not get done and save your energy and your political capital for those to really drive them through.

[0:26:21] Guy Podjarny: Yeah, I guess the – it's focus, focus, focus, right? It's all about choosing the things that you want to handle well, versus handling a whole bunch of other things.

[0:26:31] Kyle Randolph: Yeah, yeah.

[0:26:32] Guy Podjarny: This was super fascinating and I've got a whole million questions to ask, but I think we're going to run out of time a little bit. Before we part, just as a word of advice, I try to ask this of all the guests. If you talk to a development team right now looking to up their game in terms of security, what's your one area that you would advise them to do, or one thing that you would advise them to start with?

[0:26:58] Kyle Randolph: I think that getting all of your security issues out in the open, discussing them, like a lot of teams will say like, “Oh, yeah. We've known about that, or I heard there was a security bug over there,” but you don't even see them in their bug tracking system. Can I even go see what are all the security concerns that we have? I'm not even having that transparency about what your security exposure is, or what you would like it to be. Without that information, you don't even know how to do the right thing.

I think, getting all the concerns on the table, what do we expect our security to be and what are the known problems that keep us from being there? That's a good way to bootstrap on what is the security that you need to do and what priority?

[0:27:37] Guy Podjarny: Yeah, it makes perfect sense and get your head out of the sand, right? It's much more convenient to just not think about these things and hope for the best. Just forget about it. We see this a lot with third-party security, right? You use them. It works. You'd rather just not really open up Pandora’s box. Yeah, otherwise, if someone just start surfacing it, I guess, you'd be more motivated to actually start making a dent in that list and improve them.

Well, this was fascinating. Thanks a lot, Kyle, for coming over and yeah, looking forward to chatting some more.

[0:28:08] Kyle Randolph: Great. Thanks a lot.

[END OF INTERVIEW]

[0:28:11] Guy Podjarny: That's all we have time for today. If you'd like to come on as a guest on this show, or want us to cover a specific topic, find us on Twitter @thesecuredev. To learn more about Heavybit, browse to heavybit.com. You can find this podcast and many other great ones, as well as over a hundred videos about building developer tooling companies, given by top experts in the field.