Managing Security With Julie Tsai

“JULIE TSAI: I never intended to go to security, but it was always something that needed to be handled. At some point, I realised it was always those kinds of problems that I couldn't stop thinking about. Part of what makes careers in security so interesting, every professional skill you have you will use. Security is both science and art, right? We have to find these very objective results and metrics. At the same time, it becomes a judgment call. There's something beyond just the compliance piece of it because if you look at all these companies that did get breached, they were compliant. How do you tell the difference between a good security person and a bad security person? You can't. They both say no.”

[INTRO]

[00:00:38] GUY PODJARNY: Hi. I'm Guy Podjarny, CEO and Co-Founder of Snyk. You're listening to The Secure Developer, a podcast about security for developers covering security tools and practices you can and should adopt into your development workflow. The Secure Developer is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market. For more information, visit heavybit.com. If you're interested in being a guest on this show or if you would like to suggest a topic for us to discuss, find us on Twitter @thesecuredev.

[INTERVIEW]

[00:01:09] GUY PODJARNY: Hello, everybody. Welcome back to The Secure Developer. Today, we have an awesome with us. We have Julie Tsai. Thanks for coming to the show, Julie.

[00:01:16] JULIE TSAI: Oh, it's a pleasure.

[00:01:18] GUY PODJARNY: Can you tell us a little bit about yourself, about what you do, and I guess, kind of how you got into security?

[00:01:23] JULIE TSAI: Today, I am a leader and a director in security organisations, so right now working for a major a cloud content management company. I started out my career more than 20 years ago now debugging network controller hardware. I spent about 13 years in system administration and infrastructure space on the actual delivery side of it, TechOps. In that process gained security knowledge because it was often part of our second jobs that weren't named.

[00:01:53] GUY PODJARNY: Yes. That was never named but had to be done anyway.

[00:01:55] JULIE TSAI: Absolutely, absolutely. It was not named. But if there was a problem, they would be calling us.

[00:01:59] GUY PODJARNY: Absolutely.

[00:02:00] JULIE TSAI: I gradually made my way into doing a couple of years of development. I found out that I was a better sysadmin and then went back into doing more DevOps and DevSecOps types of things and eventually moved into management.

[00:02:12] GUY PODJARNY: What drew you in? What about the security aspects of that, so sysadmin, attracted your attention?

[00:02:17] JULIE TSAI: Yes. It's an interesting thing for me, at least. I never intended in the beginning to go to security, but it was always something that needed to be handled. At some point, I realised it was always those kinds of problems that I couldn't stop thinking about, either because of the acuteness of it or because of the complexity. Then eventually, as I started to get in deeper, I would find myself more frequently having conversations with my managers and executives about, “We really need to do this. I really, really need us to not forget about it. This is how we need to do it. It was starting to happen at company after company."

Finally, when I was interviewing about five years ago over at Walmart, I met a really great VPF platform from over there. Originally, they were interviewing me for a DevOps role. I was like, “Ah, I don't know. These things get defined in so many ways.” He looked at my resume and said, “You seem really passionate about security because you're always doing security problems.” I was like, “Yes, I think it's important.” He said, “Why don't you consider working for the security department?” I said, “Sure, we'll give it a go.” It actually ended up being great career advice.

[00:03:21] GUY PODJARNY: Yes. Oh, awesome. I think there are today, for the people, the biassed selection of the people that come on this show, there's oftentimes either sort of a Dev or an Ops or sysadmin and Ops. They really are just different terms for very similar types of activities that evolve them into security. That passion that sort of draws you in puts you in a stronger starting point for it.

[00:03:43] JULIE TSAI: Oh, absolutely. You have kind of a love-hate relationship with it. You may find yourself cursing out the security people you work with but then wondering why you're always going to go talk to them or work with them and figure out those problems. Someday, you will be there, too.

[00:03:57] GUY PODJARNY: It sounds like you made that transition in Walmart. But really, I think part of the conversations we had here ahead of the show was around your experience spanning sort of a good number of sizes of organisations.

[00:04:10] JULIE TSAI: Absolutely.

[00:04:11] GUY PODJARNY: I think maybe let's sort of talk a little bit about being kind of a security leader or sort of driving security in organisations. Maybe let's go from small to big.

[00:04:20] JULIE TSAI: Yes. My career roughly progressed in that direction as well. I loved being at startups and started out my first few professional years working for organisations that were 7 people, 30 people, and thought this is the way to go. Be in two rooms with a bunch of really smart difficult people and get to know each other well. Eventually, over time, the infrastructure problems that I needed to get good at and to solve were starting to go to slightly bigger and bigger companies. Let's say 100-person companies, 200-person companies. I realised, okay, if I want to understand how to do scale and performance at this stage, I need to just go there and found great experiences there as well, bonding with the team, learning how other people did stuff.

Also great to get hands-on experience implementing security at that ground level when the buck stops with you. You need to be able to do it in a sustainable way, and you know what the answer is going to be if you go back and ask for more budget. I started learning how to design and think about things in a way that I thought was going to be scalable for my teammates and myself and not create more work. My first encounter with what we call some of the DevOps tools with configuration management with CFEngine. After I finished reading a bunch of articles on bootstrapping and infrastructure and some of Mark Burgess's papers from the late nineties and said, “Oh, this is how I think it should go. I haven't had a chance to implement it, but let's go ahead and try it at this company.”

At this company, I was – it was maybe about 15 people to 200. We had level one service provider PCI compliance we had to meet. Actually, the standards are pretty high for a company that was basically trying to punch above its weight class, right?

[00:05:57] GUY PODJARNY: Yes, to do. In this context, you're still in a sysadmin role or, you know.

[00:06:00] JULIE TSAI: I am.

[00:06:01] GUY PODJARNY: You’re not officially in security.

[00:06:03] JULIE TSAI: No.

[00:06:03] GUY PODJARNY: You're doing it as one of those things that needed to get done.

[00:06:05] JULIE TSAI: That's right. That's right. There was no security department that was separate at the time. This was maybe 10 years ago. When I think about the bulk of my system administration experience, it was always – your first responsibility was going to be reliability and uptime, as well as the responsiveness to the customer and the product folks. But the hidden responsibility was actually security. The more I was thinking about it, I realised this hidden responsibility is actually as big as or possibly bigger than our other responsibilities because there's a reputational risk here for the company, as well as for us individually. The liabilities for this are maybe more than what that particular feature is going to make in terms of money.

How do we express this to the upper brass? That became a sort of a trajectory that I saw going through my career in the beginning, just kind of figuring out, okay, how do I make these changes really efficiently? I'm going to use these really cool tools that are going to let us scale and keep things in perpetuity and be bulletproof. Over time, I started to see like, actually, we have a really serious thing here where we have to be able to express risk about all these different things we're doing in the technical space to people who haven't yet been quantifying it.

[00:07:13] GUY PODJARNY: What were the kind of key tips and tricks that help get that done? Do you remember some core – the approaches that did work and the ones that you thought would but actually ended up bombing?

[00:07:24] JULIE TSAI: Yes. I think it's a – this is one of those things that you have to use the carrot, the stick, and every bit of social influence that you need to have. Part of what makes careers in security so interesting, every professional skill you have, you will use. In the beginning, we’d use the lever of compliance because, in their world, I started to realise compliance was in some way our product version of security. When we were able to say, okay, we were PCI-compliant or we're meeting SOX obligations or GLB, we were providing a level of assurance to the customers that they could actually use to gain more trust with and, therefore, productise in their fashion. I realised, okay, for so long, I've been working in this space where we weren't sure how to quantify what we were doing. But this helps give a label and a price to it.

[00:08:07] GUY PODJARNY: Yes. No, it’s a clear cut of accomplishment as well.

[00:08:10] JULIE TSAI: Absolutely.

[00:08:10] GUY PODJARNY: You achieve compliance versus you reduce risk, which is this nebulous thing that's out there. Yes.

[00:08:14] JULIE TSAI: Absolutely. Yes. That is 100% true. Security is both science and art, right? We have to find these very objective results and metrics. At the same time, it becomes a judgment call. Is that good enough? Was that risk brought down just enough? What do I think is going to happen in the future? Eventually, over time, I started seeing that, hey, some of these companies were selling our reliability and were building products that are security-focused. There’s something beyond just the compliance piece of it.

Once we could get past compliance, it was also about elevating the conversation to the next level, right? Once the leaders understood, hey, this is important. This is the language around compliance and security. By the way, it's still not enough because if you look at all these companies that did get breached, they were compliant. We all know on the back end of it that the paperwork is only part of it.

[00:09:05] GUY PODJARNY: Yes. Compliance is a minimum bar. It's not a solution bar.

[00:09:08] JULIE TSAI: Yes, that's right. You haven't gotten there. It's your starting point.

[00:09:11] GUY PODJARNY: To an extent, I guess they're sort of on the same continuum. I really like – it kind of resonates with me, the notion that the first element is compliance. It's a stick. To an extent, you have to be compliant. You don't really have much choice. Sometimes, you do have the choice. I know we've gone through at Snyk. We've gone through the SOC 2 compliance. Sometimes, some companies demanded that.

But for many others, once we have it, it's almost this stamp that says, “Okay, we are secure,” or sort of we're at a certain calibre of security that actually helped foster the conversation. That's a compliance version of the second bit that you're referring to. But in general, the more you boast humbly, but that you can demonstrate your security posture, then you can communicate to your customers that you are a trustworthy partner to work with.

[00:09:50] JULIE TSAI: That's right. I think also along that continuum of bringing it past the stick into something positive, right? Because we're dealing with things that people generally don't like to think about. They don't necessarily like to think about penalties, regulation, or failure, making it a point of pride. What we found was, eventually, when people were starting to solve really hard problems and sometimes under difficult circumstances, man-made or not, you could find that sense of pride in what they had accomplished and the work.

Over time, even if they hadn't identified as security people before, like me and like a lot of us, they would say, “Actually, I really care about this. I am passionate about security,” so finding that passion in the activity of it. Once they start to identify with it from that standpoint like, “This is something that's important to me, and I'm proud of it,” you're able to bring the practice to where you want because what you really want is people to internalise it at that level of how can I better do security and think about security for our results and for the customer. It's not about the book. It's not about passing the cert. It's not about the grade. It's like how do we do this better? So it's getting it moving along that continuum.

[00:10:55] GUY PODJARNY: Yes. No, I love that, making it so – I think oftentimes you talk about security in the context of shame, and pride is basically the reverse of it. You don't – it's not – yes, I love that, making security as a source of pride. I guess that's the evolution of techniques, and this is in the context of this company that is still relatively nimble. You don't need too many procedures. You just need to sort of make it make sense as you're driving because it's this 200-ish person company.

[00:11:18] JULIE TSAI: Yes, that's right. In that context, I could see it start to happen in the individuals. But in my career, I got to see it in greater skill as I moved through into bigger companies. I moved through into companies that were either in the process of going to IPO or passing that SOX compliance. You could see, okay, now the problems are starting to get not just about how do I solve this locally within my own division, but how do I get these other departments to also move down this parallel work stream of equally complex complicated stuff, and we all meet at the same time. We have all these different divisions that are responsible for the people and the financing. They also have to be on board with it, too.

I spent about three years over at Walmart eCommerce. One of the things that I saw my CISO at the time do very, very smartly was the alignment of the interests between the development teams and security teams. They were actually measured on their performance for their security bugs. Security bugs were qualified as quality defects, not just security. It was not a separate thing. It spoke with their language and their values, and it grafted onto their existing process. I think getting back to what is it that a particular group feels pride in or values, you could see that continue to resonate. Once a particular group was becoming an example of how to do things, they would want to continue that example, and you could see that evolution.

[00:12:39] GUY PODJARNY: The quality bugs as security bugs makes a lot of sense, streamlines the process. How did the prioritization of those bugs come into play? Do you know how –

[00:12:50] JULIE TSAI: Great question.

[00:12:51] GUY PODJARNY: To an extent, if there was a burning customer problem, then that bug gets prioritised more than if your colleague just gets annoyed by it.

[00:12:58] JULIE TSAI: It can. I think this is where you need to have good expertise on both sides and then the alignment of the interests, right? The groups would have their own prioritization standards and finding what those standards were in terms of what do you consider a P0, have to get done within one or two days or as soon as possible? What do you consider a P1, get done within a week, whatever that vernacular was? Being able to slipstream into that becomes really important to say, “I need this group to evaluate. We consider this security bug to be an ASAP bug, so it needs to graft onto how you guys treat ASAP issues, whether or not this is new or par for the course.”

In the security world, there are a lot of objective definitions that are debated but are out there, especially with regards to NIST standards and CVEs and vulnerabilities. As much as possible, I usually encourage my teams to borrow from best-in-class standards that we know, right? If we can leverage what we know to be the definitions within this, the CVSS ratings, and then give whatever mitigations that we know to be context and prioritise there and say, “Okay, that's where I want this to land ultimately.” Take the objective standard, figure out how it maps in our world, and then give that over to the developer teams and the infrastructure teams.

If we have to have a discussion or negotiation with the leadership there on why that is or isn't considered important, then it does have to click up to that management level. But the thinking behind it is that it's not your priority versus my priority. It's a, hey, we understand this to be a objective priority. What does it take to fit that within your group's urgency?

[00:14:40] GUY PODJARNY: Yes. That makes a lot of sense. If I echo it back, you can say something like within each of these silos for the infrastructure world or the Ops world, it might be more like an outage as a P0. Your servers are down.

[00:14:49] JULIE TSAI: Exactly. That’s a liability.

[00:14:51] GUY PODJARNY: On the development side, it might be a major bug that is blocking a big deal, and that might be your sort of top element. It might be a CVSS 10-type vulnerability from the security side, a standardised score, sort of severity score of the top rank. But at the end of the day, once you've classified it as a P0, let's say, in that context, then your methodology of dealing with P0s doesn't need to defer between whatever these three buckets or whatever other buckets that you have for those.

[00:15:18] JULIE TSAI: Yes, that's right. I think a big part of us being able to leverage known art on the standards is being able to take some of the egos out of it and also help people develop their thinking about these things. Because a lot times, we're working with a lot of really smart disciplined engineers, and they're going to come up with their own system, which is great for the thinking process. Again, it's well-trod territory. Take what's out there and then develop on top of that, as opposed to starting completely new.

[00:15:47] GUY PODJARNY: Yes. At the end of the day, this is not that dissimilar from evolution of engineering practices or others. When you're a 200-person company or even before that, right? When you're a 20-person company, organisation, most of the people are engineers. You start by just having smart people, and they know what good looks like. Later on, they need to define what good looks like, but they still kind of work a little bit more methodically. As you grow, you have to at least get better at the definition and communication of what good looks like, even if you keep the autonomy of reaching that goodness level within every one of these groups. You still need better definitions.

I guess you sort of started those talking about alignment of organisations and tracking it. Is this also the way that these metrics basically got reported up, security issues basically reported? Or sort of from a development perspective, my security bugs were just counted alongside my outages and my quality bugs?

[00:16:41] JULIE TSAI: Yes. They were very much reported up and also reported out. In terms of how we manage things in the large enterprise company, there were regular SLA measurements in terms of how well they had accomplished meeting the bar for the different criticality levels, for the criticals versus highs and whatnot, and who is adhering to them or over-performing, and which groups were not?

To some extent, there was alignment at least on, hey, this is going to feed into how you measure your group's performance. Managers were left to have their own discretion in terms of some of the mechanics on that. But there was also a sort of fun competition aspect of it in a place that allowed it, where we had a leaderboard of, hey, these are the groups that are ahead. These are the groups that are not. It was not a problem in terms of punitiveness. It was just –

[00:17:31] GUY PODJARNY: Yes, just gamification, trying to sort of get some competitive juices going to get people running for –

[00:17:37] JULIE TSAI: That's right, which I think is an important part of it. It has to fit with the culture that you have and be able to keep people constructively oriented.

[00:17:45] GUY PODJARNY: The notion of engaging people, and maybe back to your previous comment of making it a source of pride, you need to somehow be proud of something, so it might be intrinsic. Have you seen good recognition models that work for security? I mean, you've been in all these elements where there are good ways to reward security that you've seen work well?

[00:18:06] JULIE TSAI: Yes. I believe I've seen pockets of it. We had a program as part of sort of the security enthusiasts and embeds through the org that might be interested. We would have some regular meetings or help give a more – we actually would subsidise trainings for certs for CSSLPs or CISSPs. That was one incentive. There's various ways of – again, back to the hard and the soft recognition, right?

I think that it is really, really important, especially once your organisation has gotten to the point where security is its own division because a lot of times, security is a completely business-critical and mission-critical function. But if the organisation is separate, at this point, a lot of what has to happen has to happen through influence and goodwill to the rest of the org, to the groups that are running the systems and the people.

Every opportunity that you have to have a good interaction, as opposed to something that's felt to be a negative or a penalising interaction, is really important to leverage. I think the models I've seen at different companies depends on what's valued there, right? Some of the more startup-y culture companies might be really interested in having public accolades in various forms or schwag or things that are kind of indicators of stuff, right? Companies where things are more formalised recognition, maybe things like performance recognition, bonuses, and that kind of thing.

I think it's important at that point as a security leader or a security professional when you come in to really look at what seems to drive behaviour at that company and also the makeup of the employees there because it is going to change. It depends on many things like the profile and the character of the folks that are there.

[00:19:55] GUY PODJARNY: Yes. But you do come back to the notion of alignment, the notion of not just alignment within the organisation but alignment of the means through which you incentivise or you drive security practices with the way that you incentivise other stuff in the organisation and other activities, which makes a lot of sense. I love sort of these patterns around almost this evolution of security or evolution of security practices. It feels like your path took you from the small companies where security needs to permeate. There's no nobody's job. No individual person's job. So it needs to be an extent everybody's job.

But a lot of it is done through these informal interactions through growing, creating maybe a security department, and then going to a place in which the organisation is already large. You need to have your sort of tentacles out, right? You need to have some champions, some advocates that are to an extent almost coming back to that first tier, which is you need enough people that are local that are sort of embedded within a smaller team. Back to that original small team that get it done, despite the fact that it's not really their job. You're now permeating many, many small teams.

[00:20:59] JULIE TSAI: It is an excellent point because, ultimately, what good looks like at the end is not too different in the practice. Everyone has to have internalised it. I think that with security becoming a deeper and more pervasive challenge nowadays, it really does become everybody's job on some level. The engineering groups get the messaging and the focus first because so much of what we touch in for the product has a security implication, and we're going to get that direct feedback.

If you think about it really, every group that, say, brings in a vendor, that's a vector for you. You look at how many companies had issues with third parties or contractors or consultants. Every part of the business has a role to play. I think that a lot of what you're trying to do in creating a larger company that works, especially for security, is replicating the healthy intuitive behaviours that happen in those small environments. In a startup that's on the right trajectory, the people understand each other, understand the mission, and almost through osmosis start to absorb each other's priorities, right?

Then as the companies get bigger and you get more distance in the people, as well as just what's going on and the knowledge levels, things have to work on their own. At that point, the structures, the incentives, the goals, the communication channels all have to start reinforcing what was there in the beginning in the intuitive stage.

[00:22:19] GUY PODJARNY: Yes. I guess the mistake that you might make which, frankly, kind of goes counter to agility as a whole is that you can just continue that original structuring and take it to the extreme because you've gone from osmosis to giving it more structure. But when you're larger, the continuation of that is not even more structure and mandated path. It's actually to say, okay, you've given structure. I've grown very fond of the word scaffolding. But basically, that structure needs to become scaffolding, and you need people climbing on them. You need some element of these sort of local champions and knowledge that permeates it, which you see a lot of companies fall into that mistake of the path.

[00:22:57] JULIE TSAI: Oh, absolutely. It is very, very common. You're starting to hear about the various companies have those kinds of different groups. The good thing for security is that at some stage of career, a lot of engineers want to be involved with security. There's an aspect of it where even just being part of the evangelical or embed aspect of it becomes appealing to folks. That's great for us in terms of being able to reach out. The part that we have to make sure is really there is that we're not mistaking soft influence for the hard functional lines and authority that's also needed to be there.

I think that you bring up a really great point about it's not just the things or the structures that are important. It's the behaviours and the internalised actions that are really what you're looking for in security. I think that's a big reason why there's so much around DevSecOps these days. What is it about DevOps that was allowing people to build infrastructure and scale in an agile and in a cost-efficient way? Well, it was because all of these ideas and technical controls were actually embedded and internalised upstream. It's not that they went away.

Similarly, for security, I think both from that technical piece of it, it does need to be embedded upstream. But then as you're looking at how a security programming is functioning for a company, the things you end up wanting to measure ultimately aren't just the activities of your security team. I always start out and say, okay, great that we got a report of what we did, and the number of touches we had, the number of tickets we handled, the number of this and that. Ultimately, how many security incidents were averted? How many were reported? How much did we minimise by shrinking our attack surface? Ultimately, you want to be looking at the risk and the overall impact, not just the activities that we did to get there.

I think in that way, companies that are comfortable with moving quickly and evolving quickly from an org level, as well as a technical level, have an advantage because over time, a lot of the structures that emerged in the industrial age were designed for a non-networked world, right? So we have a lot of procedures and artefacts that are built up around this idea of like what are we doing to show control and show rigour in a non-networked world. In a digitally connected and interconnected world –

[00:25:14] GUY PODJARNY: And fast-moving.

[00:25:14] JULIE TSAI: It is very fast-moving. Both your attacks are fast-moving. Your defences are fast-moving. If I really did the control correctly, how many artefacts do I need along the way after the fact?

[00:25:24] GUY PODJARNY: Yes. I think it's oftentimes indeed sort of the essence of the eventual outcome of DevSecOps is agility. But, yes, I'm entirely with you. I had this whole article talking about how DevSecOps is oftentimes confused or sort of – I don't know if confused because it's a buzzword, so you can interpret it however you want. But interpret it as really at the end of the day security for DevOps technologies or security for DevOps methodologies. It’s container security or it's microservice security or container security. But sort of the true holy grail is really the DevOps-shared ownership that you want to embrace as you want to say, okay, this is about we all own it together. It's everybody's problem. It's inclusive security. Again, it comes back to that osmosis.

I think this is kind of great evolution. I think it's sort of naturally builds it. It sounds like a super interesting kind of a career and flow here. I have a whole bunch of other questions to ask you, but I think we're sort of around the edge of the time. Before I let you go, I'd like to ask every guest on the show, if you had one kind of piece of advice or maybe a pet peeve or something that you would want to share or tell teams that want to level up or improve their approach to security, what would that be?

[00:26:36] JULIE TSAI: I would say it's encapsulated well by a joke that my infrastructure friend told me about security. He said, “How do you tell the difference between a good security person and a bad security person? You can't. They both say no.” As a former infrastructure person, I said that just sucks. I got into this to build stuff, not just to tell people no. But it kind of reminds me of the whole point of this is for us to be fuelling creativity and product for the customer in a secure way, right? So being able to keep that bigger picture in mind on all sides, I think, is for me that northern star.

[00:27:16] GUY PODJARNY: Yes. Remember that at the end of the day, you're there to drive business, to make it successful. Julie, thanks a lot for the great conversation. Thanks for coming on the show.

[00:27:24] JULIE TSAI: Oh, you're welcome. It was a pleasure.

[00:27:26] GUY PODJARNY: Thanks, everybody, for tuning in, and I hope you join us for the next one.

[END OF INTERVIEW]

[00:27:30] GUY PODJARNY: That's all we have time for today. If you'd like to come on as a guest on this show or want us to cover a specific topic, find us on Twitter @thesecuredev. To learn more about Heavybit, browse to heavybit.com. You can find this podcast and many other great ones, as well as over 100 videos about building developer tooling companies given by top experts in the field.