Dynamic Authorization - The Evolution of Access Controls With Aren Sandersen

Aren Sandersen: "Security is a very broad brush, I would say. I think ease of use and security are probably going to be at odds for a while until we come up with something a lot tougher than what we have now for verifying identity. There's many companies where I would come in to help out, and I would ask for access to production systems. I would get an email of the private key. Security is all about trade-offs at every stage. Security is almost by definition brought in later than it should be. It's an art to figure out when the right time to bring these tools in is, but you got to know that they exist. There are no silver bullets."

[INTRODUCTION]

[0:00:34] Guy Podjarny: Hi, I'm Guy Podjarny, CEO and Co-Founder of Snyk. You're listening to The Secure Developer, a podcast about security for developers covering security tools and practices you can and should adopt into your development workflow.

The Secure Developer is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market. For more information, visit heavybit.com. If you're interested in being a guest on this show, or if you would like to suggest a topic for us to discuss, find us on Twitter, @thesecuredev.

[INTERVIEW]

[0:01:06] Guy Podjarny: Hello, everybody. Welcome back to The Secure Developer. Today, we have Aren Sandersen with us. Hi, Aren.

[0:01:10] Aren Sandersen: Hello, thanks for having me.

[0:01:12] Guy Podjarny: We are going to talk about a bunch of things, including access controls, and how do you control all these pesky users you have on your team and still stay secure. Aren, thanks for coming on. Can I ask you just as we get started to say a little bit about yourself? What's your background? How you got into security?

[0:01:26] Aren Sandersen: Absolutely. I'm Aren. I am the founder of Foxpass. I started out doing software development on server-side teams for about 10 years. Then, I migrated more and more toward the operational side, helping manage data centres, and got really an in-depth in that space. I ended up running tech ops and IT teams for several social networks. Then, I did a period of DevOps, scalability and security consulting, where I worked with a lot of startups struggling with these areas. From my experience at Pinterest, which is one of the companies I was at, and at all of those startups, I realised there's a big gap around access control that a lot of companies face. We created Foxpass to sort of solve all of those access control and identity needs for the infrastructure. Who can be on what servers, VPNs, wireless networks, what can they do, how long can they be there, that sort of stuff.

[0:02:15] Guy Podjarny: Okay. Cool. Sounds like you're sort of personally going through the dev, then DevOps, then DevOpsSec kind of revolution, sort of evolving your skill set. Let's sort of go back a little bit. I mean, you worked for a bunch of companies, maybe we start around there. You mentioned you worked at Pinterest. In that role, you were saying you're a sort of more in an ops role there. What was that experience like as it came to security? What type of security activity? What's going on?

[0:02:43] Aren Sandersen: Yes. I would say, when I got there, there's nothing wrong with it. But there's always room for improvement, and especially the rate at which the company was growing around when I got there, and especially after. There's a lot of improvement on how do we onboard, people quickly get them access to the things they need quickly. If someone needs to change their SSH key, how can we do that quickly. Just so we really can focus more on getting people up and running when they onboard, getting working on the product, and trying to just enable them to work quickly.

[0:03:14] Guy Podjarny: Was there a dedicated security team at the time?

[0:03:17] Aren Sandersen: There was not, like many young companies. It really fell on the developers early on. Then, as we added more DevOps types folks, it was in their role. A dedicated security team didn't come until years later.

[0:03:29] Guy Podjarny: The security responsibility was just generally shared, or did it fall more, like I said, as the ops team came up, it kind of became more ops responsibility versus dev?

[0:03:37] Aren Sandersen: I mean, security is a very broad brush, I would say, that there's usually a pretty clear division early on when there's no overarching security team that AppSec falls into the more senior app developers. And OpSec falls to those who've had – very senior devs or have had more DevOps or DevOpsSec experience.

[0:04:01] Guy Podjarny: It's funny, to an extent, young companies, because of lack of alternatives really sort of operates at the holy grail mode of the large companies. Because large companies, you kind of go through this crunch, where you build a security team, and you want them to give guidance. Then, much of that security team's time is to try and get engineers in office to actually embrace, and saw the security practices in their sort of core competence. When there's somebody with that role, people may confuse that as, "It's not my job. It's that sort of person's job.” Okay, cool. So you're at Pinterest, I know you have security as kind of run as a part of those teams. You're not in a security title at the time, so you're dealing more with OpSec at the time.

[0:04:38] Aren Sandersen: That's right. It was OpSec, DevOps, I wrote some app code as well. In the early days at a company, I wear a lot of hats.

[0:04:45] Guy Podjarny: Yes, anything that's needed. That sort of evolved then. After that, you went on to do kind of Ops and security consulting?

[0:04:53] Aren Sandersen: Yes. I mean, based on many of the challenges we face, scaling up Pinterest, especially on the sort of the operation side. I went ahead and helped a bunch of young startups figure out their operational challenges, whether that was scalability, whether that was security, whether that was just foundational things. I did that for a little over a year.
[0:05:13] Guy Podjarny: How much was a security component in the ask? Like when people went out to – when they found you, then they sort of told you what they wanted to help, what they believed they needed help with. How big a role, if at all was security in those project definitions?

[0:05:29] Aren Sandersen: I would say they were rarely asked upfront. It was rarely, "Hey, come in, and tell us what we're doing from a security perspective." More often, what I heard was, "Can you just come in and look around and give us improvement suggestions across the board?" I would always make a point of honing in on security and highlighting areas for improvement, as much as I would all the other operational challenges that a company might have.

[0:05:50] Guy Podjarny: To be sort of engaged on the Ops front, so come to help us operate better. But from your perspective, that includes security, even if it wasn't stated in the bullet items.

[0:06:01] Aren Sandersen: That's right. Oftentimes, it was pretty obvious from the get go if security was an issue for this company. Because I onboard as a as a contractor, and get to go through the flow of how new employees onboard, how they get access to the production systems. What other systems they would have access to that didn't go through any sort of access control. It was pretty easy to write that report because I was living it.

[0:06:23] Guy Podjarny: Yes. I guess, what were the insights you had? I mean, maybe now you can contrast a little bit. You've sort of seen Pinterest that a certain size, you've seen a bunch of companies that you worked with, that sizes that were similar, maybe smaller, as you're building out. What did you see was typically handled well from a security perspective versus things that that weren't.

[0:06:41] Aren Sandersen: I would say, almost all these companies go through the same arc, which, I think is pretty common for all companies of this size. You start out writing your app, somebody spins up a couple Amazon machines, gets their app running, and then you add more people. Normally, they're all sharing the same credentials, until you get to this certain point where someone decides to bring in some sort of configuration management that's usually fairly late. Then, that the company keeps growing, then you just keep iterating security slowly. Then, at some point, they'll bring in a dedicated security team.

[0:07:14] Guy Podjarny: You talked about giving access to the systems and things like that. We're going to sort of deep dive into that topic shortly. For other practices, you're sort of seeing in terms of operations security, what were typical types of mistakes that you would walk in and you would say, "Quite obviously, like pretty much all the other companies I've worked with, I can see you were doing X wrong."?

[0:07:35] Aren Sandersen: Yes. I think there's many companies where I would come in to help out and I would ask for access to production systems. I would get an email of the private key for the Ubuntu or the ec2-user. This happens a lot. I know, it's easy for many listeners to sort of scoff at that, but it's extremely common because that's the way Amazon sets you up. They don't really tell you that this public-private key pair that they gave you is kind of just for you, because there's no way to add more users. It's in many ways, the cloud has been a setback from an operational security perspective in that regard, that there's a lot of foundational security that is missing when you get that public and private key pair from your first Amazon machine.

[0:08:16] Guy Podjarny: Clearly, that sort of statement about the setback is an interesting one. Cloud is obviously a productivity boost. I think there's probably no arguing that right, that it made us produce more faster.

[0:08:27] Aren Sandersen: That's right.

[0:08:27] Guy Podjarny: The notion about access is the ability to access all of these machines, and whether we're sort of more fast and loose with it. Do you see that as something that's cloud specific? Or, do you think that's a symptom of the speed? Is cloud the thing that you sort of feel is setting us back or is it just about moving fast?

[0:08:46] Aren Sandersen: I wouldn't say it's the cloud, specifically. I just think it's the current process of what developers go through when spinning up a cloud server. I think the cloud overall as a whole actually makes things more secure. But I think the current process sets people up where they think they're doing everything right because they're in the cloud. But in reality, there's a big gap between what they have in their hands when they spin up that machine and where they need to be to make a secure environment from an access control perspective.

[0:09:10] Guy Podjarny: It's interesting, I think, sometimes around the ease of use for it. Security and usability are oftentimes at odds. It's oftentimes if you make something easy, you sometimes make it less secure, or the other way around. If you want to sort of hardened access controls, if you want to harden the ability to do actions, then that comes at the expense of usability. We see this across the board. We see this with the consumption of packages and publishing a new source code package in many of these platforms, in npm, and the likes. You don't need a password for it. If somebody got on your machine, they can now publish stuff on your behalf. To an extent, that reduces security. At the same time, it's easy to publish. So, the community thrives because you publish a lot of this type of code.

Similarly, probably for sort of cloud access, you want the entire team to be enabled. It's DevOp, it's everybody's responsibility, or everybody's empowered, if you will, to ship stuff, to access these machines, to grow them. In the process of making it more usable, you lose maybe some of those controls that you had, as opposed to maybe the hardcore. You've got a physical machine, and you're sort of guarding access to it. It's slower, but it's safer in some respects.

[0:10:19] Aren Sandersen: I think security and ease of use are always going to be at odds. You have to put up some gates. You can't have a process that anyone can do anything at any time without any checks. I think even some of the more basic things that are must-haves, two-factor auth, for example. I think that process has to exist. It's a little annoying when you're trying to do something and you're stopped, you need to dig out your phone, or to enter the token, or whatever. But, I think the right balance is important, and the right balance oftentimes depends on the what stage of the company you're at, what are the security requirements of your particular company at your particular stage. But I think ease of use and security are probably going to be at odds for a while until we come up with something a lot different than what we have now for verifying identity.

[0:11:02] Guy Podjarny: Yes. I think, playing devil's advocate a little bit here, just to sort of flesh out the thought. On one hand, when everybody has access to everything, sort of in this sort of extreme and very common case, in platforms. Then, anybody can make bad mistakes. Also, if anyone's machine gets compromised or something like that, then, people would have very broad access. At the same time, there's oftentimes when you talk about the improvements in security that continuous processes have, is that you are fast to fix. When there is a problem, everybody has the ability to fix it if the problem occurred for different reasons.

I don't know. It's sort of hard sometimes to counter. You're probably right, that it's just about the balancing act. Maybe that's the even security angle, not just usability that pulls you in the other side, is you still want to make the security sufficiently easy that you can respond quickly. So, it's the actual security demand.

[0:11:54] Aren Sandersen: Yes. I mean, we've talked before about the arc that startups go through around security. The early days, every engineer does have access to everything. When things break, you don't know who's around, who can fix it. But as you split up into separate teams, which companies do when they're midsize, then, it might be time to start restricting which engineers have access to push code, to which servers to access which servers. If you're a company where you have auditors, or you have very sensitive data, it's going to be a requirement that you have role-based access control and start limiting who can do what. Most of those companies, they struggle to build those tools.

[0:12:29] Guy Podjarny: Let's sort of dive in and dig into access control. Before, I have sort of a bunch of questions that dig in. Let's start to sort of give it a definition. Can you kind of scope this for us a little bit|? We're talking about access to machines. What do you see as being included under that title of access control?

[0:12:44] Aren Sandersen: I mean, access control covers a huge spectrum. I mean, you've got companies out there that control access to third-party websites. Those are the Oktas, and OneLogins, and VDMs of the world. Where your company credentials control what other outside websites you can get to. Then, you've got sort of your internal access control, who can be on your wireless network, who can access a VPN, who can be on your admin portal, who can be on your servers for the engineers. There's a huge spectrum in the access control world.

[0:13:15] Guy Podjarny: So, kind of defining this a little bit. This is kind of the broader access control. When you talk about a startup, what would you say are the sort of primary areas that I need to think of. I'm starting a company, whatever this young company is, sort of whatever, 10, 20, 30 persons, or even earlier on. As they just started, just a handful of people. What should I think about it and what should I explicitly not think about right now, shouldn't be a focus area?

[0:13:41] Aren Sandersen: I think you should think about, if this person left my company today, what would they take with them? If this person left now if they're an engineer, would they still be able to access your servers from their home? Would they still be able to access your DNS portal, where they could cause some problems with your name servers if they wanted to if some of those have some of those have a weak TTL? Those can be very painful to fix. Even if that means that a very small company just writing this stuff down, what's your off-boarding plan. Just so you know that there's nothing this person can do to cause you problems later.

I think, as the company grows, an off-boarding plan, on paper doesn't work as well, as having more formal processes. That's when you bring in companies like I mentioned before, companies like us to make one-click off-boarding or roll changing much easier.

[0:14:28] Guy Podjarny: Yes. So there's less human error in the process. Probably something to keep in mind in those cases is, it's not just a matter of trust. It's not just about whether that person is nefarious, and is going to do something badly. It's also about just sort of your attack surface. If you have somebody that left the company, they may accidentally give access to those systems to another party, they may get infected with some malware, or something like that, that gives us those machines. It's about reducing your attack surface.

[0:14:54] Aren Sandersen: I mean, that's a really, really good point. I mean, you tend to think of it in terms of what could this person do if they were nefarious. But really, the question is, what could this person do if their account was compromised, if their credentials were compromised, if they were phished. Humans are normally the weakest link. If this person was fished, what access do they have, and how can you create policies where their access is low until they need higher access?

If they don't need to be on your credit card database server, they probably shouldn't without prior approval, whether that's from a peer, whether that's from a Jira ticket being open, or whatever. Because if those keys were stolen, it's nice to know that, hey, well, they didn't have access to anything really sensitive, so the attack surface was much lower.

[0:15:35] Guy Podjarny: Okay. I think that's a really kind of good way to think about it when you just get started, is to say, what happens if somebody leaves. Maybe there's a little bit more of that as well as you sort of figure out even your sort of team culture, and structure as well, and just sort of understand it. Step one, think about what happens if that person leaves, even if you're sort of fully trusty in the employees as they're sort of still in the team. Then, sort of step two sounds like, write down and off-boarding plan, and then later on, structure it, automate it, so that there's no human error or omissions. When that happens, those are I think good steps. What's the next level? Look, you've done that, that's about people joining, leaving. What's the next step up in sort of access control?

[0:16:15] Aren Sandersen: Yes. We've mentioned it just briefly, which is what we call dynamic access control. Your access is zero or very low until it needs to be increased. Some examples we give are, you don't have pseudo access to any machine or some machines unless you happen to be on call that week. Then, your access is increased, and then it's automatically decreased. Or, when you do give access, it has to be approved, like we said, appear a manager, but it's always time bound. You can only be on that server for one hour, then that access is reduced back to normal.

What you don't end up with is what we call the master key. If you've been at a company a year, two years, your key gets you in any lock. If that key gets compromised, then that's very bad. If instead, your access always was lowered when you're off a project, or when the time expired, or whatever. Then, that keeps the attack surface much, much lower. That's really where we see the future of infrastructure access control, is everything is time bound.

[0:17:19] Guy Podjarny: I think that's a really, really good kind of destination. It sounds like one where it really is about tooling. Because technically, you could have implemented a human process to do everything you've just described. Except, that's very impractical. It slows everybody down. But also, you need a certain size company to do it. While on the flip side, if this was sufficiently easy, if it was just the two-factor auth, it's just a 10% overhead on the process of giving somebody the master key. Then, why not, assuming the costs are correct, as well. But, I think a lot of it is about that effort.

[0:17:51] Aren Sandersen: Yes, that's right. I think, access control is a manual process. Updating someone's access, upsetting someone's SSH key means updating a Git Repo, pushing that up, setting that to the chef server or puppet server. Then, if it was supposed to be temporary access, you'd have to remember to undo that, which is, again, a lot of overhead. It's a lot of people's time. If we build automated processes, you get better security, because you have these tools in place, you don't forget to change the permissions if they're time bound, because it's automatic. I think there's a lot of improvements over the current state of the art tools in this area around access control.

[0:18:24] Guy Podjarny: Many of these controls we talked about are really good when you have some sort of central user management portal. Oftentimes, you have multiple user management systems. If I use something simple, you might have a GitHub account, and you maybe use Google Apps. Maybe that is sort of a central focus for a bunch of the other things. Maybe you have some Jira account, or there might even be like Twitter account or stuff like that, if it's about social media access. Not all of them kind of connects together. Is there kind of a good practice that you can do to try and wrangle these things.

[0:18:57] Aren Sandersen: There are a lot of tools out there, I think. On the low end is a common list of passwords in a spreadsheet. Not really recommended, but at least know where to look. There's also password management tools that have group features. LastPass is one of them, 1Password is another. Then, you can move up to things like Meldium, which is very similar to those previous tools. Then, you're looking at the federated access systems like Okta, OneLogin. They have abilities to tie all of those systems together.

They'll hold on to the passwords. For accounts that don't have federated access, they'll provide federated access for sites that have it. They also will do provisioning, so you add a new user to the marketing team, they're going to get access to Twitter, they're going to get access to your ad management portals automatically, but someone on another team won't have that access.

[0:19:48] Guy Podjarny: Cool. That sort of evolution is a good sort of listing, and probably sort of the key question would be what's the right time for me to sort of invest in that type of system and maybe it's when the pain is sufficiently high.

[0:19:58] Aren Sandersen: It always turns out when the pain is sufficiently high. The right time is actually day one. I don't think anyone actually does that. I really can't fault them. I think product market fit is probably a startups first priority. If you don't have that, then what's the point of all the security? But security is almost by definition brought in later than it should be.

[0:20:18] Guy Podjarny: You need to build assets that are worth protecting sometimes, before we sort of invest in that protection. At the same time, you have to manage that risk about saying, what could shut you down tomorrow. For instance, losing your customers data, or things like that could make those assets dissipate very, very quickly, those hard-earned assets.

[0:20:36] Aren Sandersen: I'd say, security is all about tradeoffs at every stage. That's one of many.

[0:20:42] Guy Podjarny: Cool. I guess, describe, just to echo back some of these flows of evolution in your sort of access control level-ups, going from understanding off-boarding, and what happens if that person leaves, to automating that processes. Documenting, then automating, to simplifying onboarding, and then having dynamic, as fine-tuned as possible minimal permissions. Maybe if I understood it correct, there's actually two tiers here. The first is minimum permissions, and then there's temporary permissions on these different systems, which probably go hand in hand with tools. This is sort of an area where Foxpass kind of can help you out or is this in the OneLogin space.

[0:21:17] Aren Sandersen: The temporary access, specifically around servers, because really, our focus is all infrastructure. Temporary access, or temporary group membership on servers is really something that we push. We believe that no company should have those master keys floating around, so we give you all the tools to make that happen.

[0:21:35] Guy Podjarny: Cool. That definitely is sort of a good path. Then, we talk about the future a little bit in the sort of the dynamic path. Is there a lot of evolution as well in tooling around understanding that it happened? In many security conversations today, there's the prevention aspect, but then there's also a detect quickly when a problem occurred. Is there an equivalence to that and the world of access control, of just being focused on anomalies and the like around access?

[0:22:02] Aren Sandersen: I think you have to combine a bunch of different sources, which is the nice thing about I think the current DevOps ecosystem, is there's APIs everywhere. So, you would make sure that products you use have logging output that you can get your hands on. So you can look for failed logins, or odd-looking logins. You can use a lot of cloud-based tools to aggregate that data, and then build reports on it. I think there are a lot of silos out there. We'd be a silo of login information. There'd be another silo of network traffic, to look for anomalies. There are a lot of companies out there who are doing a good job of mining that data from all these different sources and looking for patterns. I think it's really about the combination of all the pieces.

[0:22:43] Guy Podjarny: I really liked that idea. You can probably sort of dashboard, just like you measure a whole bunch of the kind of ops metrics, just some of these access-related stats, so you can sort of spot anomalies about failed logins, around whatever how long between logins, or things like that to just draw attention to it, so you can inspect, if that seems legit, or if there's a problem there.

[0:23:03] Aren Sandersen: Yes. If you see a user trying to access a machine that they should have long lost access to, that looks pretty weird, that's something that can be flagged in one of these alerting systems.

[0:23:13] Guy Podjarny: Yes. I think it was really, really good sort of education around access control, and I guess, where we're evolving. One titbit that does come up in my mind when you talk about access controls and these keys is just the relationship of access control to secret management. Because with secret management, oftentimes, the secrets are tokens to access, the different systems. How in your mind do you delineate those two worlds?

[0:23:34] Aren Sandersen: It's a good question. Foxpass itself doesn't do anything with secrets management, per se. Normally, secrets management is API keys that your application needs to talk to third-party services. There are a lot of those that float out there. From our world, the biggest secret is actually on the user side, which is that SSH key on their laptop. How do you protect that? We talked about disk encryption, a lot of things on the user side. But in terms of secrets management in a production system, you want to look at tools like HashiCorp vault. There are a lot of companies who have built systems around the key storage systems built into AWS. There's a big world out there of those.

[0:24:12] Guy Podjarny: It sort of keys that are not necessarily associated with an identity. They're just sort of access. Well, I guess there's still access control, their authorisation means, but they're not associated to a person. Maybe that's where terminology fails us a little bit, around that, because there's still control access, but they don't represent a person behind it.

[0:24:29] Aren Sandersen: That's right.

[0:24:30] Guy Podjarny: Cool. Well, before I let you off here, I often ask my guests. If there was one thing today that's kind of your pet peeve or your key recommendation, if you're talking to sort of a security team that's trying to up their game in the security space, what's your one recommendation for them to do so?

[0:24:48] Aren Sandersen: When I'm talking to developers about sort of operational security, my biggest tip is just know your tools. Get to know where your cloud provider leaves off in terms of security, what's missing. A lot of that have documentation about, okay, we've left you here, but we recommend a bastion host, we recommend a VPN, we recommend to do all these things. Your work is not done. Amazon has this pretty fine line of, "We're responsible for everything below this line. You're responsible for everything above." You got to know those, and then you have to know what tools will fill those gaps. It's an art to figure out when the right time to bring these tools in, is, but you got to know that they exist. I think. a lot of security does boil down to knowing how to use the tools that you're using, and where their weaknesses are. Otherwise, you'll just be blindsided.

[0:25:35] Guy Podjarny: Yes, it's an excellent recommendation. I think. much of software development today is assembly. You're sort of pulling together all these different services, and tools, and packages, and you put them together. When you put them together, you need to understand the seams and understand how does the eventual puzzle work together well. That is sometimes almost goes without saying from a functional perspective because otherwise, it wouldn't work. But it's very easy to overlook that from a security perspective.

[0:26:00] Aren Sandersen: Yes, and there are no silver bullets. You have to know the limitations of what you're using.

[0:26:05] Guy Podjarny: Yes, for sure. Cool. Well, Aren, thanks. Thanks a lot for coming on the show. This was fun.

[0:26:10] Aren Sandersen: Thanks so much for having me.

[0:26:11] Guy Podjarny: If somebody's trying to find you on the Twitter sphere or online, what's the best way to contact you?

[0:26:16] Aren Sandersen: Just search for Foxpass, F-O-X-P-A-S-S, either our Twitter handle under that name or our website as well.

[0:26:22] Guy Podjarny: Cool. From there, it's probably an easy path to get to you.

[0:26:25] Aren Sandersen: That's right.

[0:26:26] Guy Podjarny: Cool. Well, thanks, everybody for listening in. Hope you found this useful and tune into the next one.

[OUTRO]

[0:26:33] ANNOUNCER: That's all we have time for today. If you'd like to come on as a guest on this show, or want us to cover a specific topic, find us on Twitter, @thesecuredev. To learn more about Heavybit, browse to heavybit.com. You can find this podcast and many other great ones, as well as over 100 videos about building developer tooling companies, given by top experts in the field.