Keeping Cloud Foundry Secure With Molly Crowther

Molly Crowther: “Cloud Foundry is an application developer-focused platform. Developers write applications and it helps them easily get their applications up on the Internet. You can just run one command and it takes care of getting your code to the right places. It is open source like all of the intellectual properties managed by the Cloud Foundry Foundation. We've had to get the engineers to understand a persona of the malicious operator.”

Guy Podjarny: “You love your users too much, to believe that they would do something so evil.”

[INTRODUCTION]

[0:00:33] Guy Podjarny: Hi, I'm Guy Podjarny, CEO and Co-Founder of Snyk. You're listening to The Secure Developer, a podcast about security for developers covering security tools and practices you can and should adopt into your development workflow. The Secure Developer is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market. For more information, visit heavybit.com. If you're interested in being a guest on this show, or if you would like to suggest a topic for us to discuss, find us on Twitter, @thesecuredev.

[EPISODE]

[0:01:04] Guy Podjarny: Hello, everyone. Welcome back to The Secure Developer. Today, we have with us Molly Crowther from Pivotal and Cloud Foundry. Welcome to the show, Molly.

[0:01:13] Molly Crowther: Thanks. You actually pronounced that right. Really impressed.

[0:01:17] Guy Podjarny: I get a lot of practice with the Guy Podjarny. Not the most catchy tune that you'd heard. Well, thanks for joining us today. I think we have some really interesting conversations to talk about today about Cloud Foundry security and open source as a whole. Before I give away too much, do you mind introducing yourself a little bit?

[0:01:32] Molly Crowther: Yes. So, I work at Pivotal. I've been there for about a year. I'm actually a Technical Program Manager, which is also called a TPM. What we do is we bridge the gaps between all of the small, highly focused, microservice-focused engineering teams at Pivotal, help them talk to each other find, breaking dependencies between them. And we also tend to focus on horizontal aspects of the whole Cloud Foundry platform, including security or docs, or just other things that every single team needs to get right, in order to create a coherent platform.

[0:02:13] Guy Podjarny: Okay, interesting. So, the security ownership itself still lies with the different teams and you're more the overseer, or is it the other way around?

[0:02:21] Molly Crowther: It's very distributed. So, we have a few kind of product teams that work on security. We have a security enablement team that builds tooling that helps the teams get better at security. I also kind of co-PM a security triage team that ingests vulnerabilities from customers in the outside world, and tries to understand how bad they are and the priority of getting them fixed.

[0:02:48] Guy Podjarny: Okay. Sounds like a pretty broad role. So, I'm tempted to ask a bunch of questions there. But maybe before we dig too much into that, you work on Cloud Foundry for Pivotal. Can you give the audience who doesn't know, just a couple of moments on what that is?

[0:03:02] Molly Crowther: Yes. So, Cloud Foundry is an application developer-focused platform. Usually, when I'm explaining to people who know nothing about the cloud, I just say that developers write applications and it helps them easily get their applications up on the Internet. I think the best thing about it is, once you have your application, you can just run one command, and it takes care of all of the routes and getting your code to the right places and putting it in containers, and making sure that containers are healthy, and all sorts of things like that.

[0:03:34] Guy Podjarny: Yes. Kind of the wonderous CF push command to just move it. It's a platform as a service, right?

[0:03:41] Molly Crowther: It kind of depends. So, depending on the company running it, because it's an open source product, it can be hosted, it can be on-prem at customers. So, it is distributed in all sorts of ways.

[0:03:54] Guy Podjarny: Interesting. Yes, it's definitely a very broad platform. I must say that I was aware of Cloud Foundry and Pivotal for quite a while, but really only dug into this ecosystem in the last year and had been quite overwhelmed by both how powerful it is and how widely adopted it is. For instance, the fact it's the backbone for IBM Bluemix and the SAP Cloud, and just the wide use that it has in enterprises that are looking to modernise their processes and control it. So, definitely an interesting environment. Sorry to, maybe, belittle it with the platform as a service, but I do see that sort of concept. I sort of say Cloud Foundry amongst other things, as the manifestation of platform as a service as well.

[0:04:41] Molly Crowther: Yes, what's really great about it, working on it, in general, is just, it is open source. It's managed, like all of the intellectual property is managed by the Cloud Foundry Foundation. So, while Pivotal is like a really big contributor to it, we're among all of these other huge companies in this ecosystem and then we're working together to improve it.

[0:05:02] Guy Podjarny: Cool. What was the history of that? I mean, how did that come to be?

[0:05:07] Molly Crowther: Yes. It's kind of complicated. It started I think Cloud Foundry started as a project at VMware, back in the early 2010s, and then at some point, EMC bought what was Pivotal Labs at the time, which has been around since the eighties and is the software consulting arm of Pivotal. Then, at some point, they took Pivotal Labs, spun it out with Cloud Foundry and a bunch of data products as Pivotal Software, and that was in 2013. Then, I think the first Pivotal Cloud Foundry distribution came out at the end of 2013. Then, at some point, all the IP was turned over to the foundation so that we could build a larger ecosystem with a lot of other players and it's not just Pivotal.

[0:05:56] Guy Podjarny: Okay, definitely an elaborate history, but also, I think, an interesting model today for sort of running this between the foundation and Pivotal, but also, IBM and ATF and a bunch of others that are participating.

Okay, we'll get back to that little bit. Thanks for the background. Let's dig into security. So, you run, at the end of the day, or at least kind of help manage a lot of the security practices of the Cloud Foundry platform out of Pivotal. What are the highlights there? I mean, what are the kind of key tenants in managing Cloud Foundry’s security?

[0:06:33] Molly Crowther: It's interesting, because as of about a year ago, a year and a half ago, a lot of Cloud Foundry’s security was somewhat ad hoc, and very, I would say, crisis-oriented. So, people would find things, and everyone on as many teams as possible would kind of rally around whatever the issue was, and fix it as soon as possible, and do whatever disclosures like public disclosures were necessary. But that was a painful process. I think it was also kind of a scary process.

So, around the time when I was hired, we started focusing more on trying to refine the security process a little bit more. We spun up a triage team that takes care of incoming reports and understanding like the CVSS scores of those and being able to handle communication back to a reporter. Because we, obviously, when somebody reports something, it's important, and we want to fix it as soon as possible and we want to make sure that they're – like that person is happy and wants to keep reporting things to us.

[0:07:42] Guy Podjarny: How do you receive such reports when you’re in there? Because it's an open source project, but you don't really want somebody to just open a GitHub issue. Does that happen?

[0:07:50] Molly Crowther: That does happen sometimes. We try to balance, somewhat discouraging that with also understanding that like, we are open source and trying to be as transparent as possible that there are issues. We accept reports at various email addresses. Security@cloudfoundry.org goes to me and a couple other members of the foundation who work to triage issues. Then, we also get reports specifically from Pivotal customers to security@pivotal.io, which also kind of goes through the same process.

[0:08:22] Guy Podjarny: Okay, cool. So now, there's an email. If somebody reported a vulnerability in your system, that goes to you and a few others. The triage team kicks in and understand what's to do. What's next?

[0:08:34] Molly Crowther: What we usually do is we take a first pass of like, is this issue reproducible? Do we think that this is a real issue? Then, we've done a really good job at personally talking to most of the different product teams. So, we have connections with all those teams. When we find an issue, we'll jump into their Slack channel. Or if they're in the right city, we'll go tap them on the shoulder. We've learned a lot from that, just because I think when we first started doing it, and people weren't really sure what the triage team was doing. Anytime I would walk around the office, people would just like be afraid that I was coming to talk to them. It was kind of like, you're the principal of the school and nobody wants to get in trouble. I think people have learned that when we're triaging an issue, we're really just trying to understand the severity of it, and not like making them drop everything immediately to fix it, because that can be really disruptive and somewhat terrifying.

[0:09:33] Guy Podjarny: Yes. Not a very pleasant experience. Okay, understood. So, you're doing this and you're running around. This is, like this communication and these product teams span companies, right? Some of them are Pivotal, some, the foundation, some may be IBM, or SAP or others?

[0:09:46] Molly Crowther: Yes, definitely. Just depending on what comes in because there are various things that either aren't maintained by Pivotal teams or open-source teams that happen to work in the Pivotal office. We have you people from SAP, IBM, VMware, like Spring in general, which is a Pivotal product, but it's also open-source. So, we have tendrils into lots of different organisations.

[0:10:13] Guy Podjarny: Cool. Then, let's say, it was resolved or not, I guess. You take care of getting a CVE or process like that?

[0:10:23] Molly Crowther: Yes. I'm pretty proud of where we've gone with our CVE process. I think last year, in 2016, we were actually, cvedetails.com, or something. There's like a website that kind of aggregates a lot of CVE information. I think we were in their top 50 companies of like number of vulnerabilities reported and that was really cool, because that was mostly me, like manually writing out CVE notices and having them posted.

We've also kind of split up a little bit. All of the notices used to go like directly on the Pivotal website and it was really confusing for people who are like, this is an open source component. why is this on the Pivotal website and nowhere else? So, we split all those out onto cloudfoundry.org. There's good RSS feeds for them now, so you can subscribe to them. We're, in general, just really proud of how open we are with when there are vulnerabilities, like making sure that they're publicly disclosed, and fixed as quick as possible.

[0:11:22] Guy Podjarny: So, Cloud Foundry is maybe by its very nature, and it's a platform, and this is how I sort of see it. It's a platform that is a good mechanism for enterprises to modernise their practices. As a result, it's adopted by many security-conscious enterprises. If you look around and you see who's using Cloud Foundry, many of those are banks or insurance companies, or these telcos-large companies that are very familiar with security and have pretty stringent security requirements. How do you communicate with them? These like organisations, my experience is that organisations like that, oftentimes, for instance, like to get early notifications. They like to have somebody to blame. They like to have some clear ownership. I'm trying to avoid saying somebody to sue. How does that impact your work, if at all?

[0:12:10] Molly Crowther: It definitely impacts in terms of how we explain potential vulnerabilities to the development teams and the product teams. That was one thing that we had to learn pretty early on, that when a customer comes to us and says, “Oh, I think there's something here where an operator could exploit the system.” So, someone actually running the platform at the company, like one of our employees, like this is a way they would exploit the system. For a lot of teams, it's like, well, they're an administrator –

[0:12:42] Guy Podjarny: Yes. They’re an admin. They can do everything anyway.

[0:12:43] Molly Crowther: They can do everything anyway, so why do we have to prevent them from doing this one particular thing? We've had to get the engineers to understand like a persona of the malicious operator who either has socially engineered their way into a job or is disgruntled, and wants to set booby traps for later. That's like a big education thing that we've had to do, just to understand how to securely develop your code, while planning for people within this system trying to break into areas that they shouldn't be.

We've also had to learn a lot about interacting with large enterprise customers, when they come to you with a penetration test or results of security scanning. Sometimes, based on the results that we get, it's not that we have to turn them away, but we do sometimes have to compromise with them, and what ways we're going to address some of these issues, because a lot of security teams, at some enterprises don't necessarily know – they have the security experience rather than the specific Cloud Foundry experience. So, if they run scans on every single VM, or every single container within a Cloud Foundry installation, they're just going to come back with 100 pages of like, “I got to an IP that I shouldn't have been able to get to”, or something like that. So, there's a lot of work that we do with our customers to understand how to address issues in an agile way rather than a waterfall way.

[0:14:23] Guy Podjarny: Yes. I love the personal aspect of it. Definitely a difference in the maybe the sensitivity indeed, to which malicious actors to be aware, depending on the audience you're in. So, before we move off this, maybe dig into the Agile dev and such a couple other topics, I’d love to still explore on the vulnerability reporting side of it. Do you give any early notifications to – I mean, you're an open-source project. So, to an extent as soon as it's out there, as soon as a vulnerability is fixed, that fix is in the open. Is there still some mechanism to have the enterprise customers find out about the vulnerability ahead of time?

[0:15:03] Molly Crowther: There have been a lot of philosophical discussions, I could say, about this, within the foundation and a lot of the companies that belong to the foundation. Currently, as a closed-source distributor, we don't offer early access to information to customers.

[0:15:23] Guy Podjarny: Sorry, as a closed source?

[0:15:25] Molly Crowther: Yes. So, in this process, we have to wear a lot of different hats, like me, in particular. I'm a representative of the foundation in the process, but also a representative of Pivotal. The information that gets shared with customers who purchase closed-source distributions, is restricted until the vulnerabilities are public, if that makes sense.

[0:15:53] Guy Podjarny: Yes. Let me kind of echo this back and see if I got this correctly. Technically, you distributed closed source, Pivotal Cloud Foundry, so you could introduce a fix there before you made it publicly available. But that would make it somewhat unfair to those who use the open-source version for it. And in the dynamic between the closed source vendors and the open source foundation, especially given that you manage security issues for the open source foundation as well, that's just, the lack of neutrality here is too big of an issue to bear.

[0:16:23] Molly Crowther: Well, it's not that we can't release software that is patched, it's that we have to be very careful about what we can actually say about why we're releasing that software, because the member companies themselves do receive some early warning about things that are coming out, and we try to reduce the amount of time between when a fix is committed, and when that information is actually public. So, once the fix is committed, people are working quietly to patch everything that they can, and like all of the member companies are doing that. But we're not allowed to say anything about it until the vulnerability itself is made public.

[0:17:09] Guy Podjarny: So, the fixes themselves would make it into the open-source code, without much detail. And maybe after that somebody would come back and edit some commit messages.

[0:17:19] Molly Crowther: Exactly.

[0:17:21] Guy Podjarny: Afterwards, to clarify it. Okay. So, I find that relationship between Pivotal and the foundation to be a fascinating topic, especially when it comes to security. You work as you said, you have this, at least dual hat if not more. And beyond that, there's also other member companies that are, to an extent, competitors too. They're not really because I think every one of these platforms has its own unique differentiators. But other commercial Cloud Foundry-based solutions that are benefiting from Pivotal’s work and specifically from your work on the space. How do you see or how does it work, sort of the split in responsibility between security responsibility, between the open source, Cloud Foundry Foundation, and Pivotal?

[0:18:08] Molly Crowther: It’s interesting. Pivotal is a company, and most of the member companies do their best to not differentiate themselves based on security, because we think that is detrimental to the platform as a whole. So generally, everything that we do in security is either, if it's a separate service, it ends up being open-sourced. CredHub is a really good example of – it's a credential management solution that's tailored to Cloud Foundry that has been recently open source. We definitely work as closely as we can with the other companies, sometimes, in particular, time zones are like a huge issue for me, just because most of the other people that I work with at the foundation on security are tend to be in Europe. So, getting time zone overlaps is really hard.

As far as I've seen the relationship between the companies and their relationships to the foundation are all really productive. It seems like they would be competitive and I think probably the sales departments are very competitive. But the actual product development, like we all understand that we're working on this together and for the ecosystem to survive as a whole. That has to be our main goal, is to collaborate well with each other.

[0:19:26] Guy Podjarny: Yes. To me, it seems great. I love to see both, multiple companies collaborating on security, and specifically the relationship with the open source element in it. I think, it's also again, fascinating, because for the most part, when you look at open source business models out there, differentiation on security actually is the differentiation, oftentimes. Not just security, but rather on sort of enterprise-grade reliability, from assurances and support and all of that jazz. But also, security and security of the platforms and handling vulnerability management. So, I'm sure this is a frequently discussed topic in the context. But it's great to hear that the companies are collaborating well on it.

Maybe switching a little bit to a slightly different topic and I think it's interesting to look at how security is managed on the platform itself, which is what we've discussed so far. But I think another area of interest to our audience is to talk about the platform, maybe as a means for security, right? To talk about securing in an Agile development environment. Maybe if we can discuss that a little bit, Pivotal is maybe one of the most committed development organisations in terms of like modern methodologies, pair programming, all sort of extreme programming techniques around. It's interesting. We heard a little bit. You told us a little bit about security playing a role there in the security enablement and the likes and the teams running there. What aspects do you see, like what interesting development practices you think people can adopt from how Pivotal does security, and which interesting security practices they see running on the platform itself? If that makes sense.

[0:21:08] Molly Crowther: Yes. So, one of the things that is really cool about Pivotal that you mentioned is the focus on extreme programming and Agile development in general. What's interesting about Cloud Foundry is that we're kind of an embodiment of those philosophies and we're experimenting on actually taking that pivotal model and scaling it. So, it's not just one development team that's doing pair programming. It's hundreds of engineers, within like a 2,000-person company all over the world.

There are some interesting things that we've been doing with security, if you kind of consider each of the little product teams that are building the platforms. It's kind of their own application or service teams. One of the things that we've been doing a lot more, is running what we call security workshops, which are both to level up the experience of particular engineers with respect to understanding threat-modelling and just basic security concepts, while also trying to learn how to attack their specific part of the system. So, we're both improving the security of the system, and hopefully teaching these engineers more about security so that when they inevitably rotate to other teams, they can bring that expertise and help kind of scale the security education as a whole for the company.

[0:22:35] Guy Podjarny: Is this an educational workshop? This is like a mock application that you run through with some frontal, like some presentations in it? Or is this literally, pen-testing their applications and in running tools?

[0:22:48] Molly Crowther: It's taken a couple of different forms. We actually have some cognitive science, pivots, who are working with us on it to kind of design better –

[0:22:58] Guy Podjarny: Sorry. Pivots is like Pivotal employees?

[0:23:00] Molly Crowther: Yes. Pivotal employees. We have some cognitive scientists who are helping us with understanding how people kind of make a switch in mindset. So, one of the things that's really interesting is, the product teams are often really focused, obviously, on their own code and pushing features and things. But they also work from a mindset where they're empathetic to their users, and empathetic to each other, and assuming the best of everyone, which doesn't necessarily help you get better at security, right? Because you need to understand who the people are that are attacking you, and the fact that they don't have any respect for your stories, or your code, or the law.

So, those security workshops, we usually have a couple of different activities that are related to understanding threat-modelling, and understanding that mindset of an advanced persistent threat against your part of the system. But also, draw an architectural diagram of what your component looks like. Where does the value move through that system? And how can we protect it?

[0:24:07] Guy Podjarny: First of all, like cognitive scientists working with the dev team. Wow. I guess I never really thought of the complexity or of how building empathy and thinking about your users, and putting them first could actually get in the way a little bit of security mindset, because you trust her. You love your users too much to believe that they will do something so evil.

So, you do the workshops, and you teach those components. Have you managed to – it sounds like the dev teams themselves own many of the security aspects, or at least, should care about security. Have you seen this as a result? Do you see initiatives, security-related initiatives coming from the team? Do you see central tools that are actually initiated by the teams?

[0:24:52] Molly Crowther: Yes. So, one of the parts of the Agile process and Cloud Foundry is running what are called inceptions, which are, you basically take a large feature and understand the scope and the inception is a meeting where you meet with the whole engineering team and other stakeholders to understand the goals and anti-goals of that particular feature initiative. So, I've been to a number of these, where they were related to like specific security improvements for the open-source that weren't initiated necessarily by an engineering director telling them they had to do it, or somebody from a security team telling them that they had to do it.

But it's like, “Oh, here's something that we have to accomplish. And it turns out, if we do it this way, it'll be more secure.” So, teams are definitely starting to understand that security is debt, that they have to pay down, and that when they do new things. If they do it in a more secure way, they don't have to spend as much time paying down that debt.

[0:25:56] Guy Podjarny: Yes. Interesting. Well, I think Pivotal has always been kind of at the forefront, as we just discussed around development practices. So, we'd love to see the Pivotal dev process or well, in fact, maybe we are seeing it being an example of embedding security into that process as well, of sort of having that security ownership there. I think everybody believes that security should be an aspect of quality. It should be something that's built-in, and it's just really, really hard to do it. I also really like, even just that name. I mean, I've mentioned this a few times today, the name of security enablement, as a name of a team, because I think it very much demonstrates the fact that this team is there to enable security that is performed by the dev teams, because that's the only way to scale. That's the only way that it would get embedded in.

[0:26:44] Molly Crowther: Exactly.

[0:26:46] Guy Podjarny: I still want to sort of dig into – I think we have time for one more short topic, and then maybe a bit of a closing question. So, when you think about this was all about sort of Cloud Foundry and your processes. When you talk about your users a little bit, you say Cloud Foundry organisations, and you know they're using a pass or maybe they're running containers on it. Let's sort of focus a little bit on the, so the platform as a service angle of it. How do you see that impacting security? Does that make it more secure, or less secure to use a PaaS versus say, running containers or running your own systems?

[0:27:21] Molly Crowther: I guess, I think back to my experience previously, before working at Cloud Foundry, I was working at a small energy efficiency consulting firm and we had a particular person who is more excited about DevOps than most of the other developers. And he basically, like, figured out Docker like before it was – before Docker was a thing. He was trying to run Docker and trying to run our applications. I think of people basically building their own platform and not realizing it, versus, okay, we have a company with 500 people, dozens of teams working on this for you. There are always going to be things that you, as someone trying to roll your own platform don't think about. And that, I think, for our customers and Cloud Foundry customers in general, the actual line of where people are providing value in their organisation is at the application level. They're not winning awards for running Linux servers, securely. So, they should focus on finding a platform that they can live with in terms of its security, because the alternative is to do it yourself and probably mess up.

[0:28:32] Guy Podjarny: Yes. I think, I would say even more strongly and I'm going to think maybe it's the same concepts that you're saying right now, which is, the more you do, the more opportunity you have to mess up security in it. When you run with a pass, then you have the opportunity to put your security constraints and practices into the platform itself, and have a very small number of people do that. If at all, maybe somebody else is operating the cloud for you, and that reduces the opportunity for a developer to mess it up. Fundamentally, is the pros, hopefully, that are sort of running the platform itself.

I definitely believe that that type of notion extends to serverless and function as a service, which is at the end of the day, just a variant of platform as a service with some specific attributes to it. I know I'm a fan from a security perspective of centralizing security, when it doesn't get in the way of Agile.

[0:29:27] Molly Crowther: I mean, the point is to make it as secure by default as possible, and also build in kind of as many layers as possible to prevent your valuable data from getting taken.

[0:29:40] Guy Podjarny: Cool. This was fascinating. Before I let you off the hook. One question that I ask all my guests on the podcast before you disappear. If you had one tip to give a dev team looking to level up their security practice or maybe even just a security pet peeve that you're trying to get people to finally get right. What would that one thing be?

[0:30:03] Molly Crowther: I'm afraid people probably have said this before, but credential leaks are a huge deal. I think they're in the news practically every week. Somebody made an S3 bucket public, or somebody's credential was in GitHub and Bitcoin miners spun up a bunch of VMs and cost them a lot of money. So yes, they're very preventable. We've built a lot of tooling around securing credentials and trying to scan for when they get leaked. So, I think developers in general should just be really careful about their credentials.

[0:30:37] Guy Podjarny: About their credentials. Cool. I think it's a good one. I'm not sure if others have mentioned it on the show and I think it's this single point of failure that you have, right? You build all these controls about who can run what and who can do it, and then you end up leaking that person's credentials, and it's all free and open for the attackers. Cool. That's a good tip.

Well, Molly, thanks a lot for being on the show today.

[0:30:57] Molly Crowther: Yes. Thanks for having me.

[0:30:58] Guy Podjarny: And thanks to everybody who tuned in, and join us on the next one as well. Thank you.

[OUTRO]

[0:31:03] Guy Podjarny: That's all we have time for today. If you'd like to come on as a guest on this show, or want us to cover a specific topic, find us on Twitter, @thesecuredev. To learn more about Heavybit, browse to heavybit.com. You can find this podcast and many other great ones, as well as over a hundred videos about building developer tooling companies, given by top experts in the field.