Cultivating a DevSecOps Culture: Real-world implementations

Throughout the continued journey of implementing and maturing a DevSecOps model, sharing successes and lessons learned can help everyone improve. The following are examples from organizations who have adopted DevSecOps and have worked to achieve higher levels of maturity.

DevSecOps in Auth0

• Creating a “frictionless” experience for cloud developers

• Leveraging data analytics

• Risk prioritization early in the SDL

• Cooperative enablement, flexible development practices

Since its earliest days, Auth0 has heavily leveraged AWS cloud infrastructure to deliver identity management solutions to their customers. Through their rapid growth both in organizational size and service offerings, Auth0 has had to meet the challenge of ensuring their cloud infrastructure is secure. Duncan Godfrey, Senior Director of Security and Compliance recently appeared on The Secure Developer podcast and discussed their strategy for ensuring the security of that environment.

A dedicated cloud security team within Auth0 is responsible for ensuring the security of the AWS environment. However, to ensure that the team did not fall into a traditional SecOps mindset, automation with a focus on monitoring has been keenly important for that team. According to Godfrey, “The charter of that team is to ensure that we're collecting data from every possible facet and nook and cranny of AWS and pulling that in for analysis.” The result of this focus is that the team has touch points with the development teams that are functioning in a DevOps model. They’re able to achieve greater overall visibility by establishing that collaboration with the DevOps teams. 

For Auth0 and their cloud security team, creating a “frictionless” experience for the developers is crucially important. Security integration begins early in the SDL with a short form that helps guide how much collaboration will be needed with the security team. High risk functionality, such as exposing public endpoints, are supported with a greater degree of security involvement. “Working them through so hopefully upfront we have the requirements in place and the good controls in place that we end up with good secure software, and then at the end we're running some more tests.” according to Godfrey.

The overall culture at Auth0 is clearly one of cooperative enablement. Ensuring that developers are free to create software in a method that fits their needs but also demonstrating maturity in security practices. This has helped Auth0 continue to leverage new technologies while maintaining a confident security posture.

DevSecOps in Segment

• Walk a mile in the developer’s shoes

• Embracing infrastructure as code to enable DevSecOps

• Ensuring new tools are usable by developers

• Cross-functional embedded resources

On a recent episode of The Secure Developer podcast, Leif Dreizler and Eric Ellett talked about the importance that customer data platform provider Segment places on the collaboration between development, security and operations resources. Segment doesn’t do sprints across their organization, instead teams operate independently. However, through a consultative model, the security team is still integrated early in the development process to provide threat model and design review capabilities.

At Segment, the idea of establishing empathy is baked into the culture of the security team. Within the team, Segment has employed a concept of “Walk a mile in the developer’s shoes”. Ellett explained that the security team goes to great effort to understand how their security processes impact other areas of the organization. For instance, when they sought to roll out Multi-Factor Authentication, Dreizler spent a quarter embedded with the development team. This provides the security team with invaluable context on the challenges that the development team faces in terms of what they’re trying to protect.

However, the collaboration focus doesn’t end there. Ellett explained that there is the intention that similar initiatives would happen in the other direction. The plan is to bring people from other areas of the organization to sit with the security team and understand their world as well. Dreizler stated, “I think that this is what the goal of DevSecOps should be. Similar to DevOps, where you have operations people learning how to code and now everything at Segment's infrastructure is code.”

Segment also firmly believes in creating the “paved road”. A guiding principle that the Segment security team operates under is “Would this tool be used by the developer”. In other words, there’s a keen focus on ensuring that adoption of security controls is enabled by ease of use. The ultimate focus, according to Dreizler, is "Just make it as easy as possible for people to do what the right thing is."

Through this cooperative and empathetic approach, Segment has been able to grow a strong culture of collaboration in their organization — an example of how the promise of DevSecOps can be realized by ensuring all functions are aligned in their goal to do what is right for the organization.

10X Banking Embraces DevSecOps

• Transparency of security vulnerabilities across the organization

• Cross-functional daily standup meetings bring everyone together

• Leveraging automation to manage secure code deployment

• Using threat modeling card game to engage stakeholders in the process

Neil Drennan, 10x Future Technologies, joined Guy Podjarny on a recent episode of The Secure Developer podcast. Mr Drennan shared how the organization leverages some key communications strategies to improve engagement in security practice while building the first cloud-native banking platform. From ensuring transparency and effective communication, to driving collaboration in daily activities and proactive security measures, 10x places focus on making security part of everyone’s job responsibilities.

During their discussion, Guy and Neil started to explore how 10x makes collaboration across external security teams and internal platform teams work. Neil discussed how at 10x, everybody is able to view the current state of vulnerabilities that are being tracked by tools they’ve introduced to the environment. “When we're using tool sets, when we're alerting off any vulnerabilities that are found, then that's transparent and open to everybody. Anyone in the team can go and see what the current state of vulnerability is across the entire org.” according to Drennan. This form of transparency is a crucial measure for ensuring that all teams understand their role in the shared responsibility of delivering secure software.

However, the collaboration goes beyond just visibility into current vulnerabilities. Through daily stand-up meetings that include resources from across various teams within the company, 10x ensures that diverse perspectives are considered when discussing current security vulnerabilities. Neil shared, “We review [current vulnerabilities] in our daily standup across all teams, so it's not just a functional delivery piece. It's functional, non-functional and security, that everybody all comes together every morning.” Drennan identified this strategy as particularly crucial to how their organization deals with the increasingly dynamic threat landscape.

Drennan expanded on the effectiveness of these interactions, “I think that really helps the teams become more effective and having that open communication channel across those teams is super, super important”. This again builds on the concept of a shared responsibility for secure software, ensuring that it is inherent to the culture of the organization. This is a great way that 10x is able to build greater empathy and understanding between roles in the company.

Neil also described how 10x ensures early acknowledgement of security in the delivery pipeline through threat modeling. In their 2019 State of DevOps Report, Puppet identified that collaborative threat modeling efforts have a very significant impact on the overall security posture of an organization. To ensure that stakeholders are well engaged in this crucial activity, 10x leverages threat modeling card games to identify threats according to the STRIDE model. “Getting teams sitting down with their pack of cards and then playing out different security vulnerability scenarios is a really engaging way to get the teams excited about security and thinking about the right things on a continuous basis”, according to Drennan.

By focusing on the foundations of people, processes and technologies, 10x has been able to implement unique practices that have built a culture of security around software delivery. Their approach demonstrates how all three elements come together in a cooperative fashion to ensure 10x delivers software that is efficient, reliable and secure.

Addressing DevSecOps Culture at Datadog

• Breaking down gates, moving toward integrating security in the delivery pipeline phases

• Embedding security personnel in development to build empathy to shared responsibility

• Security as functional contributor to automation of the pipeline

DataDog serves as an example of how the quest for automation and integration of security into the pipeline is enabled by addressing the people issues first. They established a program to build empathy between organizational silos and drive better collaboration. They have also engaged the security team as an active contributor to automation and tooling in the pipeline. At the time of his appearance on The Secure Developer Podcast, Douglas DePerry was the Director of Product Security for Datadog and he shared with us some of their innovative approaches to solving these key DevSecOps challenges.

Trying to move towards a faster and better way of developing in software, often doesn’t work. When you’re pushing production dozens of times a day, there’s no way you’re going to be able to review everything.  Datadog embedded security engineers with development teams, sometimes for a few weeks, sometimes for months at a time. For them, raising awareness is a priority. Also setting the tone that security is everyone’s responsibility: “let me help fix some of these little hanging fruit or these security bugs.” and help kind of teach and learn along the way.

According to DePerry, “Security needs to write more code. More automation is needed. That is how you are going to get this force multiplier effect and be able to keep up with the speed at which code is being deployed.” One way Datadog tackled this was to develop a tool that was tailored to their needs and helped DataDog’s security team provide the development teams with actionable messages and remediation tasks based on the code that they wrote.

Datadog had previously invested in a Static Analysis Security Tool (SAST). It was initially brought in to demonstrate compliance but unfortunately it did not fit their needs and didn’t integrate well into their existing DevOps pipeline. So they innovated and wrote their own tool to handle SAST and Software Composition Analysis (SCA) scanning. DePerry shared,  “We unimaginatively named the tool Middleware and it was essentially like a framework that we could create additional plugins for. So we had our static analysis plugin and then we had our dependency vulnerability plugin.”

DePerry also told us that “you need to focus on the thing that will get you first; but how do you really decide that? And I think the more that you can, because it is difficult to measure certain things right now, it just makes that very difficult or makes that error prone...so I think forecasting, if done correctly, can really help with that.”

Cloud Native Culture at Pivotal

• The importance of making security a cultural change in cloud native transformations

• Create guardrails instead of gates to steer developers to the secure route

• “Pairing” developers and security practitioners to drive empathy and shared responsibility

Adoption of DevSecOps and transformation to cloud native technologies often go hand in hand. To ensure the organization is able to adapt to these new paradigms without compromising security, it must undergo a cultural transformation as well. Security, for it’s part, must move away from the idea of implementing controls as gates in the delivery pipeline. Much of this can be accomplished by building empathy and shared responsibility between developers and security practitioners.

Steve White, field CISO for Pivotal (now a subsidiary of VMWare) recently was featured on the Secure Developer Podcast. In his episode he shared some of his thoughts regarding cloud transformation, security’s role in the DevSecOps pipeline, and how bringing security and development together can lead to greater success. Steve spends a great deal of time working with security leaders and executives on engineering security architecture. He conducts hands-on workshops with representatives from across the various DevSecOps disciplines, educating them on the realities of cloud-native security.

A crucial element for transforming an organization through adoption of DevSecOps is ensuring that we move beyond just a focus on tooling. ”The first tenet of change in this space is that it's not just a technology change, although there is some technology shift that needs to happen. It’s a culture and a perspective change that ultimately is the larger piece of what’s to happen in information security, like it has happened in the rest of the business”, White shared. This acknowledges that DevOps has driven many changes in culture that security needs to catch up on as well.

One of the key perspectives that security practitioners have struggled with adapting is just how we incorporate security practices into the pipeline. White stated “The phrase I like to use is we’re moving from gates to guardrails, right? So security’s function in the enterprise moving forward should be to provide these – They’re like the safety net, right? There's a top and bottom guard rail that would protect you from sort of exceeding really bad parameters but within those guardrails.” This viewpoint highlights a key focus for how security can embrace the DevOps movement in a credible and more importantly a compatible way.

Another way to build that credibility for security among the developers involved in software delivery is the idea of building empathy. Having security practitioners and developers working together in their daily functions can establish credibility and an appreciation for the jobs that each other are doing. White elaborated on this point, “When I speak of pairing, I would love to see it be in the true sense of paired programming where two heads sitting in front of one screen, working on solving one problem together. If one of those folks is a security engineer and one of them is a feature developer, they’re both learning a lot and adding a good chunk to the conversation.”

Overall, White touches on a few of the key tenants that must be a part of any DevSecOps culture. With cloud native often driving a DevSecOps adoption in some cases and DevSecOps adoption driving cloud transformation in others, it’s easy to see how the two go hand in hand. Organizations should be aware of the link between the two and be prepared to adopt both as part of a great shift in organizational culture to derive business value from both.

DevSecOps at Cisco/Duo

• “Meet them where they live” to bring security to the Pipeline

• Pushing left to define security designs when identifying new technology opportunities

• Metrics need to be more sophisticated and focused on continuous improvement

The DevSecOps culture is predicated on the idea of efficiency across the pipeline. Enabling developers to go from backlog to deployment as efficiently as possible. For security to be truly a part of the pipeline, secure practices need to be integrated frictionlessly into the pipeline. To help enable this, pushing left to identify security designs as early as possible helps pave a smoother road for developers as they begin coding desired functionality. However, if you can’t effectively measure how practices are affecting your security posture, it becomes exceptionally difficult to justify keeping them integrated into the pipeline.

Cisco CISO and former head of Duo Security, Michael Hanley, appeared on The Secure Developer podcast and shared some of the approaches that Cisco/Duo have taken to address these issues specifically. Michael came to Cisco as part of the Duo acquisition in 2018 where he spent five years directing their security initiatives. His perspectives on integrating security into existing developer tools, pushing left to be proactive with security design, and compiling meaningful metrics are drawn from that considerable work.

When building out a DevSecOps culture, many organizations think about the tooling that is needed to make the pipeline move faster. Early on in the DevOps movement, this focused on automating everything. Code commits kicking off automated builds, code promotion initiating automated regression tests, etc. However, security has traditionally struggled to bring processes and tooling that fit this model and as a result end up creating more friction.

It’s this friction that frustrates developers and can hinder adoption of secure practices. Hanley shared that at Duo, their focus was on integrating with existing tools. He told us, “...we go to where it’s easiest for our engineers to work in terms of how we engage with them so for example, using the same ticketing systems so that we use for source control and the rest of our engineering cast tracking, we tend to do a great deal of work with our engineering teams in those same forums to make it easier for them.” This idea of meeting them where they live, integrating into the tools they already know, is a powerful approach to building a true DevSecOps culture.

However, Hanley also commented on another important concept to bringing security to the pipeline, and that’s the idea of pushing left. While the concept isn’t new, the degree to which security teams in a DevSecOps culture need to push left is increasing. “...if I were to map against the software development life cycle that we have at Duo, I would say to the far left time horizon we’ve got where the labs team is engaging and part of that is strategically identify new technology opportunities but also it is to define early what good security design parameters would look like…”, Hanley said.

This is an important and very innovative approach. By beginning to identify security designs before a user story even enters the backlog, organizations can ensure that developers have documented security requirements from the very earliest phase of the pipeline. This ensures that security design work does not impact the ability to deliver code quickly. It sets security up as an enabler because developers can more quickly move from those security designs to coding. They also don’t have to worry about missing security elements that become flaws to be addressed in future sprints.

How does an organization know these approaches are having the desired effect? This is where metrics are crucial. “I generally have a real problem with security metrics and volumetric sort of bug counting...those are not really sophisticated enough to describe many programs and particularly not ours.”, Hanley said. This is a real struggle for many organizations. A simple count of the number of open bugs doesn’t have context. How many releases were done that created those vulnerabilities. If the vulnerability count is low, is that because security got better or the application hasn’t been tested in a long time. Metrics programs need to have more context.

Hanley shared how Cisco/Duo has gone about using a maturity model for measuring the success of their program. “the maturity model that we use at Duo is really a mash up of the BSIMM and SAMM...we try to think about what are key metrics that come out of that or what is the percentage of activities...where we have at least partial coverage and what is the percentage where we have full coverage,” Hanley described.

He also went on to mention another concept in terms of metrics that is crucial for any organization to consider. Hanley shared how their approach helped focus them on a model of continuous improvement. Many security initiatives fail because they look at trying to achieve astronomical goals that just are not realistic. By instead focusing our goals on improvement rather than attainment, we make room for mistakes, adaptation, and innovation. Ultimately, the business value of security practices can be demonstrated more readily.

The concepts that Hanley shared from his time at Duo speak to the needs any organization would encounter as they seek to build out a DevSecOps culture.

Cloud Security with 2nd Sight Lab

• Enabling developers by teaching best practices for cloud security

• Building empathy between developers and security practitioners

• Governance as a component to cloud security

With cloud transformation taking center stage in most organizations’ IT strategies, the challenges of securing those environments can be amplified. Enabling developers to leverage cloud native technologies while remaining secure can be a challenge. Additionally, ensuring the various disciplines within our organization work together and understand the bigger picture is crucial for securing complex cloud environments. Finally, how we monitor our program and learn from mistakes is a crucial element anytime we go through a transformation of our business in this fashion.

Guy Podjarny met with Teri Radichel, CEO of 2nd Sight Lab and author of Cybersecurity for Executives in the Age of Cloud. Teri explained how she got into the world of cloud security after experiencing a breach at her prior web application development and hosting company. 2nd Sight Lab is a cloud security training and consulting firm. So much of what Teri shared was not from the perspective of implementing cloud technology within a single organization, but rather her consultative work with multiple organizations.

A fundamental issue that surfaces when we discuss cloud and cloud transformation is just simply defining what cloud means. In Teri’s words, “Some people say it’s just someone else’s computer, but I always say it’s more than that because when I was at a managed hosting facility, that was someone else’s metal computer, right?” For some organizations, even the use of Software-as-a-Service (SaaS) solutions can be part of their cloud strategy.

With the complexity of just understanding what the cloud is, adding in the dynamic nature of cloud native technologies means we entrust developers with a lot of responsibility for security. Teri touched on this complexity, “If you are a developer who is now being asked to build networking or to configure S3 buckets or load balancers or CDNs, you really need to get in there and look at what are the best practices.” To address this, she had some advice for developers, “There’s something called the CIS Benchmarks out there that will tell you best practices for all three top cloud providers.”

Of course another key to addressing the complexity of cloud security is getting developers and security experts working together to share knowledge, understand each other’s jobs, and work toward a common objective of secure software. Radichel mentioned that security needs to take an active role in this regard. “I think that there is a huge function of security which is not involved hands on keyboard. It’s not about application security and making sure you don't have a cross-site scripting flaw in your code. Security is about risk,” she shared.

Finally, in any large transformation, ensuring that defined processes and procedures are followed consistently and that we learn from mistakes is also terribly important. Radichel acknowledges that, “But also people make mistakes, so you have to think about your overall governance, so then your organization as well and how you’re going to set it up, so people can’t make those mistakes, right?”

Ultimately, leveraging the shared responsibility model of a true DevSecOps culture drives better practices across the cloud transformation. The observations and key suggestions Radichel offers bear witness to the importance of all teams involved in the delivery pipeline working together, sharing knowledge, and focusing on a common goal.