Empowering your developers

Developer empowerment is about whether or not a developer has the space to make their own decisions based on various inputs, to solve their own security issues, and ultimately take the responsibility and ownership of the security of their applications. Empowerment requires support. It’s something that is given, grown, and supported by other teams. Let’s start with the top-down support that’s needed to empower developers to spend the time needed to deliver secure code.

"If you read the Netflix culture memo on their website, one of the things that tends to pop out at people is this discussion of freedom and responsibility. And that is a concept that has driven how their engineering organization works. It is interesting because, as a human, we tend to index on the freedom side. We say, “Oh, this is great. I can do whatever I want. And I’m sure I’m going to have to be responsible for something. But we’ll figure that out.” In reality, there is a huge responsibility when you’re building out software for a company like Netflix. You don’t want that service to go down, so reliability matters deeply. You don’t want it to be hacked, so security matters deeply. So somehow, you need to be very responsible as an engineer as well. And as a security organization, we often saw ourselves as there to help support and enable the business. What we wanted to do was allow the software engineers at the company to focus on the areas that they were hired for, effectively. But if they got to a place where security was really, really critical to what they were doing, then they would be leaning on us for help. And exactly where that boundary is becomes a judgment call, that if you have a good relationship between the security team and the engineers, you can figure that out."

Bryan Payne, CISO at BetterUp, former Engineering Director, Product & Application Security at Netflix

What drives developers to spend time on security?

Whether developers want to spend time on security or not, they need to be given the time to do so, and in a way that allows it to be prioritized alongside other functional or non-functional deliverables. Developers should be able to decide how much time they need to invest in secure development practices, versus reliability or scaling considerations, based on the needs of the business. When they make these decisions, they should ultimately be able to state what they’ve done and achieved at their end of year review and be supported by their manager. It’s up to managers to then reward them for their decisions, so developers know they’ve spent their time the right way. If this is not the case, the business is not empowering them to spend time on secure development.

Who should prioritize security for the team and business?

Ultimately, this comes from the very top: the CEO. If security is a business concern, it should be a CEO concern. This needs to trickle down the organization to the CIO and CTO, to the SVP/VPs of engineering, the directors, the managers and team leads, and ultimately to the developer actually running the tests and fixing issues. Once there is executive sponsorship, teams can dedicate a percentage of their sprints to engineering health, which includes security, rather than always being forced to focus on new features.

When this support and prioritization doesn’t come from the top, it’s extremely difficult to argue that security testing is as important (or possibly more important) than the next feature or a customer need. If these types of actions or activities are seen as extras that require justification, the default action becomes “do nothing”, and discussions start with “why” rather than “how”.

That’s not to say that the CEO and developers are doing it for the same immediate reasons. A developer takes pride in their code. They want it to be accurate, fast, reliable, and secure. And the CEO cares about the brand and bottom line, and they worry about how damaging a security incident or data breach could be to their business. While both are good reasons to be secure, the fact remains that a top-down approach to security adoption will have more impact than just relying on developers to add more to their workload.

"In terms of executive buy-in, [ours] came from the CISO, who had the CTO’s ear. Consequently, it was really from the top-down in the technology org. Right from the top, there was the understanding and the need for security. In terms of implementation, you’re dependent upon the software engineering organization and the leaders there."

Nicholas Vinson, DevSecOps Lead at Pearson

Does your security team support your developers?

While it’s the responsibility of the development team to secure their applications and code, they require support from the security team to do this well. When working collaboratively, the role of the security team moves away from an auditor position, where they run the tests and provide the results (and are seen as a blocker), to more of a security engineering team that helps the developers automate this into their processes, providing expert advice and consultation around the highest risk areas and prioritization.

The security team can’t force developers to prioritize security work over their other tasks. As mentioned earlier, that’s part of the overall engineering team and business priorities to decide. Historically, security teams have been looked at by developers as blockers, giving them security audits late in the development lifecycle. By empowering developers to become their own auditors, testing their code themselves, the security team can take a step back. They can then help development teams work through any results that require security expertise, shifting themselves from blocker to enabler. Additionally, this approach gives security the room they need to identify security champions with development teams, creating even more opportunities for collaboration.

Security can also educate the dev teams on vuln types and risk of exploits. This can be done in formal security champions programs, where other activities such as process roll out can also be well coordinated. While development teams work more independently, there still needs to be an overall governance structure and visibility for the business to understand its risks and exposures. The security team can provide the teams with guardrails backed with policies for their security testing, which can be applied to their pipelines and practices to help the development team with their early identification of issues and staying within their SLAs.

"When I think about security champions and successful models, you actually identify several folks in engineering teams that are responsible for security. You build a scorecard that they are responsible for, that actually shows the things that we’re doing to ensure that we’re securing our product or feature in the right way. There’s a path for engineers to talk with the central security organization as needed, and we’re also providing them with the latest and greatest in training. If the security team is responsible for building tooling and things like that, those security champions are helping you go and drive adoption. So security champions have to be within the engineering organizations. They can’t be sitting outside of it. They’re really driving security."

Rinki Sethi, VP and CISO at Bill.com

Another really important part of developer adoption is documentation. It’s often written by developers, providing their side of the experience with usage advice, explanations of decisions, and nuances. The security team can help coordinate and share this documentation with other teams that are also onboarding similar practices, processes or tools, and also have a hand in their creation. Giving the development team a good self-serve experience, that they’re familiar with all good developer tools, is a key aspect of successful adoption.

How are the security rules/process decisions in development workflows made?

One measure of empowerment is how much input and direction a developer or team has on their own processes and pipeline testing. When security rules and policies are made, it’s understandable that they’re driven with input from the security team, and then rolled out to development teams that take their advice on how to implement them into existing processes and roll out effective education. The important piece here is the way it is rolled out and implemented. The development teams need to be able to take ownership and make decisions and changes to their workflows the way they want. That’s not to say every development team should do it in a custom way, as it’s imperative teams learn best practices from each other. Development teams need the space to make decisions based on input from other development teams and the security group, such that the solution they put in place addresses the needs and standards that the security team requires.

One of the advantages of this approach is the dynamic it delivers. First of all, by not mandating a practice or process on the development team, you as a security individual can work with the development team to solve a security problem or requirement, rather than face the natural push back when something is imposed by an outsider. Having the security team take an advisory position offers a supporting function that is often more welcomed by development teams to help them make the right decisions.

Ultimately, this boils down to a question of ownership. If you try to have your security team own the security of the code being written by the development teams, you’re asking them to scale their services very broadly across organizations and likely become a bottleneck. Traditionally, security tends to assume the role of a group that imposes requirements onto a development organization — often seen by developers as doing so without a mandate.

If you decide you do need to apply restrictions on technologies or a stack that you feel the development team should be using, it’s important to create a paved road option that provides development teams with a choice of routes that are well supported by security and other teams. For example, it’s a common practice to have a set of golden container images that have been fully tested by security and other teams, that developers can pick from and build upon. If you give options that also act as a paved road, you won’t be seen as a blocker as you help teams stay secure.

Visibility and transparency across teams

Gaining visibility into the security posture of your applications, pipelines, and processes is the first step to identifying risk and exposures of your teams and business. However, not acting on the findings makes gaining visibility a pointless exercise. During this study, we found that using visibility to regularly summarize security posture in the form of an accountability scorecard or report was one of the defining factors of what made developer adoption successful.

The scorecards among those who achieved the best adoption covered security as well as many other aspects, including feature work, reliability, performance, and more. Their scorecards were generated monthly, showing operational metrics such as vulnerability counts, security tool adoption, project testing metrics, education levels, and much more. Very importantly, the specific metrics that each company included on their scorecards were aligned with business goals. Let’s go into how these scorecards were used in more depth.

Prioritization and justification

With a scorecard that covers business metrics — including security — it’s easy to see whether security is the most important thing to work on right now, or whether there are bigger problems elsewhere to address first. Scoring more than just security is important, as informed decisions can only be made when you have all the data at your disposal.

Scorecards should be generated at various levels, with the data tailored for the person consuming it. For example, an executive will want to see less detail than a team leader, as well as a closer connection to the overall business goals. The executive team can adjust their priorities based on the results of the report of where the business needs to apply greater focus. However, a team would want to know where they should spend more of their time, as well as how their scores compare to the rest of the organization.

All of this data allows developers and teams to better prioritize what they need to work on in upcoming sprints, as well as the justification for doing so. If anyone asks why a particular focus on security is happening, it’s important to provide an objective scorecard rather than a subjective explanation.

As part of these scorecards, the security team should also provide specific prioritization recommendations for the development team. For example, they could highlight a list of the top vulnerabilities (or vulnerability types) to focus on to generate the greatest impact.

Accountability

For security to be a priority, there needs to be accountability from top to bottom. The CTO is accountable to the CEO, VPs of Engineering are accountable to CTOs, managers are accountable to VPs, and so forth all the way down to engineers. When everyone has a reason to care about security, it will be adopted as a standard part of any role.

Fortunately, scorecards are an easy and effective way to hold the right people accountable to the standards expected of them. By publishing scorecards, everyone within an organization can quickly know where they are in the red or are trending in the right direction. And when defined thresholds are met, those that are accountable for specific metrics can determine the causes, justifications, and possible solutions.

Just as we found with security prioritization, security accountability truly needs to go all the way to the top. If it doesn’t, the message that it’s important to the business (and therefore should be important to the development team) is heavily diluted and buy-in will be lost.

Lean on developer values and gamification

Typically, developers care about delivering an application that isn’t just working, but is also fast, reliable, and secure. There are many things that can block a developer from delivering code that’s up to their desired standard, but ultimately, they have pride in their deliverables. Scorecards are a great way to gamify this desire to deliver the best code possible, changing accountability from stick to carrot.

Making sure your development teams have visibility into the status of their projects is a good way to show them where they’re going well or improving, as well as where they’re struggling or falling behind. The pride a developer or development team has can take a hit if they see their scorecard is the worst in the department or significantly below the average for the business unit or overall company. They will naturally want to rectify this, but they need to be aware of it.

Gamification works well when it’s done well. How you decide to do this will largely depend on your company culture, but sharing scorecards across teams and providing visibility prompts a gamification response. Nobody wants to be the worst team, and people enjoy seeing their scorecard progress and be above average for the local area — or even one of the best in the department. Equally important is the sharing of ideas and successes through announcements and discussions. These lead to new initiatives in teams wanting to replicate or push the ideas further in their areas.