Improving security adoption in your development teams

Best practices and tools for driving adoption

0 mins read

So far we’ve heard a lot of accounts from various different groups and covered scenarios that both help and hinder developer adoption of security activities. In the remainder of this paper, we’ll take a look at some actionable tips or mini-programs that you can consider applying in your organizations. Because these tips have come from a variety of companies at various stages of their digital and security transformations, they may not all be as relevant to you, so we’ll start with a number of playbooks based on some of the most important considerations that would be applicable or most well-received by your development and security teams. Afterwards, we’ll go into more detail about some of the more tactical tips and approaches you can adopt or introduce in your teams.

Actionable playbooks

In this section we’ll talk about trailblazing teams, adopter teams and inertia teams. Before we get started, let’s define what we mean by each of these terms:

Trailblazing teams tend to be your high-performing teams who are the first to adopt processes and technologies because they are beneficial to the team or delivery. Typically these teams are a minority. Trailblazers are less likely to be overwhelmed, and more likely to discover and define best practices and standards for other teams in the organization to follow.
Adopter teams tend to be solid delivery teams, adopting processes, techniques and technologies once they’re proven by the trailblazing teams in the organization. These teams typically cover the majority of the development organization.
Inertia teams resist change. They’ve done development a certain way for some time and are very comfortable with that approach and prefer to continue doing that because of familiarity and continuity.

In the table below, consider where your pain points are, whether cultural, process, or tooling based:

Type of rollout	Trailblazer	Adopter	Inertia
Security champions (Culture)	When initially rolling out a security champions program, you should predominantly involve trailblazer teams. These are the teams you should use to help define the program with you. Use the trailblazers as a way to recruit security champions in the wider org. They should also have a big part in the content of what should be taught and where pain points exist.	The larger rollout of a security champions program will be among adopter teams. Using trailblazer champions to evangelize the program and benefits they are seeing is core to bringing adopters onboard. Listen to the adopter teams to understand how their problems differ from trailblazers, and try to pair them to solve problems together.	Don’t expect people from the inertia teams to volunteer to join the group unless there are organizational pressures or rewards they consider valuable. Encourage adopter teams to champion security to these teams gently and show value and rewards they will be most interested in.
AppSec tool(Tooling)	Onboarding of any new tool should start with trailblazing teams. They can help you identify the correct paths of interaction between the security team and development teams with the tool. They can also assess and advise where tools can best integrate into workflows and help with documentation, processes, and paved road style implementations that adopters can then use in a bigger scale rollout. Trailblazing teams are more capable of taking on new work and are less susceptible to being overwhelmed, so build integration options as far left as possible, including IDE, Git repo integration testing.	Once you have seen success from your rollouts to trailblazer teams, consider rolling out to a few adopters that are representative of the wider organization. Have them test out documentation and new workflows that were identified and created by trailblazers. Make changes as necessary and perform a wider rollout, keeping in mind that team capacity of additional work may be a greater challenge. Focus on future development testing first, rather than working on your backlog right away, to prevent overwhelming teams with new work. Also, be mindful of where adopter teams prefer to use tooling in their process and workflows and ensure their preferences and historical usage of integrations are taken into account.	During rollout, you will likely need to start further right, potentially at CI build time, if that’s where the majority of their other tests are running. The earlier the testing, the more overwhelming the new results will be, which can increase frustration, pushback, or workarounds. Make sure documentation and paved path usage is very clear before trying to onboard the inertia teams, including support from the adopter champions and security teams as needed. Only focus on the most critical issues that tooling identifies initially and for new development features only, to reduce new work. Onboard much slower than you did with adopters and ramp up in the tool usage slowly, increasing guardrails over time.
Scorecards(Process)	Scorecards across teams should show similar data and exposures. Your expectations of what can be achieved by the various teams should differ. Use the data from the scorecards to show how activities from the teams translate to results, and then celebrate them.	Scorecards across teams should show similar data and exposures. Your expectations of what can be achieved by the various teams should differ. Use the data from the scorecards to show how activities from the teams translate to results, and then celebrate them.	Scorecards across teams should show similar data and exposures. Your expectations of what can be achieved by the various teams should differ. Use the data from the scorecards to show how activities from the teams translate to results, and then celebrate them.
Celebrating success(Culture)	Trailblazers are leaders. When sharing their success, show them off as leading the way with new processes that improve the business. Show other teams how their adoption of new tools or techniques made their application development more secure. Highlighting how they are driving the engineering team forward is some of the most important recognition trailblazers can get, particularly from engineering leaders. Celebrating individuals is a useful way of driving other high performers. Swag always helps!	When celebrating adopters, it’s less about new techniques, and more about how successful they’ve been with the adoption (and the fact that they have adopted). Focus more on implementation and adopter results. Make sure you celebrate team achievements, the integrations they’ve built up, and over time the number of issues they’ve fixed, etc. Use swag to get teams on board. Recognize teams at events or meetings in which many other teams are present, like monthly department or business unit meetings.	With inertia teams, lower the bar to what success looks like. Consider a no- or low-touch rollout as success and share with other inertia teams how easy it was. Be focused on the low-effort and high-value aspect of any new success. Work to build upon success in small steps to the next milestone. Learn what is valuable to your inertia teams as reward, as it can be different from adoption teams.
Pipeline automation(Process/Tooling)	In order to build automated security testing into existing workflows, you need the trailblazer teams to be a big voice in where the automation should be and how they want to consume results. Depending on your team structure, it might be down to the trailblazer team to build the automation themselves, or work with a DevOps team to do so. With high performing teams, it’s much more possible to push further left, and include security solutions into IDE, local build environments, and Git repositories. Make sure that enough is documented so that folks know what is running, the guardrails that are in place, and the processes for handling failures. Automation in the pipeline can also be used to push reporting and metrics, which can be led or helped by the trailblazing teams.	Adopter teams often take a subset of the automation points that trailblazers use based on how their existing stacks are built. For those who are using consistent stacks and pipelines, it’s not uncommon to duplicate the automation points. Try not to mandate that teams have to add automation everywhere or even in most places, but make sure the value of adding the automation in various places is clear and that it’s accepted by the development team. Initially, be sure that the automation has little or no impact on the developer workflow. Add it for visibility initially, and ramp up the policies, guardrails, and expectations of the development teams.	Depending on the inertia team you’re dealing with, it’s possible that you’re more likely to get success further right in the pipeline, closer to the CI build process. Again, be sure to offer automations that can pull in security automation further left, but you need to match the team style with this. You want to be very careful about how you impact the developer workflow. So start off in a visibility-only mode, only moving (and doing so slowly) once the team gets more education around secure development processes.

Best practices to improve security

Let’s dive into more detail around more tactical tips and best practices that you might want to consider when improving the security in your teams.

1. Security task forces

Create temporary task forces that are predominantly a group of developers, likely from cross-development teams, who all share a common goal of improving security adoption among developers. They help bridge the gap between developers and security by encouraging ownership of security in their teams through process and workflow changes. Note that this is a temporary group, and while it may share similar goals to a security champions program, it’s not as formal nor as long-term in goals or strategy.

2. General security Slack/Teams channel

We all need another channel, right? Having a channel for security questions means people always have a place to go when they need help. This channel is often just triage, with questions being routed to the right person or place.

3. Top 3-5 vulnerabilities

Most projects will have a small number of vulnerabilities that are either more frequent or more dangerous, than others. Rather than trying to train your development groups into being general security experts, provide quality training on the most common vulnerabilities that affect them. Be sure to share the risk and impact, which ultimately shows why this is so important to their specific project. If you can add examples, red team findings, or even production incidents, this brings weight to the importance of the issues.

4. Security champions program

If you are able to create a more formal program dedicated to improving the collaboration between development and security teams, this is an activity that has shown significant improvements in communication and rollout of security processes. For more information, read our deep dive on how others have seen success and failures in rolling out security champions programs to their organizations.

5. Make time for security

“Easier said than done,” we hear you say! This isn’t really a decision a developer should make. It’s the top-down responsibility of the organization to empower the development teams to dedicate time to security, as well as other activities that contribute to general engineering health. Leadership can dedicate percentages of sprint time to these necessary activities, which engineering, product, and other teams respect the importance of.

"There has to be a percent of job time dedicated to security. f it’s just extracurricular work, it’s never going to happen. That was actually one of the failures I had at [a previous company]. We didn’t have buy-in from the top. I was working the program grounds up, and it just wasn’t the right way to do it. As soon as we went to the top and said, “Okay. GM of this business or engineering head of this business, we want you to help us make this a part of the goals for the security champions,” you saw a different accountability model, a different engagement back with the security organization to help craft what the model would even look like."
Rinki Sethi, VP and CISO at Bill.com

6. Celebrate success

As mentioned in the previous section, celebrating success and creating security role models and best practices breeds further improvement among others. Whether during your security champions meetings, your security task force meetings, or even your department meetings, it’s really important to shout loudly about the successes of your security rollouts. While it’s important for the security team to help with this, it’s most effective when the development leadership recognizes individuals and teams for their success and provides a platform to share new tools, processes or even interesting vulnerabilities they’ve identified or fixed.

"We give out t-shirts that say, “Security Hero” on them. This is more exclusive, so it makes people want to step it up and really go above and beyond to make a security contribution. Maybe you eliminated cross-site scripting… you built it into the framework your team uses. Then that would warrant a “Security Hero” t-shirt. And then you get recognized in front of the whole company."
Kyle Randolph, CISO at Verkada

"Swag is key. And change it up, so you don’t always give the same hoodie or the same sticker or the same t-shirt. Change it up, because if people expect, “If I do this I’m going to get this thing,” it cheapens the experience. [Make it] somewhat of an unexpected surprise. They don’t know when they’re going to be rewarded, but they realize that there’s a culture of recognition. When the software engineering team gets together, every couple weeks everybody gets together for an all hands. We will sit down with the security team and call out and publicly thank people for very specific actions. They’re not asking us to do that, but it definitely goes a long way, and it promotes a cultural momentum that these are good things to do and that it is OK to take the time to produce better quality code. I’m an evangelist about that."
Zach Powers, CISO at Benchling

7. Communicate, communicate, communicate!

We’re saying it three times because it’s only when you think you’re overcommunicating that you are communicating at the right level. Make sure you’re keeping the development teams in the know when new changes roll out, why they’re happening, and what it’s going to look like, before it happens. This could involve internal blog posts describing the new changes, demos that help with internal enablement, and of course, lots of reference documentation to help support your teams. If there are any problems, be sure to fail fast, fail forward, fix the issue, and continue with the rollout. And while you’re communicating, be sure to also listen.

8. Build trust, not fear

In order to get someone else to do something, it’s tempting to share the impact of what would happen if that thing wasn’t done. While it is important to understand risk and use that as part of a prioritization exercise, it’s not a useful stick to use over and over again. To build up trust, work with the development team to build up and jointly understand the risk; be proactive about measures, whether tooling or advice; and start slowly in a diplomatic way. Trust takes time and a shared goal which you both agree on and work towards to make sure that you’re not doing things because they’re mandatory, but rather doing things because they’re necessary. The difference is subtle but extremely important. The security team is there to make the development process and work easier, not be a blocker or an annoyance.

Techniques to improve adoption

In addition to some of the bigger tips and programs mentioned above, there are additional techniques that can be used to improve the developer experience while performing security tasks. Here are some of the tips shared by those achieving strong developer adoption:

1. Make the right thing the easy thing

There are many good reasons why developers should spend time on security issues or tasks. However, all it takes is for a process to be complex, or even just a little unclear, for someone to avoid doing it or put it off for later. Making the path easy, predictable, and achievable — in as much of a self-serve way as possible — can really help developers do the right thing, easily. Examples might include providing a template in Jira that developers can clone that has clearly defined issues or tasks that need to be completed.

"We developed our automation, deployed it, and then I went on paternity leave for a while. And then COVID happened. We didn’t really advertise the automation or push people to use it, but what we found was there was roughly 60% rate across teams, where around 65% of the PRs that we created so far have been merged and closed, which is a significant number considering we hadn’t sent any communications out to developers regarding this. This tells you that everyone wants to do the right thing, but maybe doesn’t have time to do it. So when you make it as simple as possible, they do the right thing."
Yashvier Kosaraju, VP of Security, Compliance & IT at Sendbird

2. Good documentation

Never leave your developers stranded in a place where they don’t know what to do, don’t understand the process, or don’t know how they should use their tools. Good documentation is paramount to successful onboarding and a smooth adoption that is more self-serve, rather than needing individual from another person or team. It’s important that when teams use documentation, they keep it up to date. As teams build new processes or automations, it’s important that they document it for other teams to follow. Asking developers to help with the documentation to make sure it meets their needs and is accurate is essential in making it relevant.

3. Automate, automate, automate!

To make it easier for developers to adopt new tools and practices, make it an automated part of the developer workflow. Additionally, automate it in the right place for the team. If your team likes GitHub actions, make sure the tool runs as a GitHub action. If they love feedback in IDEs, make sure they’re aware of the tools IDE plugins.

4. Use tools that are designed for developers

If your development team is really struggling with running tests, understanding results, or fixing their security issues, it could well be because the tooling they’re using is not designed for them. Traditionally, the security team would perform security testing and audit a codebase or application, giving the development team the results in a problem-oriented way. This creates friction for a developer who is much more familiar with a developer tool that integrates with their existing tool set, that they are most familiar with, which produces results that have suggested solutions they can take to remediate the issues. This is what empowers developers to fix. There’s more on what makes developer tools in the next section.

5. Consolidate tools

If your developers can use one tool or vendor for multiple types of security risks, it helps with consistency and adoption. For example, if a developer is in their IDE and has some Java files open, a pom.xml build file, a Dockerfile, and a Terraform script, do you think they want them to run four tools to test for vulnerabilities and issues, or just one?

6. Identify ownership

Ownership is always important, although this time we’re not talking about whether security or development own a task or responsibility. It’s more about which development team should own an issue. It’s easier for developers who work on a project or piece of code than someone not familiar with it. Try to tag projects with owners so that issues can be routed to the right people in the development teams. This is most important for projects that are shared and used by multiple teams.

7. Support your developers

The days when security teams could pass issues to a developer and expect them to fix them by themselves are behind us. Security teams need to be there to help when security expertise is needed, and to aid developers with issue resolution where required. Working as one team and sharing not just goals but also problems and solutions is key to the success of a developer-first security program and its adoption.

8. Avoid overwhelming developers

Depending on the maturity of your development teams and developers, each will want to consume security activities at different levels to avoid being overwhelmed by issues or activities. For example, a team which is only just starting on their security journey would be overwhelmed seeing 100,000 vulnerabilities in their backlog. It’s better if they start by only looking at the security of the changes they’re making, to start off with, and likely just the highest most critical issues that are being introduced in those changes. The threshold should be movable based on your team’s capabilities and capacity. For those that are more advanced and ready to look at backlogs and improve the overall security of their projects, it’s important to prioritize issues so that they see the top five or ten issues they should fix, rather than the 100,000 that they don’t want to know about right away. This should be an exercise of prioritization and triage that the security team and development team leads or security champions can work on together.

What makes a tool a developer tool?

The way to scale security in organizations is to empower developers to take responsibility for the security of the applications they are creating. This shift in responsibility and attitude to developer security in our culture needs to be matched in our processes and of course our tooling. Tools need to shift to be developer-friendly and provide what developers need for them to do their job, develop and deliver code securely.

The big difference between what developers and security teams want from a tool is intent. Traditional security teams are auditors. They want to test, report, and provide developers with a list of problems. Developers don’t want this. Developers want a tool that they are familiar with, that acts predictably, and gives them actionable solutions instead of more problems to research.

Here are a few of the common traits of a great developer tool:

Successful self-serve adoption

Practically all successful developer tools have great self-serve usage, including easy onboarding, intuitive workflows, and great documentation. This is the way developers like to consume tools, and it forces vendors to ensure that their tools relate to developers without a salesperson pushing it. Always look for tools with a proven track record of self-serve adoption by developers.

Seamless integration into workflows

Developer tools, for the most part, look to meet developers where they are instead of asking them to open yet another tool. They integrate elegantly into their IDE, Git, and pipelines and provide value at the right point in time. Integrations are about more than technical hooks — they need to adapt to workflows and best practices or they will be rejected.

Rich APIs and automation-friendly

Although opinionated integrations are needed to get started, a developer tool must also be a Swiss Army knife. A rich API and automation-friendly client (CLI/SDK) are mandatory, both to tune the tool to every pipeline and to allow the community to build on it. If you can’t build on a tool, it’s not a great developer tool.

Adoption by open source and community

Developers look to fellow developers to validate a tool’s credibility, and open source adoption is the best indicator of that. Seeing open source projects integrate a security tool or finding open source projects that are built on it are great developer community validations. When inspecting a security tool, inspect its adoption amid open source and draw your own conclusions.

Expert guidance, automatic fixes

A great developer tool makes it easy for a developer to do the right thing, even when they don’t know what the right thing is. For example, any linter will find bad code, but a good linter will tell a developer how to fix it. A greater linter will apply fixes with a click or even automatically.

"The traditional way [to handle application security] has been to create tickets that go into the backlog and then have your management team, which owns an SLA, talk to the team. But there is friction in that. The easier you can make it for someone to be secure, the more likely they do it. When we think about the tools that we want to provide to developers, the first questions we ask ourselves are, “how can we make this easy for them to use?” and “how can we make it easy for them to fix the findings?” Tickets are nice for tracking, but they don’t make it easier for [developers]. What makes it easy for them are comments and PRs. Or a PR to fix the issue itself. Or [guidance] that tells them what they need to do. That is the philosophy we have been using."
Yashvier Kosaraju, VP of Security, Compliance & IT at Sendbird

Application context

Developers have a never-ending checklist, so a developer tool needs to help them prioritize their work. This means providing the context to know what’s critical and what can wait, as well as how work fits into the bigger application picture. If the action items from different tools should be prioritized against one another, sharing a backlog can increase efficiency and results.