DevSecOps Program Success

Improving secure development is a journey that takes time, and starts with getting visibility into the existing security processes and practices that are done by each team today. If this isn’t done in an empathetic way, this process can be perceived as a reaction to development shortcomings. When others think there’s blame or judgment, it’s easy to get defensive responses. A great approach to avoid this is to ask the champion to complete a self-assessment scorecard of their team’s processes and practices. This allows good reflection time without creating a them vs. us dynamic. The scorecard is a set of questions that identify various security behaviours, testing types, processes, etc., which the development team can state how much is currently being done in each area. It’s common for teams to focus more on certain areas than others and that’s perfectly usual.

The next step is improvement and that’s where the security team can help and support the development teams. A great example of developer-led improvement was to ask the development team which items on the scorecard they wanted to improve. They only needed to pick a couple of things, followed by a commitment of when they would want to have implemented an improvement process. An example may be that they want to introduce some security questions into a code review, or perhaps they might want to eliminate all high severity vulnerabilities from their backlog with a CVSS score of 9.5 or higher. Whatever the goal, it’s important that it’s developer led and owned. The security coach is there to help the champion make it happen, whether they need education, advice, pipeline changes, etc., they could be there to make that security champion rollout the change they want.

One core premise to this, however, is that the success (and possibly a formal KPI) of the security coach hinges on the success of the security champion. Only if the security champion can complete their goal or progress plan is the security coach successful. This joint measure of success underpins the joint goal of secure development that both teams are together trying to achieve.

Another common activity driven through the security champions program is vulnerability management. Visibility into where vulnerabilities exist can be great, as it really highlights the hot spots of your applications and deployments. However with larger projects and deployments, this can soon become overwhelming with the sheer number of vulnerabilities a developer may be faced with in their project backlogs. The act of triaging, ignoring, prioritizing vulnerability backlogs as well as newly identified vulnerabilities can be done by both the development and security organizations. The development teams truly understand the application flows, architecture and design, far better than the security team. Likewise, the security team understands threats and risks better than the development team. Working together with open communication channels through, but not exclusively, a champion and coach relationship, makes these discussions, and assessments much easier and quicker.

How to start and rollout a security champions program

First of all, remember that clarity and simplicity are very important. When starting up, make sure there’s enough information available for people to read and understand what the security champions program is, the goals, and add the sponsorship and importance to the business. Make sure it’s written with a developer in mind, so get their feedback when writing it.

The most important lesson to remember when thinking about the initial size of the program is not to go too big, too fast with your rollout. Identify and learn best practices specific to your organization and your teams that can be used for further rollout. When selecting teams to be a part of the initial rollout group, consider which teams of your organization have the following:

• The most important services, applications, and teams in your business that are highest risk

• The teams that are most mature in security and development practices that can adapt to change

• The teams with individuals that have existing relationships with the security team

• The teams with individuals that are security curious and open to improving their team’s security posture and hygiene.

Starting with just a few teams to start reduces the risk of a rollout failing as you can spend plenty of time as needed with each team as needed. When identifying team needs, you could also consider the common denominator of problems across the teams, if known this early. For example, if you roll out the program across three teams and they all want to start adopting threat modelling, you can share feedback across teams and focus efforts on fewer moving parts.

Picking the right security coach can be just as important, as they are the main contact point for the development teams. Choosing a good communicator, who (preferably)has previous engineering experience and is empathetic to developers, can make a big difference.

Be generous with your rewards and recognition early on to provide encouragement and grow interest in people progressing further. Make sure you start building up a best practices guide to use later for other teams. Additionally, update any existing documentation and guides as needed so that the next teams have up to date and useful information.

As you grow out into the business, you might want to consider picking other groups that live in different parts of the organization to get a better representation of all areas of your business. Maybe consider a roadshow to gather more data about what other help is needed that you’re not providing, as well as sharing your successes from the existing teams that are involved in the program. Additionally, sharing these successes within a program so teams can see how each other are doing can often create a race and teams end up making each other better.

As you roll out the program to a development team, make sure you reward their actions, new learnings, and responsibility with more responsibility! This may sound strange, but we ultimately want development teams to become more autonomous. As they show themselves to be more capable and willing to own security, give them more authority and ownership.

Gaining developer adoption

Understanding the pain points and needs of the development organization is key to the developer adoption of your program. Irrespective of whether you decide that you want to grow your program entirely with volunteers or by management selection or a mix, you still want to make sure the individuals in the program participate, and contribute to it.

A key framing for the reason why the program is there is important. Clarity over the program goals as well as the roles and responsibilities within the program for development and security individuals really helps everyone feel comfortable within the program and give that engagement. One important program characteristic is that development and security are a team, working together on a joint goal of developing and delivering secure code and applications. It’s important, particularly for developer buy in and enthusiasm for secure development work to largely be considered just development work.

It’s important to always be mindful of the development audience when choosing topics of education, asking developers to perform tasks and even tracking tickets etc. Developers often care more about best practices rather than learning basic information that they’re often already aware of. Making content tangible, technical, and  applicable to problem solving is key to its effectiveness.

Details are often important, such as where communication and interactions happen. For instance, if you want to raise a ticket for the dev team to work on, make sure you use the dev team’s preferred ticketing system. If they’re already using Jira, for example, create Jira tickets for them. You should consider every additional tool or service that you expect the development team to use as an additional barrier to their participation.

As mentioned, it’s equally important to provide clarity in the individual responsibilities and duties that are expected from security champions. It’s a lot easier here, when there is an official role, that states an amount of time that they should specifically spend on security related activities for the team. Either way, stating two or three goals and activities you want champions to work on over the next 3-6 months is plenty to avoid overwhelming your champions and focus them on the most important activities they should be spending their time on.

We mentioned the importance of rewards and recognition earlier, sharing some ideas for how to show the value from the business of these activities. Celebrating success doesn’t always mean flying someone out to a conference though. Most of the time it’s about highlighting someone’s efforts to encourage them to do more, effectively being an advocate or champion for them. At the same time, this behavior encourages others to want to go the extra mile for the same appreciation.

What does success look like?

First of all, it’s imperative that you don’t expect amazing results in weeks or even months. The reason you should be creating a program like this is to improve the development process and practices so that you are able to deliver software securely. There isn’t an overnight fix for this, and when dealing with people, these changes take even longer. Realistic timeframes for change and success are needed to ensure decisions are made correctly rather than to hit an arbitrary target.

Adoption

A key stat for any security champions program is how well it is being adopted. As we have said, it’s important for each champion’s participation to be voluntary, so a sign of success of the adoption could be the total number of champions you have, the growth of the group over time, and the retention of champions.

Another good metric to track is the number of development teams your security org has reached through the champions program and what proportion of the development org that covers. Remember that it’s easy to fail by growing too fast, so targets should be achievable and organic rather than aggressive and forced.

Engagement

Engagement of the members is the next stat to track which shows if the program and relationship is effective. Tracking whether champions are indeed investing 10-20% of their time in security activities is hard and likely not something you want to do anyway. This is more about officially supporting engineers by making this security work a part of their job, rather than an add on which may or may not happen in their own time. Also, the time commitment is ballpark and will ebb and flow from week to week based on the demand.

A much better metric to track is around what they’re engaged in from regular meetings to initiatives they’re running in their teams. The former is easy to track by making note of how many champions regularly attend monthly calls, or engage in weekly touchpoints with their security coaches. This data shouldn’t be used against the champions, but rather used to improve the content and activities discussed in the program to increase the appeal and relevance for the champions.

Impact

As previously mentioned, it’s important to empower the development org to own changes and improvements in their team. It’s up to you how far you’re ready to take that, but understanding what the team is taking on, what they’re improving, and what they’ve completed is a real measure of the impact the program has had.

A good way to achieve this using the scorecard technique is to ask the development team to self-assess their practices and processes. First of all, how many people are doing their scorecards for their teams? After you work with them so they are aware of the greatest risks, the quickest fixes etc, have they put together an improvement plan that they own, and they lead, with support from the security team? And of course measuring where teams are against their plans. It should be a KPI that the champion and coach both own that measures this progression of improvement.

Progress

As mentioned already, there’s a difference between the types of education a champion and a developer not so interested in security might want. And while that’s fine, it’s still good to measure and track their learning progress. Once you have decided upon how you want to educate, whether that’s internal, or external certifications, or even hours or hands on security work, use a belt or medal/certification model that shows how skilled your overall teams are, with targets of improvement over quarters.

Dashboards

The final area to measure, which may be seen as more of an influenced metric, are the product security dashboards that list stats on your vulnerabilities, number of tests, adoption among teams, etc. Without context these figures can easily not reflect the quality or risk that a project has. For example, the fact that one team tests more than another, will likely mean they will show more vulnerabilities, as they have the visibility and insight into where their issues are. This doesn’t make them a higher risk, despite showing greater numbers.

Being able to show impacts by team with context and reasoning is very valuable. While measuring these metrics, be sure you also measure processes and practices that influence them. For example, having your team run threat models for every feature they write heavily reduces the risk in that project. It will have an impact on the number of vulns found when developed, and this context adds great richness to the results.