Talking visibility, scalability, and relationships in secure development with Phil Guimond of ViacomCBS

I recently caught up with Phil Guimond, Principal Cloud Security Architect at ViacomCBS. He describes his role as a fancy way of saying he likes to be involved in All The Things „¢. This includes cloud security and architecture, application security, penetration testing, and digital forensics and incident response, and even vendor reviews and risk management from time to time. He works in a very cross-functional team.

We had a great discussion, and I wanted to share it with all of you. We talked about security practices, how to approach modern secure development, and more. I hope you enjoy the talk as much as I did.

Firstly, I asked Phil what his general approach to security was like, and what he considers to be the most important things to focus on when building an information security program.

Guimond: My approach to InfoSec is visibility, scalability, and relationships. This is a system I’ve developed over the years thanks to many of the amazing people I’ve worked with, including our Streaming CISO, Jonathan Keith who has been a great mentor to me. Over the last several years while working on incident response as a consultant, I noticed there was a lot of missing information. No logs... nothing. I was basically flying blind almost all the time and had to rely entirely on standard CLI tools to identify problems. My number one recommendation on each incident was to build out a proper logging system to provide visibility. It doesn’t matter how much you spend on security tools if you can’t see what you need to protect or what happened if something was compromised. How do you know what to protect if you can’t see what happened?

Visibility is often used as a key indicator allowing us to prioritize risk and insights throughout the organization. I was keen to hear from Phil how he uses visibility data in his day to day work.

Guimond: Visibility allows us to respond to an incredible amount of issues anywhere we have them, including old, new, and emerging threats. There are so many things you can do with visibility that it’s sometimes shocking. It’s definitely exciting when you encounter something new and your visibility tools show you exactly what happened. Visibility dramatically improves incident response, penetration testing, application security, cloud security, risk and vulnerability management, and pretty much anything.

In the case of supply chain attacks with compromised open source libraries, or with open source library vulnerabilities, you’re able to see which applications are using these libraries and quickly eliminate them from your projects, upgrade them, or fix them yourself. When it comes to compromised cloud or network infrastructure, you’re able to quickly find the culprit, where the attack started from, and what the attackers did with the resources they targeted. And most importantly, which bad practice led to this compromise. Then you can easily isolate them from any infrastructure and eliminate all of their unauthorized actions in one go. Or if you had a compromised employee laptop, you’re able to see what the attacker did with their credentials and which resources they attempted to access.

When building large-scale InfoSec programs, it’s important to have visibility into as many things as you can. You want to understand which technologies you’re using, which open source libraries exist within a project, your entire cloud infrastructure including IAM policies, buckets, containers, infrastructure as a code and stuff like that.

Being able to see all, or most, of your problems is huge. You can see which teams are using good practices that lead to better outcomes, and which ones aren’t. And when you find teams not using best practices, it’s the perfect opportunity to reach out and help them. When engineers start using best practices, the vast majority of vulnerabilities suddenly start vanishing. And you have the added advantage of giving the engineers live training. Of course, visibility alone isn’t enough, you still have to be able to fix stuff at scale or you’ll always be fighting fires.

The last two parts are very important for me. First of all, recognising that the goal isn’t just to find, but to act on those findings, and secondly, making sure you’re set up to scale. Using data across the entire organization allows you to identify the state and coverage across your product teams and use that data as part of your risk assessment. Scaling across teams when you’re not ready can share pain across teams rather than best practices. I asked Phil what scalability meant to him.

Guimond: To me, scalability means doing something once and having that fix everything. For example, if you have multiple project teams that all need a single specific piece of functionality, then the easiest way to fix the problem of having a dozen codebases that all do the same thing, and all have their own unique vulnerabilities and challenges, is to develop one central project which solves this problem, and then let all these teams use it. Basically, you want your solutions to scale for as many teams as possible with the fewest amount of actions and projects. Part of that is also creating better outcomes by using best practices.

I’m also a big believer that you cannot have true scalability without visibility. If you can’t see what you’re supposed to protect, how do you protect it? Do you even know it’s there in the first place? In most cases, you wouldn’t unless you had a lot of inside knowledge that comes from working for the company for years. What happens if the person who had all that inside knowledge quit or got hit by a bus? It’s suddenly gone. And you still wouldn’t have much insight into the vulnerabilities within the projects. That approach doesn’t scale at all.

Lack of scalability will quickly overwhelm your security team and the engineers who work with you. Neither you nor your developers have time for this approach. Scalability is also a big problem with today’s approach to penetration testing. Criminals absolutely do not care about your scope, or the few small areas you tested. Now I’m not knocking penetration testing. I think it’s absolutely necessary, but your standard pentest is going to be expensive and not scalable at all. In today’s cloud-first environment, you absolutely need a whole-of-infrastructure approach to pentesting. And it has to be continuous. 

As Phil mentioned, scaling too fast, or not at all, can overwhelm the security team, particularly if the processes, documentation, etc, aren’t ready for the challenge of scale. So I asked how Phil achieved to scale successfully.

Guimond: Through a visibility-first approach with tool consolidation into a single pane of glass. I know this sounds cliche, but it really works. You want to make the lives of the engineers easier, not harder. Forcing them to use 25 different security tools to secure their projects is a ridiculous approach and that’s how you get endless pushback, lack of scalability and visibility, and much less interest from business units in actually wanting to work with security teams.

So if you can find a way to make their lives easier, by giving them as few tools as possible which handle the most amount of problems in a way that’s scalable, then you’ll have better outcomes. Much better outcomes. And that translates to better visibility, which in turn translates into vulnerability management at a massive scale. Traditional vulnerability management does not scale; it’s dead in the age of cloud computing. Best practices will eliminate most of the problems. And for the problems that remain, it’s important to build compensating controls when it’s not possible to fix the issues without significant architectural redesign.

I asked Phil what are some of the best practices that he would suggest others can follow to help get the best visibility data for their organisations.

Guimond: Tool consolidation and elimination. You need tools that expose your bad practices AND help you fix them with as few steps as possible in as few offerings as possible. You also need to eliminate tools that provide no real value to your security team or developers. If you hire people just to handle a single tool, or a set of tools, then this can lead to silos which can often result in a security approach that doesn’t scale.

You want your team to be as cross-functional as possible, so let them spread their wings and swim a bit in addition to whatever else they’re supporting. If individual contributors are in meetings all day, nothing will get done. Work with vendors whose products have a solid API that will provide all of the information which their own dashboard provides.

Now, as for how you can achieve this tool consolidation, you can work with vendors who protect massive portions of your infrastructure in a single product offering. For example, open source libraries, containers, SAST [static application security testing], infrastructure as code, cloud inventory, etc. Then, you would pipe all of this data into a dashboard which displays it in a single pane of glass, or as few panes as possible. You want to be able to drill down into problems by category: application, network, infrastructure, or even by project.

It’s key to get buy-in from parties that can either support or enforce, or from people who are able to actually do the work and get things done. I asked Phil what advice he would give to people wondering who they need to get support from before going down a path of achieving visibility across their groups.

Guimond: If you are building a supportive InfoSec team. First you need to build relationships with the teams or business units that you’re supporting. Reach out to them, schedule a meeting, ping them on slack — whatever works for you — and talk with them to get a sense of what their challenges are. Once you understand this, you’ll better understand how you can help them. If you know a team has had specific security issues in the past, it’s important to come up with a solution to help them fix those problems. Developers listen if you do.

It’s very important to listen to the team’s directors, managers, developers and engineers, and understand their point of view, especially points of views which may often be different than your own. If you don’t accept any other perspectives, if everything must be “my way or the highway,” nobody will want to work with you, and you’ll have poorer outcomes for your InfoSec program. But once you’ve built those relationships, it’s often easy to get the buy-in you need. People know you’re approachable and won’t be breathing down their necks. Then you onboard them with best practices and toolsets.

Finally, I asked Phil what general advice he would give to someone wanting to create their own security team and program in an organization.

Guimond: Hire a diverse and cross-functional team. People with different backgrounds than you are going to look at things differently. Understanding other perspectives will help everyone grow professionally and keep teams innovating. For example, I spoke with a team who had 20 different projects which all solved the same exact need. Each team was using their own codebase, and each one had their own vulnerabilities. So they kept trying to fix these problems, one-by-one. My approach was to show exactly how I would fix common vulnerabilities associated with this project, but they said something that surprised me: they not only did that, but they also created a single project for the same need and let each team use it. This solved the problem of going back and forth between all these codebases, and eliminated the need to pentest 20 different projects. Now they just need to test one. Sound familiar? It’s the example I used earlier!

So it’s important to keep an open mind, listen to other people’s approaches, and don’t expect to do the same thing over again at every company. Each company and team is unique.

As for why a cross-functional team is important, what happens if you eliminate several tools which no longer have enough value to justify paying for? The teams who previously supported these tools may now need to become something else. If they already trained the other skill sets, then they’ll adapt much more quickly to change. And InfoSec is always changing. This lets them continue to grow in their career without having to go somewhere else, and it makes your security team more adaptable.

Now, I’m not saying everyone needs to be a jack of all trades, but it really helps when multiple team members can fulfill multiple roles, while still specializing in their primary duty. This is the perfect opportunity to let your team develop new skills, and to train them as well. It helps attract more diverse candidates when the team has the opportunity to learn other things and grow professionally.

You don’t want your team focusing entirely on one thing, or it will harm them professionally and stunt your InfoSec program. And as a manager, if you let this happen, you’ve failed them and your company. You’ve limited their career opportunities by providing them a silo and now your InfoSec program is stuck within the limits you’ve defined for it. And over time, the technical debt will lead to increased blind spots, less scalability, and a lot more vulnerabilities.