Secrets Management With Doppler's Brian Vallelunga

[INTRODUCTION]

[0:00:30] ANNOUNCER: You are listening to The Secure Developer, where we speak to industry leaders and experts about the past, present, and future of DevSecOps and AI security. We aim to help you bring developers and security together, to build secure applications while moving fast and having fun.

This podcast is brought to you by Snyk. Snyk’s developer security platform helps developers build secure applications without slowing down. Snyk makes it easy to find and fix vulnerabilities in code, open-source dependencies, containers, and infrastructure as code, all while providing actionable security insights and administration capabilities. To learn more, visit snyk.io/tsd.

[EPISODE]

[0:01:12] Danny Allan: Hello, and welcome to another episode of The Secure Developer. My name is Danny Allan, and I'm joined today by Brian Vallelunga, Founder and CEO at Doppler. Brian, welcome to the show. How are you?

[0:01:22] Brian Vallelunga: I'm doing well. Thanks so much for having me.

[0:01:25] Danny Allan: I'm excited to have you on, Brian. The reason for that is, this is actually my first show as CTO of Snyk Security. It's really important topic that we're going to dig in today, which is secrets management. So I thought, maybe, you could just introduce yourself to the audience, and share something about the inception or beginning of the Doppler technology and company.

[0:01:48] Brian Vallelunga: Yes, sure. I am Brian, one of the founders and CEO. Doppler is a secrets manager, designed for developers and DevOps teams, with the real goal of being the central source of truth for all your secrets across all your teams, projects, and infrastructure. You kind of think about it as like GitHub for secrets. Before we kind of like jump too much into it, I think it's first probably worth sharing a little bit about what are secrets, so people know what they're being told to care about in a way.

Secrets are highly sensitive pieces of data. They're typically API key. So if you're using Snyk, or Stripe, Twilio, you're given an API key as a developer, or database rails or certificates, anything generally super sensitive. Also, we kind of think of configuration as well. If you have like a port variable, or a logging statement or feature flag, that's typically all bundled under secrets. So you can kind of think about it as like, every application has three parts, usually. They have code, compute, and secrets. There's been a lot of great tools for code and compute, but not really much for secrets, and that's where Doppler steps in, and is trying to be that GitHub for secrets. A little background about how we got started, if that'd be helpful.

[0:02:55] Danny Allan: Yes, that'd be great. Love to hear that.

[0:02:57] Brian Vallelunga: Doppler has a pretty unique founding story, I think. It actually started out nothing to do with secrets at all. It started out as a crypto machine learning marketplace, and all the buzzwords to one. This is very much one of those projects where it felt like pushing a boulder up a hill. You move like one foot forward, and slip five feet back from exhaustion. I was working on this crypto machine learning marketplace while I was at Uber. So I'd work at Uber during the day, and I'd work on Doppler at nights.

At some point, I'm just like, eight months in, deeply frustrated, nothing's working from a product go-to-market perspective. I decided to take a trip to Mexico, and with the goal of just taking a break from it all, and relaxing, and resetting. The first day there, I just realized that I'm not going to be able to do that. I keep thinking about the business, and I realize that it's not going to work no matter what I try. It's just like, I'm either not the right founder, or it's just too early as a business.

I started looking at all the problems I was facing while running this, and managing environment variables, and secrets, and API keys just was a theme that just kept popping up again, and again, and again. I've always been inspired by other founders. I think, Stewart Butterfield from Slack, is like probably one of the best in the industry. I would say, like failing upwards. He creates a video game and the video game fails, but out of that was born Flickr, which becomes a big success. Then, he goes again and tries to create a video game, and that ultimately ends up failing, but born out of that was Slack.

So I was like, "Okay. Can we do the same thing with Doppler, the crypto machine learning marketplace?" Secrets is just like this problem that I kept feeling and facing again, and again and again, and all these weird, interesting ways. It wasn't just like one problem is a bunch of problems. A couple too that really come to mind. We had a contractor and I felt really weird at some point to give them all the crypto keys. Then, at some point, when we became just a machine learner marketplace and dropped the crypto, we were using Stripe, and it felt weird to get into Stripe API key. You give them this production API key, and then they hold on to it forever. Even if you let that contractor go, they still have access to that key, unless you rotate it. That just felt very scary to me for them to have access to all that data.

The other was, we accidentally put the Stripe production key in staging, in this stage you want to prod. We didn't know that that's why we weren't doing transactions for a month. We just thought the business was failing. There was a lot of issues like that that really came to mind when I was thinking about this. I come back from Mexico, and I go to this dinner that Stripe was hosting, and it was a Stripe Atlas dinner. Where they would take 50 founders and developers, and they put them all in a room, give them some pizza, and they say, "Talk." I just asked the room, "Hey, am I a shitty engineer, as the world's broken, I can't tell any more, is secrets are real problem?" About 50%, 60% of them said that they are having problems managing secrets.

One woman in particular came up to me and said – this was a Wednesday, and she's like, "I've had three outages this week, having a solution by Sunday." I was like, "No, no, we cannot – that's not possible." And she goes," I don't give a fuck. Have it by Sunday." I was like, "Okay. It's a real pain point. There's something here." So I started talking to other companies, small, big, and it just seemed like everyone was struggling. The universal theme across all of it was that, individual developers don't really have much outside of a .env file, which has a lot of problems on its own, and in security.

Then, big enterprises would either build their own tools, and that would be built for a moment in time, or – and when scaled past that, or they were buying HashiCorp Vault, which was not really made for that use case. It was really made for completely different use case, and built for security, not built for the everyday developer. The security teams would buy it, force the DevOps team to install it, and then, developers would basically run away from it, or they run from it like the plague. I kind of joke that a lot of our angel investors invested because they bought Vault and paid the price. We believed that we could build something for developers from the get go, that had all the developer workflows that they wanted, because it's super important in building a good solution. That's kind of how we got started and how we learned about the pain point by facing it ourselves.

[0:06:48] Danny Allan: So there's a lot to touch on there or to unpack, but was this your first go around at founding a company? It wasn't, was it? You've founded some companies prior to this?

[0:06:58] Brian Vallelunga: Yes. I would say, Dopplers, like the first successful company, there were seven other companies in the past. All of them, I guess, feel like projects now compared to like the scale of Doppler. But yes, there was a lot of bumps along the way, and I just always believed that you just needed one good business to make it all worth it. So, learn why you failed, think about it, take a little break, and then get back in the game again.

[0:07:20] Danny Allan: Well, I love the founding story, because it gives you a good perspective on the challenge or the magnitude of the problem that we're trying to solve. I remember back writing applications early in my career, and you would hard code actually the database usernames and passwords. Then, you'd have to remember to scrub them out before you published it or shared it out to people. We've come a long way since then. But as you say, someone leaves a company today, the way that we handle those secrets is a major challenge.

I guess, I'm curious from a best practices perspective. Obviously, we've come a long way since hard coding, passwords and things, and secrets into our applications. What are the best practices, even outside of Doppler technologies? When people ask you, "What are the top three things that I can do?" What do you respond to that?

[0:08:03] Brian Vallelunga: Yes, I think it's a great question. I'll try to keep these completely outside of Doppler, just because I don't want it to be a whole session on pitching a product. I think the first thing you need is you need really good storage. To me, good storage means that it's encrypted, because you don't want your data not encrypted. There's a lot of good cloud providers that can provide that, AWS Secrets Manager, or Parameter Store, GCP, and so on. Almost every major past provider as well, like Vercel, platform has a secret manager in it.

The second thing, and this is super important, is that it needs to work in all of your workflows. If you're copying and pasting secrets around, inevitably, you're going from a secure place to an insecure place back to a secure place. On top of that code, your code requires the secrets, and that code is being automated into deployments. But if your secrets are being manually copy and pasted, you're guaranteed going to have a race condition, and then, does an outage. Or, break people's productivity when they're developing, because they pull the latest copy of code and all sudden, the secrets aren't there, and now, they have a broken build. they have to go figure out why and who has that secret that they should have. That's number two is, make sure it's kind of like built into the workflows.

Then three, I think is also super important, is that, developers need to like using it. I don't know of a better way of saying that than this. But like, I have found that if you build tools even that fit into the workflows, that are secure, if it doesn't have the right developer experience, which is arguably a very like qualitative thing to evaluate, it's very hard to have a quantitative opinion on this. But if it feels right, it just feels right in that DX experience, developers are going to run towards it, because a good DX is a productivity tool. It makes them more productive the more they use it. If you don't have that last part, then developers are going to always work around it, and you will never get the underlying security benefits that the company needs or the project needs.

[0:09:48] Danny Allan: You're given half a week to come up with a minimum viable product or capability that was simple. I'm being a bit facetious, of course. How long did it actually take to get something that was easy for developers? Because I have to imagine – I mean, I'm just thinking of the Snyk workflows that we have here. We have all kinds of different pipelines, and streams of activity, and different ecosystems in which we interact. How do you cover them all, and how do you make it easy for developers to do secrets management, and all these different varying pipelines with different outcomes behind them?

[0:10:19] Brian Vallelunga: It took a long time, way longer than I thought. I think we rebuilt the product three times, at least from the implementation side from scratch. The first thing to note is that every developer thinks that the way they do things is pretty common. But the reality is, it's actually not. They're, I would say, like 70% close to like most other developers, when you start cohorting them. So you start to see these big trends that pop up. As long as you can adhere to those big trends, you're good. In our case, we realised that developers and local development want a command-line interface that allows them to fetch their secrets and inject into their application. So they can use it. It has environment variables and development. And in production, you don't need to have this thing where you're reading from a file, and development, and production, you're reading from the environment.

We built a CLI, and the CLI automatically restarts whenever secrets change. If your buddy changes secrets for the same project that you're on, you automatically get those secrets, and it will restart the application. Then, when you kind of move into higher order forms of environments, like CI/CD stage and production. We found that most customers either want two implementation paths. They want it integrated into their infrastructure directly. So, write it into AWS Secrets Manager, write it into Parameter Store, or Vercel, or Nullify, or any of these other cloud providers.

Largely, they want that because that's what they're already doing today, and, it hasn't changed the SLA requirements. If you add Doppler into the hot path of your application being up, now, you've changed the SLA. We don't want to be part of that hot path, either. We just want to make sure that you always have the right secrets at the right time. That was the next big learning for us was like, okay, we need real native integrations that are super easy to set up, you want to click and you're done, you never have to think about it. Again, it runs behind the scenes. When a secret changes in Doppler, it will change in your infrastructure, and restart those application deployments.

Then, the last one was Kubernetes. There's a whole host of companies, especially the larger enterprises that are running on these extremely massive Kubernetes' deployments. They wanted a native integration into Kubernetes. Instead of reading from AWS Secrets Manager, to Kubernetes Secrets, to their application, it'd be – Doppler has a Kubernetes' operator, that then writes into Kubernetes Secrets, and restarts their applications with whatever rollout strategy they've already defined. That kind of integration across the board was good enough just as a green light, like just to get the ball rolling. But then, you have to take it up a notch because it unlocks the ability for using secrets, but not complete the developer experience. Developers at the end of the day don't want to leave their editor, they don't want to leave VS code. So we built a VS code extension, where you can edit your secrets side by side with your code. It's two-way sync and it's secure, so it doesn't write to the file system, and all that. But it behaves like a file, behaves like a YAML file, but it actually isn't a file. It's something else entirely.

That was really key, because then, developers never have to leave their editor. That's kind of like how we completed this, I guess, you could say this ecosystem for strong DX, is that, you don't have to leave your editor. When you change those secrets, it then replicates to every other developer on that team who has the right access controls and restarts their applications. When you are ready to push it into higher forms of environments, that you can open up a pull request in Doppler, and then save it, and then now, automatically right into your infrastructure, and restart those deployments. That was kind of like an end-to-end lifecycle way of thinking, and approaching secrets that just hadn't been done until now.

[0:13:26] Danny Allan: Do you see that as changing in the future? IDE is obviously where the developers live in whatever, IntelliJ, VS Code, you mentioned. As we see the emergence of backstage, and roadie, and IDPs, do you think that where you interact and manage with the secrets are going to change over time or no? You think the IDE is still going to be the primary interaction point?

[0:13:47] Brian Vallelunga: It's a good question. I think it really depends on how far in the future we're talking. I think the world is moving towards this identity-based authentication. Usually, you kind of see it on the password side first with like logging into Facebook or Netflix, you now have this thing called passkey that you can use. That's identity-based authentication in a nutshell. Whatever happens, passwords eventually happen to secrets, it just takes a much longer time horizon. But I think that's where we're going too, is eventually, instead of having a secret, you'll have an identity connection. For example, a customer could have an identity connection with Doppler and Stripe. And just the fact that you're logged into Doppler, and you're using the Doppler CLI or the Doppler Operator, you'll automatically be authenticated in Stripe. There's no API key that needs to be managed, or rolled, or anything like that.

I think that's where we're going long term. In a world like that, developers are now thinking like checklists, right? They're like, "Okay, I need Stripe, enable Stripe automatically. It's provisioned, it's set up, and you have all the authentication you need. You don't need to worry about an API key. That's my idea of where its going.

[0:14:46] Danny Allan: Yes. I've heard you said that identity is going to be the perimeter of the artificial intelligence world. It is true that it seems to be consolidating down around identity, and by extension of that, secrets. So, we were supposed to solve all this a while ago with secrets rotation. I mean, orchestrate secrets rotation. Why has that not worked or is it working?

[0:15:07] Brian Vallelunga: I think it works in a specific set of use cases, but it doesn't cover the whole landscape, which is actually the root problem. Every major cloud provider, AWS, GCP, Azure, all support secrets rotation for their database products. They don't support it for pretty much any other product, but they support it for at least databases. That's exactly the problem that we're facing, is like, there's a very specific set of providers that actually allow for secrets rotation. For anyone in the room who doesn't know what secrets rotation is, it's the idea of issuing a new credential. and revoking the old one on a cadence.

Let's just say, every 30 days, we swap out the lock and key on the door. That's what we're trying to do with API keys. The idea there is, if some bad actor or good actor walks away with these keys, they're only valid for 30 days, or whatever time period you set. Really, the underlying problem with secret rotation is very, very few providers actually support it. I'd say outside of databases, all the other major concentrations of risk like Stripe, for example, which is payments, access to credit cards, and bank accounts does not support it, and they need to. It needs to be a full industry-wide movement to support it.

Now, to support it is actually not that hard. It really doesn't take that much change. For each provider, all they need to be able to do is have two capabilities. They need to be able to have API endpoints that allow us to create, delete, and check a credential. And they need to be able to support two keys at a time, two API keys at a time. The reason why you need to is because, you don't want to bring down production, right? You broke one, and you issue a new one. In that time period, you now have caused an outage. That's very simple. There's some great companies like MongoDB Atlas that are supporting this, Twilio supports this. There are more and more companies starting to support this. We just need a bigger effort industry-wide, providers like Doppler and other secrets managers can provide these seamless secrets rotation experience across the board.

[0:16:48] Danny Allan: Yes, it's an interesting dilemma. One of the challenges that we have as an organization is getting that industry alignment on how to do those things. I've seen industry alignment work in the past, specifically in the payment card industry with data security standard, where you have these requirements. Like, this is the way that you handle credit cards, you do A, B, and C. Do we need something similar in the secrets' world, whether it be PCI DSS enforcing it or a SOC 2 certification? Do you support that regulatory approach to helping lock down the secrets that organizations are using?

[0:17:23] Brian Vallelunga: Absolutely. Absolutely. From two perspectives. One, from the perspective we just talked about, providers being able to provide the necessary thing so that secrets management companies like Doppler, or any other can provide the functionality like rotation. So, the only way that's going to happen is, if there's a compliance requirement there, especially the compliance requirement will be great at standardizing so that we don't have to build 20 or 30 different implementations of the same thing.

But actually, I think the most important part is not that, it's on the end company. Every company in my personal belief has a responsibility to its customers to keep their data secure, to keep their private data private. You can only really do that if you're securing the keys to the digital kingdom, right? If you say, "Hey, here's this door, and behind those doors is all the data, and we're not going to protect it." You can't really make a strong argument you're protecting the data.

SOC 2 is great at setting table stakes level of security that you need to set as a company. But one big gap it has, a secrets management. It just doesn't talk about it at all. I think, there needs to be serious controls in place. Controls are a way of evaluating an individual thing in SOC 2. But basically, SOC 2 needs to encompass secrets management so that every company manages their secrets securely. Without that, you have to play this very large awareness game of getting every company to care. Secrets management is one of those things that keeps the lights on. Like the best-case scenarios, you don't get hacked and your developers stay productive. It's kind of hard to make that argument at a mass scale, when there isn't a compliance requirement.

Just like if we didn't have a bunch of other things in SOC 2, like, make sure you have secure hosting or secure code. Those probably won't be industry standards as well. I think, SOC 2 has a strong requirement, or should have a strong requirement around secrets management.

[0:19:05] Danny Allan: Do you think you can get that alignment within the industry? I guess, would be my first question. And secondly, should it be published? I've always heard that you're putting a target on your back as soon as you say, "I am doing X." If you were to highlight, "Hey, I'm doing secrets management." Do you think that's an advantage, a good thing, a bad thing? How do you see it going forward?

[0:19:26] Brian Vallelunga: Do you think it's possible to rally the industry around to get secrets management include into SOC 2? I think it is possible, but I think it's a very long-term play. Everything that I've learned about how SOC 2 changes, it's a very slow thing, and requires quite a lot of navigation. I think this is where companies like Snyk, and Doppler, and other security-focused companies can join together, and build a big megaphone that says, "We believe, as an industry, this should happen. Now. There are real reasons why this should happen outside of – us, as a secrets management company being incentivized to encourage this.

For example, there are more hacks than ever before. Those attacks are attacking big companies and small companies. We think that cyberattacks are going to become more and more frequent. If you have one single piece of attack that unlocks the entire Holy Grail of data, attackers are strongly incentivized to go after it, so you absolutely have to protect this. I don't care if you stopped or not. I need the industry to care about doing something. Every time a major data breach happens, the customers are the ones that are really affected by this.

Think about what would happen if Uber got hacked, and every trip I've ever taken, or you've taken was now out public. Your home, your families, every restaurant you've been to, every place you shouldn't have been to that you have been to. Whatever it may be, all that data is now available, and probably getting sold, and being manipulated.

[0:20:39] Danny Allan: Well, we certainly live in a world where our digital lives are online. I would agree with you, I mean, this is why you and I are in the industry to secure applications, and secrets, and all of this. Where do you think it's going? Do you think that we'll be dealing with secrets five years from now, 10 years from now? Do you think that's just going to be automated into the engines? How do you see the industry evolving forward? I know that historically, secrets were this hard-coded text, and obviously, it's changed from there. Where does it go five, 10 years from now?

[0:21:09] Brian Vallelunga: It's a good question. I think there's one thing that's going to impact it tremendously, and that is AI. We're starting to see this now, the early days of this where you have, like GitHub Copilot. GitHub Copilot is supposed to be like a really good pair programmer with you. But we're starting to see evolutions past that, like AI agents that will be completely responsible for writing the entire code base. You can look at Devin, or Magic.dev, and there's a couple others out there that are aiming at this goal. They're not there today, but they will be maybe five years from now.

Really, the question is, what does secrets look like or identity authentication look like in a world where AI is writing some big portion of the codebase, or at least across the industry? So maybe it's like, some companies choose to have AI agents writing the code, others don't. I think what we're going to end up seeing is, probably a very standardized protocol, run identity-based authentication, and it's going to be end-to-end. If you're in AWS, for example, you're now automatically authenticated to Stripe because there's some intermediary, like Doppler, or other secrets manager that's facilitating that identity-based authentication.

So, you're in AWS, and Doppler knows that, you're now authenticated. Then, because Doppler and Stripe have a connection, you're now authenticated to Stripe. By being in an AWS box, you're now authenticated to Stripe. I think all of that is going to be dynamically provisioned on the fly, all of these identities with AI. So AI will say, "Hey, we have this product goal, or go-to-market goal that we're trying to accomplish." They will build up all the relevant infrastructure. When they're sending up that infrastructure, they'll be sending up the identities, and the connections between AWS, Doppler, or swap out secrets manager from Doppler, and then whatever underlying third-party service using Stripe, Twilio, and so on. That's the world I'd imagine we're moving towards.

[0:22:49] Danny Allan: Yes. I think this is why they say the perimeter of artificial intelligence AI is on the identity because it will be smart enough to swap the data, and the identity, and the authentication authorization protocols between these different worlds. It is true that AI is changing every single industry. In fact, I can't get through a podcast these days without AI coming up. One of the questions that we like to end with on the secure developer actually is around AI, specifically. If you could take AI to automate any part of your day-to-day life, or work experience, what would it be? I mean, we're in a new environment now where we can leverage it to create emails, or manage secrets, or identity, how would you use AI in a dream world?

[0:23:33] Brian Vallelunga: There's one thing I really wanted of AI, and that's not generative text or images. It's the ability to solve hard problems. I don't think we're quite there yet, but I think we're moving in that direction. Like, if we had an AI agent that was able to solve any arbitrarily hard problem. So that could be calculus, like some math equation, or it could be just like, "Hey, my business is failing in this area, help me understand that."

Whatever it is, it doesn't matter the domain, it can go become a domain expert, either through public data or private data. That'd be incredibly powerful. Because not only do I think I need this, but I think humanity needs this, right? Like, go solve cancer in its entirety. Like, I'd love to put an AI behind that, or go solve faster than light travel, and be able to test things in the real world, but also simulate them, and be able to know when to test in a virtual world, and when to test in physical world. I think that is actually something humanity needs the most, and it can bubble up to the biggest of scale, like solve lightspeed travel, and as small as to like help a CEO make his business a little bit better. But I could choose one thing to build, that's what it'd be.

[0:24:32] Danny Allan: Well, they say we always overestimate how things will change in the next two years and underestimate in 10 years. I think that's definitely true with AI that we – right now, it's being used for natural language learning, and it's being used for image generation, and it's helping me summarize things. But I don't think that any of us really fully understand 10 years from now, how it's going to be used to help us focus on solving cancer, whatever the challenge happens to be. It definitely is an exciting time to be in the technology space.

Well, Brian, it's been fantastic to have you on the podcast. Thank you for your time and sharing your insights around secrets. It's a big motivation and driver for us here at Snyk, and all of our customers. So, appreciate your time and look forward to speaking with you again.

[0:25:16] Brian Vallelunga: Thank you so much for having me, and thank you so much for everyone who's made it to the very end for listening this far. I hope it helped.

[0:25:21] Danny Allan: Thank you, and thank you, everyone, for listening. We hope to hear you and see you the next time on the next episode of The Secure Developer.

[0:25:29] Brian Vallelunga: Thank you.

[OUTRO]

[0:25:33] ANNOUNCER: Thanks for tuning in to The Secure Developer, brought to you by Snyk. We hope this episode give you new insights and strategies to help you champion security in your organization. If you like these conversations, please leave us a review on iTunes, Spotify, or wherever you get your podcast, and share the episode with fellow security leaders, who might benefit from our discussions. Would love to hear your recommendations for future guests, topics, or any feedback you might have to help us get better. Please contact us by connecting with us on LinkedIn, under our Snyk account, or by emailing us at thesecuredev@snyk.io. That's it for now. I hope you join us for the next one.

[END]