Making Security A Requirement With Gergely Nemeth

"Guy Podjarny: Everybody wants to be secure. It's a good thing, right? People want to do it. It's just about balancing it out with the rest of the activities that you have."

"Gergely Nemeth: It's really about thinking how an attacker would think. What's really valuable for your business?"

"Guy Podjarny: If you ask yourself what is it that might ruin your company tomorrow? Generally, not investing in a security component fits the bill. Once you have a mistake or a flaw in one page on the site, you've compromised the entire domain."

"Gergely Nemeth: Most of our users wasn't really into security. It wasn't really a huge priority for them. An attacker has to be right only once. And you have to be right all the time."

[00:00:41] Guy Podjarny: Hi. I'm Guy Podjarny, CEO and Co-Founder of Snyk. And you're listening to The Secure Developer, a podcast about security for developers. Covering security tools and practices you can and should adopt into your development workflow.

The Secure Developer is brought to you by Heavybits, a program dedicated to helping startups take their developer products to market. For more information, visit heavybits.com.

If you're interested in being a guest on this show or if you would like to suggest a topic for us to discuss, find us on Twitter @thesecuredev.

[INTERVIEW]

[00:01:13] Guy Podjarny: Today, I'm proud to have here Gergely Nemeth from RisingStack to talk about how they work in security, both in the world of trace. Before I kind of give too much detail about you, Gergely, do you want to introduce yourself?

[00:01:27] Gergely Nemeth: Hi, everyone. Thanks for the intro. My name is Gergely. I work at a company called RisingStack where we originally worked on customer projects. We were a professional services company mostly working with Node.js-related projects. And we introduce Node.js to quite a few big and small companies as well.

In the past half a year, we started to build one of our products called Trace, which we will mention later. But that's me in a nutshell.

[00:01:55] Guy Podjarny: Cool. Yeah. Actually, before we dig into Trace, which is super interesting by itself, maybe kind of pausing a little a little bit, we've done a lot of work still in the customer side and in the sort of online education with various size customers. Working on Node.js projects. Can you talk a little bit – when you talk about security, how did that manifest in those engagements? Was it often an explicit requirement or a high emphasis? Did you get pulled in to respond to a security problem? How did that manifest in your work?

[00:02:25] Gergely Nemeth: Sadly, it usually wasn't a thing that was required by our customers. It was mostly a thing that we paid a lot of attention to. Also, to answer your other question, it was only maybe once when we were pulled into some security-related issue that we had to fix as soon as possible. But it's really not – it wasn't really the generic kind of contract that we had.

Most of our users wasn't really into security. It wasn't really a huge priority for them. Maybe they thought that it's something that comes naturally. But they didn't say that this is something we have to address.

[00:03:05] Guy Podjarny: Did you find you have to educate them about it to basically tell them, "Listen, this is going to take some additional amount of work. Or a part of delivering this functionality has to include these security controls?"

[00:03:18] Gergely Nemeth: Yeah, definitely. We had to tell them a lot of times that it's better to fix that before actually you heard about it in Twitter, for example. It's really something we had to tell them a lot. And I think we successfully managed to do so. After we started to work with clients, a lot of security issues were fixed in those applications, which I'm really happy about.

[00:03:42] Guy Podjarny: I guess after you did some work – you work with some customer. You educate them. Frankly, they bring you in because of your expertise. Hopefully, they're in an open mindset to take your advice. And then you went on and you build some security components and you explain those to them. You tell them what is it that you built. Did you find that was well responded to? Were people happy after the fact? Or was it just pretty much disregarded as a mandatory kind of debt but not really something they were excited to have?

[00:04:13] Gergely Nemeth: Mm-hmm. Mostly, they were very happy about it. They didn't even know that they have to address these things. They took it as an educational approach where they can get better. It was a really good experience for us.

[00:04:27] Guy Podjarny: Yeah. And I think you know I often feel that everybody wants to be secure. It's a good thing. People want to do it. It's just about balancing it out with the rest of the activities that you have as well as just sort of sheer lack of awareness. What is it that you need to do to be secure?

[00:04:45] Gergely Nemeth: Absolutely. And I think it's something – even in our customer work, what we did a lot is not just actively paying attention to security when developing new things. But, also, we proactively seek already existing vulnerabilities that these applications had. And they were very received as well. They really appreciated when we found issues that they didn't think of.

[00:05:10] Guy Podjarny: Actually, can I take you even a step further back? How did your security kind of knowledge start coming into being? Yourself and the team? How did you build that security understanding?

[00:05:22] Gergely Nemeth: It was something I was really curious about for a long time. And it was I think 3 years ago when I really started to read a lot about it. Started to experiment with it. And I just really fell in love with security. And I enjoyed doing it ever since.

[00:05:42] Guy Podjarny: Which reading sources did you find most useful when you started building up your understanding of it?

[00:05:49] Gergely Nemeth: Mostly, I started to do it by reading articles on the website of the Open Web Application Security Project.

[00:05:58] Guy Podjarny: The OWASO Project.

[00:05:59] Gergely Nemeth: Exactly. Yeah. They have a lot of really good readings for both beginners and advanced readers as well. I think that's something that everyone can start learning from. Also, I read a couple of books. I follow security-related persons on Twitter. For example, [Toro Hack 00:06:17]. I think that's it. And after that, a lot of research, whitepapers.

[00:06:24] Guy Podjarny: I guess, for those who don't know, OWASP, the Open Web Application Security Project is this pretty long long-standing initiative. I think it's maybe about 15-years-old. And it has a lot of amazing information. The website is kind of messy. It's not necessarily the ease to just navigate. But it has a lot of great output.

I guess the OWASP top 10 was one of those I've sort of seen two or three incarnations of it right now. I think it's on its third. I'm not sure if they actually have been updated the second.

[00:06:52] Gergely Nemeth: It's quite outdated now. I think the latest one is from 2013. Actually, if I'm correct, they release it every 3 years. One should come out pretty soon.

[00:07:04] Guy Podjarny: Yeah. Hopefully, it does. And I think, originally, I remember the conversations. I was working on WebAppSec back there. Sort of app shield, app scan. Sort of web app firewall. First ones. And a lot of the conversations with the OWASP top 10 was that it was nowhere near sufficient. But that's okay. Let's just start with 10 basic things and have those handled. Sort of see that people generally understand those and do those well. And then we'll move on to the next 10 sophisticated things. And that didn't quite happen. Really kind of those 10 core things just multiplied despite increased awareness to a decent amount. Thanks to that OWASP project.

[00:07:42] Gergely Nemeth: Yeah. But, also, even those 10 really basic things are not addressed in most web applications. I think really a lot of companies should start with addressing at least those top 10 security issues and after that, we can move ahead.
[00:08:01] Guy Podjarny: Yeah. Can advance. I guess on that note, back maybe to your kind of work on customer work, which types of problems did you encounter most often or did you choose to focus on?

[00:08:10] Gergely Nemeth: Most often, what we saw is people still has a lot of cross-site scripting vulnerability attacks in their applications. Actually, there are statistics that around 50% of the web applications out there has cross-site scripting attack vulnerabilities.

Actually, it's quite in sync in what we saw. Most of the customer work, what we have done weren't greenfield projects had some kind of cross-site scripting problems. And, also, the other thing we saw, if we were talking about Node.js, were DDoS attacks. Mostly denial of service attacks using RegEx. That's something that we saw.
Yeah. These two were the two main issues that we saw.

[00:09:01] Guy Podjarny: Cool. Yeah. And I think, again, just a sort of level set here, cross-site scripting. You're probably familiar with it. If not, that's a different problem. But, fundamentally, it implies an attacker can trick a user into running a militia script within the context of a certain domain. And then that violates the same origin policy. It violates the protection the browsers give us to protect session cookies and other components from the evil attacker. From the evil player.

But once you have one flaw, one of the problems with many of these security vulnerabilities and cross-site scripting specifically, is that once you have a mistake or a flaw in one page on the site, you've compromised the entire domain. And if an attacker can get a victim to click that link, they can expose session cookies and that private information. Kind of break that model.

I guess you mentioned the regular expression, denial of service. That one is one that I find especially interesting just because it's a little bit less well-known. Cross-site scripting, SQL injection. Even the general concept of denial of service of taking your web server down is relatively – at least high-level, many people know those words and the very simple meaning of what they mean. But regular expressions for denial of service, those are not always well-known. Do you want to explain a little bit what those are?

[00:10:17] Gergely Nemeth: Okay. Yeah. Sure. Regular expression denial of service attacks in Node.js can happen because the V8 engine that interprets your code represents this regular expressions in trees. And the traversal of these trees can reach really extreme situations in some cases.

As Node.js is single-threaded, it can happen that if the Node.js processor itself takes a lot of time matching the RegExps that you have, then user requests, HTTP requests, cannot be served in that time. This is why it's an attack that can cause service downtimes. Because, actually, you don't need a lot of resource to do it. You just have to have a crafted input for an API, for example. And if this request takes couple of seconds to fulfill, then, in the meantime, other requests cannot be served.

[00:11:09] Guy Podjarny: Right. Yeah. And it's interesting how, in the world of Node.js, the single – not single-threaded. But rather the event-based scaling, that it really is the core of its strength. It's what makes it so scalable and so lightweight. And the fact that you don't need even in a massive production system. You really only need a small double-digit in a sort of big system number of threads. But then that makes it a little bit of a single point of failure. If you get those threads churning sort of for a long time on the processing of anything, of any sort of algorithm, then you can tackle those. And, yeah, ReDoS are just regular expressions. You don't think of them as algorithms. But they're algorithms. And because of that, as you say, they can take this nonlinear amount of time compared to the size.

My co-founder at Synk, Danny, he did this capture the flag competition in the Chaos Computer Club. Capture the flag competition is really interesting. I might even do an entire episode on those. But the CTF had won a specific Node.js challenge that it involved a timing attack against Node.js, which was really interesting. It was, again, using the fact that it's a single thread. And the fact that there are unsafe comparisons.

When you compare two strings, the natural comparison in most platforms, in specific, in Node.js, you do equal-equal and it would compare them character by character. If the first characters defer, it would fail faster. And it seems minuscule. It seems tiny-tiny. But when you run them enough times, you can actually deduce those. And then you can effectively guess a secret one character at the time and just sort of grow those.

Again, it was easier to do in node because timing attacks are – any platform is susceptible to them in one form or another. But in the case of node, you could actually do them out of band. You can send a request in one spot and check how long it took in another spot because it was a single thread. And you can kind of rely on that.

[00:13:03] Gergely Nemeth: Performance matters.

[00:13:04] Guy Podjarny: Yeah. It's interesting. ReDoS, or regular expressional denial-of-service are interesting ones.

This was super interesting. Let's maybe kind of move off the customer work a little bit. And you mentioned that you've sort of switched to building Trace. Do you want to tell us a little bit about Trace and maybe how you handle security there?

[00:13:20] Gergely Nemeth: Sure. Trace is a Node.js and microservice monitoring solution. What makes it special is that it was built with microservices in mind in the first place. Really treat them as first-class citizens. What makes microservice infrastructures a lot harder and more complex is the additional complexity introduced by the network itself. Because in monolithic applications, you really don't have to think about that because there is little or no communication across boundaries, application boundaries. Via microservices, it can happen that you have tens of services communicating with each other and the network matters a lot.

Trace really helps debugging and monitoring these distributed systems. For now, Trace supports Node.js only. We are working on adding additional languages. But for now, it's Node.js.

[00:14:14] Guy Podjarny: Yeah. Definitely interesting. And microservices are definitely an amazing evolution. But you can't just think that you can monitor them the exact same way that you monitor servers and have 100 times the number of logical servers without having a substantially higher amount of effort to monitor them. If you really want to flatten or control that, then you probably want to rethink a little bit how do you monitor those services.

I guess when you do a monitoring system, clearly, there's a lot of sensitive systems and sensitive data that might be flowing your way. How do you think of security in Trace? Are there some specific highlights you can share?

[00:14:53] Gergely Nemeth: Yeah. Sure. First of all, we have to make a difference between our side of the things and the customer side. On the customer side, we have our agent, the Node.js agent, which is basically just a simple require away from actually start using it. And we have our own hostage service. Trace is actually a software-as-a-service product that you can start using. And we have to secure both sides.

On the customer side, it's really important to make Trace as opt-in as possible. By default, Trace won't send a lot of information about the traffic you have. For example, we don't send HTTP headers by default. We don't send query strings by default. For example, if you're using signatures or just access keys, then these things won't be collected. And the same applies for customer IDs if you have them. It's really important to know what kind of user information you can handle and what kind of user information you shouldn't.

[00:15:53] Guy Podjarny: Yeah. And this is, I guess, kind of where privacy overlaps with security as well. In part, you might expose secrets. And in part, you might just expose private data, which, for that specific individual whose data just leaked, really not that much different than a security flaw.

[00:16:07] Gergely Nemeth: Yeah. Exactly. That's on the collector side. On our infrastructure side, we try to do basically everything to protect our users' data, which means the following. We are running in Google Cloud, on Kubernetes. And all of our servers are running in a virtual private cloud. Our services are not exposed to the public network.

Also, inside the public network, we still use request signing. Even it's in a private network, we still sign every request with private keys, which are shared only between the instances that have to communicate with each other.

[00:16:46] Guy Podjarny: This is using client-side certificates in HTTPs? Or how do you enforce the signing?

[00:16:52] Gergely Nemeth: For now, we are using Joyent's signature library. It's very similar to the [inaudible 00:16:58]. We're using that. All our databases are in the VPC as well. And what we're doing is that the single entry point to this VPC is through an NGINX proxy. And only the application itself and our collector API is exposed. That's only the two interface that anyone from the public network can communicate with.

[00:17:22] Guy Podjarny: Mm-hmm. Yeah. And I guess the same challenges you're trying to address with Trace in terms of like microservices communication also manifest in terms of security flaws. It's the same notion that says you now have to care about the security of the communication between the different systems. Not just monitor their performance and the likes.

[00:17:38] Gergely Nemeth: Absolutely. Yeah. And the funny thing is that Trace is monitored by Trace.

[00:17:42] Guy Podjarny: Yeah. Always nice to do some dogfooding. Yeah. That's always fun. Building a product of your own, you're building those components and you're doing security. Do you find yourself struggling sometimes with deciding whether to add a security component versus wanting to just get something out?

[00:18:00] Gergely Nemeth: Not really. In these cases, we almost speak security. We really don't want to play with the users' data. It's really important to us that keep their information private and don't share it with anyone.

[00:18:16] Guy Podjarny: Yeah. Fundamentally, if you ask yourself what is it that might ruin your company tomorrow? Generally, not investing in a security component fits the bill.

[00:18:27] Gergely Nemeth: Exactly.

[00:18:28] Guy Podjarny: I guess if we just sort of step out a little bit, this is super interesting around kind of the activity you've had with the customers. You've also done a decent amount in educating others around security and just sort of the node ecosystem specifically. And I really like one of the pieces that you wrote at RisingStack about Node.js security checklist. Sort of I liked it because it was just sort of a nice list of these are the things you need to do. We all like sometimes to be prescribed and overthink it.
I guess what made you maybe write to that in the first place? And then are there some highlights of that that you would want to repeat here or that you think can be shared?

[00:19:02] Gergely Nemeth: Mm-hmm. Yeah. Actually, the trigger point for me for this article was one of our customers that – back in the time when we did professional services, at that given customer, we had a lot of security-related problems. And, actually, this article started as a checklist for them.

What I did is that I added some extra things that I thought was really necessary for developers that want to put a Node.js service, or not just Node.js for that, is case because some items in this list are quite generic. It can be applied for other languages as well. I just extended the original list and made it available on our blog. So, anyone who's interested can learn a lot about how to secure your Node.js services.

[00:19:47] Guy Podjarny: Yeah. And I think it's a really good list. It's always the problem that security is a bottomless pit. You can do – well, I guess, it's true for many aspects, right? For quality. For performance. Many of these things, you can put a ton of effort into them. But oftentimes, it's just hard to see how do you make some improvement, right? How do you take one thing and improve that? And I find the checklist just to be a good way to do that. Right? Here's a list of items. Are you doing all of them? No? Maybe add one of them to your next sprint. Maybe add a couple of others to the sprint after that. Just sort of make it practical, embedded into your process one step at a time.

[00:20:21] Gergely Nemeth: Yeah. But, also, what I think it's quite important here is that this list can be applied for a lot of applications. And maybe at the same time, for a lot of applications, it doesn't make sense. You should always think about your use case. Do I even have to care about this aspect? Because it can happen that it’s simply not worth the effort. Because maybe you are protecting something that for – I don't know, $10. It doesn't make any sense to put hundreds of engineering hours into protecting something that's worth $10 only. As always, you have to think about tradeoffs here.

[00:21:01] Guy Podjarny: Yeah. It's all about risk reduction. And so, the question is a little bit what happens if the risk materializes? I guess I used a bit of an extreme example before talking about something that would ruin your company tomorrow. But, yeah, you have to balance this out. You can't just sort of live in fear. You need to protect what's worth protecting.

I guess, of those, you deal a lot. You've got the checklist. I know you've given a bunch of talks around security controls and things you could do. Do you have a favorite tool or a technique that you find you often tell people about that might be a little surprising?

[00:21:32] Gergely Nemeth: Yeah. I can mention a couple. One of the concepts that I really like is called Tetris. This is a methodology which helps you to describe the security of a system. Meaning that you take an attacker's goal. For example, if you're are talking about a safe maybe, then the goal is to open it up. And in this case, this will be the root node of an attack tree. And you create subgoals. And the subgoals will be the sub-trees of this tree, which basically describe what an attacker has to do to achieve its goal.

In this case, for example, if you're talking about a safe, there can be a couple of paths how an attacker can achieve this. It's a really good technique. And, also, it's not new. It's around 15-years-old or something. You can really find a lot of materials on it if you search for it. I highly recommend to check it out.

[00:22:28] Guy Podjarny: Attack trees are sort of like thinking like an attacker. Figuring out how would you – if you were in whatever the bad guys's shoes, how would you approach it? How would you tackle it? And then try to figure out how would you now that you know that. Or at least you theorize how do you block those paths.

[00:22:46] Gergely Nemeth: Yeah. If you are thinking about the attack trees, you really should think about – as you said, about how an attacker could think. Or, also, you have to think about who are they. An attack tree won't be the same for a student who just hacks in his free time. Or someone who is sponsored by some rich guys, for example.

[00:23:06] Guy Podjarny: Yeah. Or even a government. I think when you –

[00:23:09] Gergely Nemeth: Yeah. I want to say –

[00:23:10] Guy Podjarny: Yeah. When you talk about attack trees, clearly, attack trees is a logical technique. You can use kind of paper and pencil to draw them. But, still, within that, are there some specific tools or methodologies that you use? Or do you straight up just write them down?

[00:23:29] Gergely Nemeth: Mostly, what we do is that there is an extended list at the Open Web Securities website, which has a couple of hundred items. If you want to do complete penetration testing, that's something that you can check. Also, there are a couple of automated tools that help you a lot. If you're speaking about Node.js and you want to check – for example, if you have modules with known issues, then there are already a couple of tools that you can use. For example, the Node Security Project or Snyk, which are both really amazing tools.

[00:24:03] Guy Podjarny: Well, I guess we've got the checklist that sort of has a whole bunch of other tools in there. People should just check it out. I think if you just look for – if you just go to the RisingStack blog, then you can easily find it. Or you can even look up Node.js security checklist.

[00:24:17] Gergely Nemeth: Yeah. Or just Node.js security. That would be the first reason.

[00:24:21] Guy Podjarny: Yeah. It would pop up pretty quickly. Just sort of shows how interested people are in actually finding that. I guess maybe before we part, maybe ask you one more thing, which is if you had to choose – or you're talking to a development shop that is looking to just improve one thing about their security posture, what's your recommendation? What's the one thing you would tell them make sure you do this first?

[00:24:44] Gergely Nemeth: It's a tricky question. I know my answer will be kind of a cliche. But, actually, an attacker has to be right only once. And you have to be right all the time. I think there is not this such a thing that – one thing that you can do to secure your application but it's really about thinking how an attacker would think. What's really valuable for your business? And try to defend against those attacks. And try to protect those information and those things that are the most important for you.

[00:25:20] Guy Podjarny: Yeah. I think that makes perfect sense. It's actually something that we do very naturally when it comes to product features when it comes to quality decisions. We don't just sort of spray and pray, right? We try to be a little smart about it. To talk about which features would impact more users, which aspects of quality are most likely to affect bugs. It's just about being able to step back and do the same for security. And you don't have to be some top-tier red team penetration testing. Clearly, you can be that and you would probably do it better. There's definitely opportunity to sort of bring in those types of teams to think about it.

But I find, if you just sort of stop, if you're just a technical person, you think you know your system, you just stop and you try to think how would you attack this. You don't have to have deep security expertise to get to some good conclusions.

[00:26:07] Gergely Nemeth: No. You don't have to. But, also, I think, generally speaking, it's not a good practice to do penetration testing when products are deployed into production. It's really something that should be part of the development flow. If we are talking about Agile or whatever, it doesn't really matter. But it still should be something that all developers care and think about it on a daily basis.

There is a really great article on ThoughtWorks about security and how it should be part of the agile workflow. And I totally agree with that, that if you're talking about Scrum, for example, then most stories should have a security acceptance criteria as well. Not just from a user’s point of view, but what aspects you have to take into account from a security point of view.

[00:26:57] Guy Podjarny: Yeah. Makes sense. Again, kind of bring it in into your own processes.

[00:27:01] Gergely Nemeth: Yeah.

[00:27:02] Guy Podjarny: Well, this has been great. Thanks a lot for coming, Gergely, and good luck with Trace. And kind of keep educating the world on security.

[00:27:10] Gergely Nemeth: Thank you. Thanks for having me. Bye.

[OUTRO]

[00:27:12] Guy Podjarny: That's all we have time for today. If you'd like to come on as a guest on this show or want us to cover a specific topic, find us on Twitter @thesecuredev. To learn more about Heavybit, browse to heavybits.com. You can find this podcast and many other great ones, as well as over 100 videos about building developer tooling companies given by top experts in the field.