Fixing Serverless Security Vulnerabilities
Well over 80% of successful exploits today occur due to unpatched servers. With modern approaches such as Serverless & PaaS, the servers and their binaries will be managed by more dedicated and professional teams, which should dramatically reduce the risk of outdated binaries—hurray!
Unfortunately, this transition does nothing to secure open source code packages. These packages are just as vulnerable as their binary counterpart, just as prevalent, and just as easy for attackers to exploit. And contrary to the evolution of binaries usage, open source packages can and are added by any developer on the team, and bring in long chains of indirect dependencies.
This post explains why vulnerable packages are the new unpatched servers, and what we can do to stop it.
Exploiting vulnerable binaries today & tomorrow
Before we talk about the new threat, let’s take a moment to understand the current one—unpatched servers. These are systems that didn’t apply the latest security updates, and are therefore running vulnerable binaries that are easy to find and exploit. The binaries involved are almost exclusively open source, especially in the linux ecosystem.
There are several reasons for the prevalence of exploits, the key ones being:
1. The vulnerabilities are already known. Since the issues are found and reported by researchers and the community, the attackers have full knowledge of the issue without additional effort. All they need to do is create an exploit, which is easy to do when you can replicate the software locally.
2. One vulnerability leads to many victims. OSS is free and easy to consume, and is therefore used everywhere. Once the evil-doers write an exploit to an issue, they can reuse it on many different targets. This in turn creates a marketplace for tools performing such exploits, boosting an ecosystem of low-proficiency attackers.
3. Keeping servers up-to-date is HARD. Old systems are hard to inventory and control, while new cloud systems are spawned easily and with little scrutiny. In that mix, ensuring all systems are updated is a complicated people & technology problem, which organisations consistently fail to do.
In other words, vulnerable binaries are easy to find and exploit, are widely distributed, and its hard for companies to fully protect themselves. This combination hopefully makes it quite clear why exploiting these flaws succeeds so often.
Fortunately, the modern world of infrastructure aims to leave server management to dedicated professionals, who will be far less likely to leave servers unpatched. Between Serverless, PaaS and uni-kernels, unpatched servers will hopefully be a thing of the past. Which will send attackers looking for the next easy path in…
The missing link: open source packages
Open source is not consumed exclusively as binaries. In fact, a growing amount of OSS is consumed in the form of packaged code dependencies, downloaded from repositories such as npm, RubyGems, Maven and more. The use of these packages is growing rapidly, and already the vast majority of code actually deployed in your app is likely not yours but rather open source code.
Open source packages share the same properties we just listed for open source binaries:
1. The vulnerabilities in them are already known, typically logged as GitHub issues.
2. They are prevalent, many downloaded millions of times a month, making exploits for their vulnerabilities highly reusable.
3. Tracking & controlling these packages is HARD. Any developer can add a dependency, and each dependency brings with it a tree of indirect dependencies – along with their security flaws.
This alone makes open source packages just as attractive to attackers as vulnerable binaries, but it doesn’t end there. Applications use far more packages than they use binaries (especially in Node.js); developers are often reluctant to upgrade due to risk of breaking the application; and the depth of indirect dependency chains—which are even harder to control—can get much deeper than operating system chains.
But more importantly, awareness to these vulnerabilities is extremely low, both amongst the authors of the packages and their consumers.
Real world data
This is not a theoretical problem.
In Node.js, the dominant language for Serverless, nearly 80 vulnerabilities were published this year alone, and over 4 in 5 Node.js dev shops use vulnerable packages. In Maven, 59% of reported vulnerabilities do not get fixed, and those that are fixed take a mean time of 390 days to do so. And in Ruby, two thirds of projects are vulnerable, with a mean of nearly 4 vulnerabilities per project and every 8th project having a severe and easily exploitable security flaw.
The vulnerabilities are also just as severe as those found in binaries. Earlier this year, Heartbleed-like remote memory exposure vulnerabilities were disclosed in the
request npm packages, which are downloaded nearly 20 million times a month combined. A few months later, the engine behind socket.io (3M downloads/month) was found to be susceptible to a Man-in-the-Middle attack. And around the same time, a Denial-of-Service vulnerability was disclosed for the
tough-cookie package (11M downloads/month) – which on services like AWS Lambda may also result in a massive bill.
Tackling this problem
First and foremost, we need to acknowledge this as a priority. These vulnerabilities expose our services today, and are not on a trajectory to be improved. We need to discuss this risk more often, and consider it when building new tools.
Second, we need to find and address vulnerable packages in our code today, and then integrate continuous security testing (and fixing) into our dev processes. While nascent, tools like the OWASP dependency checker, Victims DB and Snyk can you help get that done.
Lastly, we need to think about what framework additions are needed to address this problem systematically. Should these issues be discovered as part of the current Serverless frameworks and services? Should they be integrated with package repositories? And should we aim to have a stream of backported fixes seamlessly applied, like those RedHat and Ubuntu maintain for operating systems?
Serverless is a boon for security, and helps put an end to a serious security threat in vulnerable OSS binaries. Let’s not let the vulnerable OSS package threat quickly rise to take its place.