How to increase Serverless observability, monitoring and security

Written by:

July 15, 2019

0 mins read

Functions are often short-lived and deployed in large numbers and are invoked more and more frequently as you scale. For these reasons, it's easy to lose track of the flow of events or to pinpoint the root cause for any given error.

On top of that, as serverless adoption grows for an organization, it also becomes more complicated to monitor for insecure flows and malicious attempts by attackers attempting to force a function into an unsafe code path.

To deploy functions and stay secure at the same time, we recommend you:

I shared more insights about Serverless security in a previous post 10 Serverless Security Best Practices that you probably want to bookmark for a later read, but for now let's continue with the following monitoring and logging practices you can apply for your cloud based Serverless projects.

Increase visibility of environments to which functions belong

Functions are practically free, as they only incur a cost when invoked, and as a result, we deploy many of them. However, over time, each function is a security liability that may allow access to our network and data. Furthermore, over time the libraries used in each function may grow stale and vulnerable. Simultaneously, once deployed, a function is hard to delete as the visibility into the ownership and the organizational responsibility for each function may be blurry or might not have been declared.

To avoid losing track of functions, their purpose and corresponding environments, define:

strict rules around the functions you deploy, making it easy to know which functions are long-term production, which are experimental, back office, etc.
clear guidelines for when a function should be removed

AWS has recently introduced the ability to tag Lambda functions so that they are easy to track and group together. Using the Serverless framework, we can tag functions in their yaml files either globally (for all functions in the serverless.yml file) or for individual functions:


1 service:
2  name: hello-world
3
4 provider:
5  name: aws
6  runtime: nodejs8.10
7  stage: ${opt:stage, 'dev'}
8  region: ${opt:region, 'us-east-1'}
9  profile: production
10  tags:
11    foo: bar
12    baz: qux
13  functions:
14    helloWorld:
15      handler: helloworld.get
16      name: helloworld
17      description: My function
18      tags:
19        foo: bar

The above code shows two types of tags available for lambda functions on AWS:

Line 10 shows global tags in the serverless.yml deployment associated with all the functions
Line 18 shows a foo tag that is associated only with the helloWorld function

Monitoring functions for security vulnerabilities

There are many ways to deploy functions, and the number of deployed functions can easily get out of hand. Over time, the code and dependencies in those functions can grow stale and vulnerable as maintenance efforts diminish and focus is shifted towards other projects.

You want to use a solution that integrates with your function-as-a-service (FaaS) provider in order to monitor the deployed functions so these are kept in check. This allows you to address known security vulnerabilities across the software development lifecycle as it relates to functions and their use of open source dependencies.

In the following picture, we see the GitHub project lirantal/bazz-serverless after being scanned. The project includes several high, medium and low vulnerabilities. This is the source code for my serverless project, which deploys several functions. Below the GitHub project repository, we can see all six of the AWS lambda functions that I deployed. These are the actual functions as they are deployed and executed by AWS.

Snyk scanned all these individual functions for known security vulnerabilities and as each of them uses the same dependency tree we can see that they are all deployed with vulnerable libraries.

Interestingly enough, we can see that there are four low severity vulnerabilities found in the project repository, but they don’t appear in the individual functions. This is because the deployed version is different than the version stored in the Git repo, and specifically the deployed function is outdated—not including a library that I had added to the source code.

If I hadn’t tracked my project’s security, I could have gone ahead and deployed more vulnerable libraries into production, thereby increasing the attack surface of the app I’m developing.

Monitoring resource access

Imagine a scenario where an attacker is able to exploit a vulnerability in a Node.js function blocking the event-loop by forcing it to spend a large amount of time in CPU-bound computation. This may not be easily detectable due to the nature of functions as a service paradigm where functions are spawned as necessary to justify new requests. While this seems like a great feature, and it probably is in certain ways, the trade-off is that your cloud bill surges due to the per-use cost applied to functions.

Create a monitoring strategy: have policies, alerts and enforcement in-place in order to identify and block unintended function invocations. Consider the following resources for monitoring:

CPU
Memory
Input and Output consumption and operations
File access, such as read and write
Function running time
Function invocation flows and anomalies from expected invocation sources
Child process executions

Cloud providers often have built-in monitoring for application and function resources to provide this insight. Microsoft offers Azure Monitoring, whereas Amazon offers AWS X-Ray. To use the latter as an example, based on the official AWS documentation we can see a picture of the AWS X-Ray console, providing insights into the data flow through functions and other cloud resources as well as metrics, such as execution time.

Once monitoring is achieved, enforce function-related limits to further prevent financial resource exhaustion.

Logging functions

Logs are logically grouped by a set of functions, thereby providing additional valuable insights, but if not handled properly they can also present additional security issues.

To get maximum value out of logs:

Prefer verbose logging of events, their context, and failure information
Leverage cloud logging facilities
Implement a central logging management system to enable your team to easily debug and find events
Associate logs and special events with thresholds, and alerts, so that beyond an audit trail, logs also serve a purpose of deliver alerts when issues require immediate attention

With this in mind, be careful not to log sensitive information, such as credentials or excessive logging such as environment variable data. Also, ensure compliance with local GDPR such that no personally identifiable information is logged.

Summary

I invite you to read a more in-depth list of 10 Serverless Security Best Practices for more security related concerns when building and deploying Serverless functions.