So, you think your CI/CD environment is secure?

This post, co-written by Weaveworks and Snyk, explains how by using a GitOps continuous integration (CI)/continuous delivery (CD) pipeline combined with good security practices improves the overall security of your development workflow to Kubernetes.

The Typical CI/CD Pipeline

Your CI/CD pipeline might look very similar to the simplified model below. The flow begins from the far left, when the code is on the developer’s machine and is pushed to a code repository, like GitHub. The code is then picked up by a CI tool, which runs tests and then builds an artifact, such as a container image. The image is pushed to the image repository and deployed to an open source orchestrator like Kubernetes or a similar system.

But one thing that people often fail to consider is whether your typical CI/CD push model is secure. Consider the following two questions:

• Does your CI environment have direct access to the container image repository?

• Does your CI environment have direct access to the production cluster?

Let’s look at the pipeline again, but this time, we’ll focus on which stages have access to each other. In the image below, we use RW for Read Write access, and RO for Read Only access. There are probably many more red lines than you expected there! This simple pipeline violates some of the Open Web Application Security Project (OWASP) security principles, including the Principle of Least Privilege as well as the Separation of Duties. For example, the developer has Read Write access to the Code repository and the cluster.

By removing direct developer access to the image repo and to the cluster, the attack surface, privileged access is minimized and duties are separated.

Enter GitOps

GitOps is a way to do Continuous Delivery for Cloud Native applications. It works by using Git as a source of truth for declarative infrastructure and applications. Delivery pipelines automatically roll out changes to your infrastructure when changes are made to Git. But the idea goes further - it also uses tools to look at the actual production state, and then tells you when the source doesn’t match the real world.

The GitOps method addresses an insecure pipeline by running a reconciliation operator from within the cluster itself. It operates on a configuration Git repo, using separate credentials. The operator compares and reconciles the desired state as expressed in the manifest files stored in the Git repo versus the actual state of the cluster.

This means that there’s no credential leakage across the boundaries. The CI system can operate in a different security ‘zone’ rather than on the target cluster. Each pipeline component needs only a single RW credential. Because the cluster credentials never leave the cluster itself, you can now “keep your secrets close”.

Do I need to worry about security anymore?

When you adopt this approach, you reduce the security risk by eliminating the Principle of Least Privilege and the Separation of Duties issues, among other problems. This does not solve all your security concerns, of course. In fact, it places an even more important emphasis on the need for good security in your code repository.

Enter GitSecOps

Ok fine—we have enough buzzwords in our industry already, so let’s not create another one. That said, most of what James Governor, aka @monkchips, says tends to become reality sooner or later. Thanks for the idea, James, hope you like the blog post!

Here are some tips on how to better secure your code repository.

Add security testing to your PRs

All of the major code repositories have powerful event-driven hook frameworks that allow you to send HTTP POST requests to a service of your choice when events are fired. There are a vast number of events you can choose to act upon, but one of the most useful for testing your incremental code changes is the pull_request event.

There are many static code analysis tools that support hooks so that when a PR is created, an HTTP POST is fired to prompt you to test your latest updates. This is also a great time to ensure that code and config changes being made are aligned with your security expectations.

Statically analyze your repo with Snyk

Snyk statically analyzes your repo to find vulnerable dependencies you may be using and then helps you fix them. Not only can you test your repos through Snyk’s UI to find issues, but you can also use it to keep users from adding new vulnerable libraries by testing pull requests, and then failing a test if a new vulnerability was introduced.

Beyond the convenient integration into GitHub, GitLab and Bitbucket, pull requests are better than “breaking the build”. In fact, they don’t even have to block a merge since they are informational by default. The tests only focus on your changes, rather than the overall outcome, and will only fail if you introduced a vulnerable library, rather than if it had existed prior to your change.

Never store credentials as code/config

A quick search on GitHub shows how widespread the problem of stored passwords in repositories really is. The 350,000 commits returned from this simple search does not cover those that were not so obvious with their commit messages, nor those who tried to cover their tracks by removing their history.

You can also use tools like git-secrets to actively break builds when sensitive information is found in code or a config file. Having team-wide rules that prevent this from happening is a great way to police bad actions in the existing developer workflow.

There are many ways to avoid putting credentials into your repository in the first place. You should try to implement as many as you can, but even if you do, there is always the chance that some sensitive information may sneak in. You should also consider regular auditing of your repos, and make use of tools like GitRob or truffleHog, both of which scan through your codebase, searching for sensitive information via pattern matching.

If you find that you have stored sensitive data in your code repository, you will need to do a number of things to recover:

1. You'll need to invalidate the tokens and passwords that were once public.

2. Once a secret is public on the internet, you should assume it's in the hands of attackers and react accordingly.

3. Remove any history of your secrets to ensure there is no trace of them either in code or audit trail.

Tightly control access

Mandate the following basic practices for your contributors:

• Require 2-factor-authentication on every contributor’s GitHub account.

• Never let users share accounts/passwords.

• Properly secure any laptops/devices with access to your source code.

• Accounts are often personal ones, and do not naturally disappear when users leave the company. Make sure you diligently revoke access from users who are no longer working with you.

• Repository administrators should manage team access to data. Only give contributors access to the data they need to do their work.

Add a SECURITY.md file

It’s natural for most project owners and maintainers to add a README.md to their repository. In fact, these days it’s quite frowned upon if the README is missing.Likewise, it’s becoming increasingly common to add a SECURITY.md file that highlights security-related information for your project. Not only does such a file give users the important security information they need, but it also forces the maintainers to think about how they should deal with security disclosures, updates, and general security practices.Good examples of SECURITY.md files can be seen in the Apache Storm and TensorFlow repositories.

Summary

Your CI server can comfortably orchestrate the development of merging to mainline, building and testing. However, additional security considerations arise when CI servers start performing continuous delivery operations. GitOps relies on Kubernetes or the cluster to internally manage deployments based on those trunk updates. This is also called the “pull” model for CD.

Git is the single source of truth for the code in addition to the config and associated stack. This makes it a much more prominent security focus. There are many ways in which following general hygiene practices on your code repository can help, such as automatically using testing tools like Snyk on every pull request, creating a SECURITY.MD file that includes practiced security procedures, and using vaults for secrets rather than storing them in code.

To add security into your Git workflows or to your CI/CD pipelines you should test Snyk, for free! You can also book a private demo with our team to see Snyk in action.