Command line tools for containers—using Snyk with Buildah, Podman, and Skopeo

As the container ecosystem has matured, the one thing we’re not short on is options—both in terms of the software we use, and how we plug it all together.

One of these options would be the combination of Buildah, Podman, and Skopeo—three open source command line tools with their origins in the RedHat ecosystem. As its name suggests, Buildah provides a wide range of functionality around building OCI compliant containers and images, Podman provides for the management and execution of OCI containers, while Skopeo is a tool for working with container registries.

In this blog post, we’ll look at how the use of open standards allows Snyk’s container scanning to work with all of these technologies.

First of all, let’s look at Skopeo. This is a command line tool for moving container images around between repositories, and can also save images locally. Skopeo supports outputting images either as Docker archives or OCI archives, both of which Snyk supports when scanning from the filesystem. Let’s use it to download an image from Quay.io and scan that locally from our filesystem :

$ skopeo copy docker://quay.io/tutum/hello-world oci-archive:hello-world.tar
$ snyk container test oci-archive:hello-world.tar 

Testing oci-archive:hello-world.tar…

{-----OUTPUT SNIPPED-----}

✗ High severity vulnerability found in openssl
  Description: Out-of-Bounds
  Info: https://snyk.io/vuln/SNYK-UBUNTU1210-OPENSSL-374571
  Introduced through: openssl@1.0.1c-3ubuntu2.6, ca-certificates@20120623, ssl-cert@1.0.32, meta-common-packages@meta
  From: openssl@1.0.1c-3ubuntu2.6
  From: ca-certificates@20120623 > openssl@1.0.1c-3ubuntu2.6
  From: ssl-cert@1.0.32 > openssl@1.0.1c-3ubuntu2.6
  and 1 more...
  Fixed in: 1.0.1c-3ubuntu2.7

Organization:      matt-jarvis-snyk
Package manager:   deb
Project name:      docker-image|hello-world.tar
Docker image:      oci-archive:hello-world.tar
Licenses:          enabled

Tested 237 dependencies for known issues, found 25 issues.

Ubuntu 12.10 is no longer supported by the Ubuntu maintainers. Vulnerability detection may be affected by a lack of security updates.

As we can see, this particular image is using an End Of Life version of Ubuntu, and so contains multiple vulnerabilities that will not be fixed. So, using Skopeo, we can download images from any standards-compliant container registry in formats which Snyk supports, and scan them locally.

Buildah

Next, let’s take a look at Buildah. Buildah is a command line tool which builds images, it doesn’t run them, so it’s intended to be used in conjunction with Podman. It’s also designed to operate in rootless mode, utilizing user namespaces in the Linux kernel to segregate a root-like shell which only has user-level privileges. This enables administrators to empower developers to build and maintain container images, whilst at the same time following principles of least privilege within the build environment.

Although Buildah can work with Dockerfiles in a similar way to Docker itself, it also allows you to develop containers and images using other workflows, for example, Bash or Python. This can have advantages in terms of flexibility, and can also make debugging of container builds easier—Buildah allows for things like mounting of the intermediate container during the build process, so scripts can do things like check for correct execution before continuing, build software externally or even take user input. This provides an alternative to the sometimes convoluted multi-stage build processes which have evolved as best practice when using Docker. 

Let’s take a look at an example of using Buildah in practice.

One of the first things to note about Buildah is that we can just use Dockerfiles to build our containers, if that is the workflow that works for you. Taking this Dockerfile as an example :

$ cat Dockerfile 
FROM centos:8
LABEL maintainer Matt Jarvis <matt@mattjarvis.org.uk>

RUN dnf install -y git gcc findutils make help2man texinfo gperf gettext-devel autoconf automake && dnf clean all

RUN git clone https://git.savannah.gnu.org/git/hello.git /opt/hello

WORKDIR /opt/hello

RUN ./bootstrap --skip-po && ./configure && make && make install

RUN ./hello
ENTRYPOINT "/usr/local/bin/hello"

Here we’re using the CentOS 8 image as our base, cloning the GNU Hello source code, installing some prerequisites, and then building and installing the binary.

We can build this automatically using Buildah with the bud ( build-using-dockerfile) switch :

$ buildah bud

This is clearly not good practice, since we include both the source code and the build prerequisites in our final image, and this image breaks most of the best practices for secure image development.  Using Docker, the best practice here would be a multi-stage build. We could use a multi-stage Dockerfile using Buildah, but more interestingly we can also use native Buildah commands, and write scripts to do so. To mirror exactly what we did with the Dockerfile above, we would run the following script :

$ cat single-stage.sh 
#!/usr/bin/env bash

set -o errexit

# Create a container
container=$(buildah from centos:8)

# Labels are part of the "buildah config" command
buildah config --label maintainer="Matt Jarvis <matt@mattjarvis.org.uk>" $container

# Install pre-reqs
buildah run $container dnf install -y git gcc findutils make help2man texinfo gperf gettext-devel autoconf automake
buildah run $container dnf clean all

# Clone the git repository
buildah run $container git clone https://git.savannah.gnu.org/git/hello.git /opt/hello

# Workingdir is also a "buildah config" command
buildah config --workingdir /opt/hello $container

buildah run $container ./bootstrap --skip-po
buildah run $container ./configure
buildah run $container make
buildah run $container make install

# Entrypoint is also a “buildah config” command
buildah config --entrypoint /usr/local/bin/hello $container

# Finally save the running container to an image
buildah commit --format docker $container hello:latest

However, Buildah also allows us to mount the intermediate container during the build process, so we can actually do our source code build on the host, and simply copy the resulting binary into our container before we commit to an image. In this way, we can take advantage of the host resources during our builds, and do other things like use the host package manager to install packages into our container. We also don’t need root access for that, as Buildah supports using user namespaces to allow you to mount without privileges, by using the unshare command :

$ buildah unshare
$ cat host-mount.sh 
#!/usr/bin/env bash

set -o errexit

# Create a container
container=$(buildah from centos:8)
mountpoint=$(buildah mount $container)

buildah config --label maintainer="Matt Jarvis <matt@mattjarvis.org.uk>" $container

# Clone the git repository to the host
git clone https://git.savannah.gnu.org/git/hello.git hello

pushd hello
./bootstrap --skip-po
./configure
make
# Install to the container
make install DESTDIR=${mountpoint}
popd

chroot $mountpoint bash -c "/usr/local/bin/hello"

buildah config --entrypoint "/usr/local/bin/hello" $container
buildah commit --format docker $container hello
buildah unmount $container

As you can see in the above example, after running the unshare command we now have a root shell, although this is in a user namespace without full root privileges. This allows us to mount the container filesystem during the build process. Since this is a bash script, we can also potentially leverage all the functionality that a programming language gives us—we could use conditionals, loops, or even request user input during the process.

Podman

Once we run this script and build our image, we can also scan it using Snyk, and this is where Podman comes in. Podman is a daemon-less container engine for developing, managing, and running OCI containers. Since Buildah and Podman are based around open standards, Snyk has always been able to scan images created or pulled by Podman, by using Podman to save the image to disk and scanning it from the filesystem. Podman supports saving images as both a standard Docker archive or in OCI archive format, both of which Snyk supports.

[root@localhost buildah_test]# podman images
REPOSITORY                         TAG     IMAGE ID      CREATED         SIZE
localhost/hello                    latest  975f60fc1f19  15 minutes ago  223 MB
[root@localhost buildah_test]# podman save 975f60fc1f19 -o hello.tar
[root@localhost buildah_test]# snyk container test docker-archive:hello.tar

Testing docker-archive:hello.tar...

{-----OUTPUT SNIPPED-----}

✗ High severity vulnerability found in librepo
  Description: RHSA-2020:3658
  Info: https://snyk.io/vuln/SNYK-CENTOS8-LIBREPO-610057
  Introduced through: librepo@1.11.0-2.el8
  From: librepo@1.11.0-2.el8
  Fixed in: 0:1.11.0-3.el8_2

Organization:      matt-jarvis-snyk
Package manager:   rpm
Project name:      docker-image|hello.tar
Docker image:      docker-archive:hello.tar
Platform:          linux/amd64
Licenses:          enabled

Tested 172 dependencies for known issues, found 28 issues.

Pro tip: use `--file` option to get base image remediation advice.
Example: $ snyk test --docker docker-archive:hello.tar --file=path/to/Dockerfile

To remove this message in the future, please run `snyk config set disableSuggestions=true`

Now, since our base image is a public image which exists in Dockerhub, if we want to scan this image using Snyk CLI, we could also just do :

[root@localhost buildah_test]# snyk container test centos:8

In this case, Snyk will interact directly with Dockerhub and just scan the upstream image.

However, if the image only exists locally in Podman, or if it’s in a private repository in Dockerhub which requires a login, neither of those cases will work. As an example, let’s try to scan our built image directly from Podman:

[matt@localhost buildah_test]# podman images
REPOSITORY                         TAG     IMAGE ID      CREATED         SIZE
localhost/hello                    latest  975f60fc1f19  20 minutes ago  223 MB

[matt@localhost buildah_test]# snyk container test localhost/hello
connect ECONNREFUSED 127.0.0.1:443

For these use cases, Snyk uses the local Docker client, either through the Registry API or by using the Docker socket. Since there is no Docker binary on the system, here Snyk has attempted to use the Registry API and failed to connect. By default, Snyk doesn’t know those images exist locally in Podman, or how to use Podman to interface with upstream.

Luckily, Podman provides us with some easy compatibility features through the podman-docker package. This gives us a fake Docker binary, which is basically just a shell wrapper to the Podman binary, and adds a symlink between the Podman API socket and the default location of the Docker socket file.  Out of the box, this package creates a symlink to /var/run/podman/podman.sock, which is the default when running the Podman API as root.

[matt@localhost buildah_test]$ sudo dnf install podman-docker

Now let’s try and scan our local image again :

[matt@localhost buildah_test]$ snyk container test localhost/hello
connect EACCES /var/run/docker.sock

This still doesn’t work, but this time we’ve got a different error. Snyk has detected a binary called docker, and so is now trying to connect to the default location of the privileged Docker socket. Since we are not yet running the Podman API, it can’t connect and so fails again.

Now as we saw earlier, the podman-docker package creates a symlink from /var/run/podman/podman.sock to /var/run/docker.sock, assuming that the Podman API is going to be run privileged.

[matt@localhost buildah_test]$ ls -l /var/run/docker.sock 
lrwxrwxrwx. 1 root root 23 Nov 25 07:51 /var/run/docker.sock -> /run/podman/podman.sock

One of the main issues with the Docker socket is that by default it runs privileged, which is considered a security risk in certain environments. For most local development environments we don’t need to run the Podman API that way, and can run an unprivileged socket somewhere else. We also have a method for communicating that location to Snyk by using the DOCKER_HOST environment variable, which supports both tcp:// and unix:// URL formats.

In a different shell, let’s run the Podman API with an unprivileged socket:

[matt@localhost ~]$ podman system service --time=0 unix://home/matt/podman.sock

This will run foregrounded, so you will have to ctrl-C the process when finishing the test. Now we need to set the DOCKER_HOST environment variable:

[matt@localhost buildah_test]$ export DOCKER_HOST=unix:///home/matt/podman.sock

And finally, let’s try and scan our local image again with Snyk :

[matt@localhost buildah_test]$ snyk container test localhost/hello

Testing localhost/hello...

{ ----SNIPPED OUTPUT----}

✗ High severity vulnerability found in librepo
  Description: RHSA-2020:3658
  Info: https://snyk.io/vuln/SNYK-CENTOS8-LIBREPO-610057
  Introduced through: librepo@1.11.0-2.el8
  From: librepo@1.11.0-2.el8
  Fixed in: 0:1.11.0-3.el8_2

Organization:      matt-jarvis-snyk
Package manager:   rpm
Project name:      docker-image|localhost/hello
Docker image:      localhost/hello
Platform:          linux/amd64
Licenses:          enabled

Tested 172 dependencies for known issues, found 28 issues.

Pro tip: use `--file` option to get base image remediation advice.
Example: $ snyk test --docker localhost/hello --file=path/to/Dockerfile

To remove this message in the future, please run `snyk config set disableSuggestions=true`

Success! Snyk is now working correctly and talking to Podman using the unprivileged socket.

To configure DOCKER_HOST permanently in your shell, you can add the export command to your .bashrc file which will persist the setting.

Finally, we’d really like to have this whole setup automatically available to us whenever we log in. To do this we can take advantage of the socket activation facilities in systemd to automatically start our socket when we log in and try to connect to it.

First we need to configure the socket :

[matt@localhost ~]$ cat .config/systemd/user/podman.socket
[Unit]
Description=Podman API Socket
Documentation=man:podman-api(1)

[Socket]
ListenStream=/home/matt/podman.sock
SocketMode=0660

[Install]
WantedBy=sockets.target

And then tie the socket to a service :

[matt@localhost ~]$ cat .config/systemd/user/podman.service
[Unit]
Description=Podman API Service
Requires=podman.socket
After=podman.socket
Documentation=man:podman-api(1)
StartLimitIntervalSec=0

[Service]
Type=oneshot
Environment=REGISTRIES_CONFIG_PATH=/etc/containers/registries.conf
ExecStart=/usr/bin/podman system service unix:///home/matt/podman.sock
TimeoutStopSec=30
KillMode=process

[Install]
WantedBy=multi-user.target
Also=podman.socket

Finally, we reload the systemd configuration and enable the service!

[matt@localhost ~]$ systemctl --user daemon-reload
[matt@localhost ~]$ systemctl --user enable --now podman.socket

Conclusion

So, there we have it—Snyk CLI image scanning with Podman working in exactly the same way as with Docker, allowing developers easy access to comprehensive security scans of local Docker or OCI images as part of their development workflow, without requiring raised privileges. We also saw how both Skopeo and Buildah can be used with Snyk, through the power of open standards!

Don’t already have a Snyk account? Sign up for free and use Snyk to scan both container images and open source dependencies.