Kubernetes Monitoring Guide

Why you should monitor your Kubernetes deployments

Kubernetes is widely used due to its ability to simplify the process of deploying containerized applications on distributed systems. Running a single container on a single node requires careful configuration and ongoing management. Expanding that to hundreds or thousands of nodes would require extensive human resources and workflow management. Kubernetes distills those systems and tribal knowledge into a platform that automates the know-how required for orchestrating containerized applications.

It’s important to monitor your Kubernetes deployments because Kubernetes can’t manage itself. Changes to applications, like new versions or Kubernetes security vulnerabilities, can create anomalies that potentially lead to problems. Using container security best practices helps prevent issues to begin with, but by monitoring application requests and anomalies, you can detect any issues early and mitigate them before they lead to service failures or security breaches.

Kubernetes monitoring challenges

Kubernetes monitoring brings many of the same challenges as container monitoring. Kubernetes itself comes with Kubernetes Dashboard, which provides basic, real-time monitoring of workloads and clusters. The distributed and dynamic nature of Kubernetes means robust monitoring requires special tools and processes.

It’s necessary to monitor both workloads and the cluster itself. There are a variety of things that can cause your Kubernetes services and resources to stop reporting at any time, from misconfiguration of core components to network connectivity issues, which presents a challenge for a traditional, static monitoring approach, emphasizing the need for real time monitoring. Furthermore, the organization of Kubernetes around microservices and namespaces means monitoring around physical nodes or service ports is insufficient.

Beyond production monitoring, Kubernetes requires self-configuration to ensure its security controls are properly set up. The configuration for Kubernetes is typically specified in code, whether Kubernetes YAML, Helm Charts, or templating tools. Properly configuring for workloads, clusters, networks, and infrastructure is crucial for averting issues and limiting the impact if a breach does occur. The process of configuration is another aspect of Kubernetes that needs to be monitored to uncover any issues before they impact production environments.

Important metrics you should track

There are five important areas you should monitor to ensure your Kubernetes clusters and workloads are healthy.

Performance

Monitoring the performance of clusters, nodes, and Pods helps circumvent the challenge of container visibility.

Utilization

Tracking the utilization of CPU and memory on clusters and namespaces helps ensure you’re taking full advantage of available resources.

Application tracing

This requires special care in identifying the specific metrics you need and how you will access them from the application.

Functional tests

This helps ensure applications behave as expected with the rest of the solution stack.

Security

Kubernetes provides audit logs for its API server. Monitoring DNS and other logs allows you to identify suspicious activity.

Five Kubernetes monitoring best practices

When it comes to monitoring Kubernetes, there are a few best practices to follow:

1. Use tags and labels
   Since Kubernetes is not a platform-as-a-service, applications are informally defined using metadata. It’s critical to tag and label applications by name, instance, and other common labels to simplify management.

2. Look at the entire set of containers, not at each one
   Since Kubernetes manages Pods, not containers, it’s important to monitor at the Pod level.

3. Take advantage of service discovery
   Service discovery helps monitor Kubernetes services in spite of their volatile nature.

4. Dashboards are not enough: Don’t forget to apply alerts
   Alerts can be set up based on what defines a healthy Kubernetes environment. Beware of the problem of over-alerting and desensitization — distributed systems contain a lot of potential monitoring endpoints.

5. Inspect the Kubernetes control plane
   The Kubernetes control plane behaves much like an air traffic controller, scheduling workloads, monitoring, and managing the cluster. Inspecting each component of the control plane — API server, controller manager, scheduler, etc — helps ensure the orchestration and scheduling of tasks continues to run efficiently.

The best monitoring tools for Kubernetes

Due to the complexity and scale of Kubernetes workloads and resources, the right tooling is necessary to monitor every aspect of the platform. A variety of monitoring tools are available in three general categories.

Open source: These include Prometheus, which aggregates time-series data from Kubernetes jobs. Prometheus was developed at SoundCloud and is a graduated project of the Cloud Native Computing Foundation. It integrates with Grafana to produce visualizations and AlertManager for alerts.

SaaS: SaaS-based monitoring solutions like Sysdig allow for more comprehensive monitoring without the need for third-party tools for visualizations or alerts.

Enterprise-class: These tools, including Sysdig, offer more sophisticated capabilities allowing collaboration across teams and actionable insights fueled by AI.

These Kubernetes monitoring tools focus on monitoring workloads in production. Snyk’s Kubernetes Configuration Security allows you to scan Kubernetes configurations as you write them, before deploying to production. Kubernetes Configuration Security takes a developer-first approach to allow developers to identify and remediate issues within their normal workflows.

To learn more about issues related to Kubernetes and containers, read these related posts:

What is container security? | Container Image Security | Snyk

Everything You Need To Know About Container Scanning | Snyk

Kubernetes monitoring FAQ