Kubernetes - known as ‘K8s’ - makes many elements of container management much easier. It’s especially great for projects that need to scale suddenly or react to demand.
The structure of software running on Kubernetes can quickly grow, though, and given its command-line focus, observability and monitoring can be challenging.
On this glossary page, we explore:
- How Kubernetes observability and monitoring differ
- What you should monitor and why
- Some best practices for observing and monitoring Kubernetes
- Kubernetes observability and monitoring tools
- Octopus’s place in Kubernetes observability and monitoring
Understanding Kubernetes observability and monitoring
When understanding the status of Kubernetes clusters, you’ll sometimes see people use the terms ‘observability’ and ‘monitoring’ interchangeably. Though similar in some respects, they’re slightly different but linked concepts.
They also mean vastly different things depending on your field of work but, in modern software delivery, you can usually simplify the difference as:
- Observability is the exposing and understanding of information. It’s the ability to understand the state of your system at a single point in time. An engineer may use observability to check if a service responds quickly enough for users, or a project manager may use observability to plan future work or report progress up the corporate ladder.
- Monitoring means actively watching and using that observable information to benefit the software, teams, and organization. It means detailed analysis of software, infrastructure, and team performance; it means using data and alerts for proactive problem-solving and improvement and reacting to changes and problems.
Both things help with Kubernetes. To help monitor Kubernetes clusters, you could use alerts to know when infrastructure is at its limits so you can act when needed. Even though Kubernetes can automatically tear down and replace broken containers, observability and monitoring can help you understand why they broke so you can try to prevent it.
Observing and monitoring Kubernetes can also help you understand and improve software delivery performance to help with DevOps and Continuous Delivery adoption.
Of course, you can’t understand everything based on raw data alone. Even if you can, it’s not suitable for everyone involved in modern software delivery. Luckily, lots of monitoring tools format the data in easy-to-understand visualizations so key information is more obvious and palatable.
What you should monitor and why
Monitoring Kubernetes clusters can benefits software teams in lots of different ways, big and small. Naturally, that means you’ll want to track many things to maximize those benefits.
Infrastructure
Tracking the performance and behavior of infrastructure is vital to understanding how resilient your software is and what you must improve.
Monitoring infrastructure can help you both proactively and reactively. You’ll be able to tell when a cluster might be about to fall over and prevent it, which is a great help if your software promises high availability.
You’ll also be able to discover a root cause far faster if the worst does happen, and have vital information ready to help with disaster recovery.
At the very least, you should aim to monitor:
- CPU use
- Memory use
- Traffic
Application-level
Monitoring Kubernetes-hosted software at the application level helps you understand how customers use your product. These findings can shape the direction you take your software and the continuous improvements you need to boost DevOps performance.
Exactly what you need to monitor will differ for every application and software architecture, but some things to consider are:
- Customer interactions
- Unexpected traffic or slowdown in application speed
- Bugs and errors experienced by customers
Security
If you’re not proactive about protecting your software, potential customers won’t trust you, and you risk data breaches and software failures. Monitoring security-related activity and traffic on Kubernetes clusters can help identify weak points in your defenses, bad actors, and suspect activity.
You could monitor your clusters to look for:
- Suspicious activity in clusters
- Attempts to access sensitive information
Best practices for Kubernetes observability and monitoring
Create meaningful alerts
You should create alerts in your monitoring software so you’re quickly aware of anything needing urgent attention, including outages, security concerns, or application activity.
Most tools let you send alerts through email and collaborative instant-messaging tools like Slack.
Depending on your tool stack, you could also trigger logging to a service management tool to quicken incident management and escalation.
Secure monitoring data
Monitoring can help you meet your regulatory requirements and commitment to customer safety, though you should ensure your monitoring stack is secure too. Monitoring tooling shouldn’t compromise your organization or customer privacy.
Label and annotate for better context
Kubernetes labels let you give a cluster’s objects meaningful names or descriptions. Annotations let you add useful information for others, like context, warnings, or clarity for why something works the way it works.
Using both options in clusters can help the flow of information and add clarity to your monitoring software.
Track your request and resource limits
Setting resource and request limits for containers helps ensure software and services only get what they need so they don’t impact your overall system. Setting these limits gives you a more predictable performance, but it also prevents unnecessary flags clogging up your monitoring data.
You should, however, monitor your cluster’s workloads to ensure the limits are still appropriate.
Continuously improve your monitoring strategies
Like all areas of Continuous Delivery and DevOps, you should look to improve everything in your software delivery processes, and monitoring should be no different.
Don’t set and forget! Instead, tinker; add new metrics and tracking items when new opportunities arise, and remove data when it’s no longer helpful.
Centralize your information
It’s worth centralizing all information you gather from your deployment pipeline. Centralizing logs, metrics, and traces can help you find correlations and interesting links you might not otherwise spot.
You’ll also be able to see exactly what impact an update or problem has on software performance and even customer reaction to it.
Kubernetes observability and monitoring tools
Without tooling, observability and monitoring for Kubernetes clusters, especially at scale, is a nightmare.
Let’s look at some of the tools available to help with understanding your clusters.
In-built tools
It’s not the most elegant method, but those used to managing Kubernetes with the ‘kubectl’ command-line interface can check the state of clusters by running kubectl get pods. This command gives you a full list of a cluster’s pods and their health. You can also drill down for more granular information on a pod using kubectl describe pod <pod-name>.
A more accessible way is to enable the Kubernetes Dashboard extension, which offers a clean, visual look at your Kubernetes cluster.
Both these options only work for a single cluster at a time, though, so they’re not viable for projects spanning multiple clusters.
Open-source solutions
If you don’t mind the time-cost of setup and maintenance, there are a few open-source monitoring tools to help with understanding your Kubernetes clusters.
Notable options include:
- Prometheus - A Kubernetes monitoring solution built by SoundCloud, now made open source. Prometheus regularly checks a cluster and keeps track of important changes, events, and metrics using time timestamps. You can add optional components to Prometheus for more functionality.
- Grafana - Grafana offers an open-source distribution to help you collection and visualization of software data.
Commercial solutions
For organizations that can afford it, commercial monitoring solutions are a great option to save themselves from the upkeep of using open-source or custom monitoring solutions.
Notable options include:
- Datadog - A popular monitoring solution for multiple software development and delivery elements. You can install Datadog’s agent on Kubernetes clusters to monitor infrastructure and application performance, plus other elements of your deployment pipeline.
- New Relic - Another full-stack software development monitoring tool with Kubernetes monitoring features. It allows you to track multiple clusters’ resources and problems using visualizations and graphs.
- Grafana - A set of premium products for observing and monitoring tech stacks.
Other tools that contribute to Kubernetes monitoring
Logging tools help make sure your deployment pipeline captures all useful information. Elastic offer 2 different logging tools:
- Elasticsearch - A RESTful search and analytics engine for data.
- Kibana - A dashboard that helps with observability and security.
Tracing tools help you manage the flow of data through a system:
- Jaeger - A tracing and troubleshooting tool focused on software built with microservices.
- Zipkin - A tracing tool that helps you find latency in service architecture.
Where Octopus helps with Kubernetes observability and monitoring
Octopus adds Continuous Delivery to Kubernetes by making K8s deployments simpler, routine, and more reliable.
Octopus also helps monitoring thanks to its dashboard, which lets you see:
- All deployments to clusters on one page, so anyone can understand what deployed where
- Detailed logging and audits about your deployment processes
- The status of connected infrastructure
Help us continuously improve
Please let us know if you have any feedback about this page.
