Managing your Kubernetes configuration
Managing your configuration
Even the most simple of applications requires configuration. This could be application configuration, like a database connection string, or infrastructure configuration, like the number of replica sets in your deployment. Deploying your configuration will demand flexibility - you'll want to override options on a perenvironment or tenant basis, and may even need to change configuration in response to incidents or bugs found after a deployment.
To help visualize managing configuration, we'll configure a small, imaginary application called bookshop. The purpose of this application is an inventory management system for books. There are endpoints to list all the books in inventory, see the authors available and their books, and endpoints to process the sales of books. This application starts from hard-coded values in source code, and we'll deploy it safely and quickly!
Externalizing your configuration
The first step to managing your configuration is externalizing it. This is the process of separating your configuration from your source code. The end goal is the ability to adjust the behavior of your code to suit any environment or tenant without changing the code.
In our bookshop example, there are some configuration values that we think we'll need to modify depending on the environment/tenant:
- Database connection details
- The listening port of the server
- Log levels
- The currency that the books are sold for
- The name of the store
As a part of externalizing, the code is now configured to pull this information dynamically through a configuration file, or environment variables. Here's an example configuration file:
Store:
Name: Read Only Volumes
Database:
Server: postgres.svc.local
Username: db
Password: db_password
Log:
Level: Info
Currency: USD
The code will also look for these options in its environment, using the full configuration path delimited by __, in all caps. As an example, the database server could also be set using an environment variable like this:
STORE__NAME="Read Only Volumes"
Storing your configuration
With your configuration externalized, you can now store it separately from your code. The easiest way to do this is using your CD tool. Tools like Octopus Deploy have features that allow you to define environments and tenants, which you can use to store configuration safely. If you use GitOps, the configuration for your application will be stored in Git, with each environment or tenant being a repository. You should not store your configuration in the same repository as your application's source code.
Storing your secrets
With your configuration externalized, you can now store it separately from your code. The easiest way to do this is using your CD tool. Tools like Octopus Deploy have features that allow you to define environments and tenants, which you can use to store configuration safely. If you use GitOps, the configuration for your application will be stored in Git, with each environment or tenant being a repository. You should not store your configuration in the same repository as your application's source code.
Sealed secrets
Sealed secrets is a tool you can install in your cluster that lets you encrypt secrets until they're inside your cluster. This lets you commit your secrets to Git safely, as only the cluster can decrypt the secrets for use. You can find out more information about sealed secrets in the GitHub documentation.
Though sealed secrets are cryptographically secure, there are operational overheads to managing your secrets through sealed secrets. As an example, if you need to rotate an API key used across multiple clusters, you need to encrypt the secrets separately for each cluster. This is because each cluster has its own encryption and decryption key, so you can't reuse the secret in its encrypted form.
Inject secrets at deployment time
If your CD tooling lets you store secrets securely, you can inject the secrets into your configuration during the deployment process. This is usually done by dynamically deploying a Kubernetes Secret resource consumed by your application. For each tool, this will differ. For example, you can use the following YAML to deploy a secret called DatabasePassword in Octopus Deploy:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: database-password
data:
database-password: #{DatabasePassword | ToBase64}
This works similarly with other tools. GitHub Actions, for example, replaces ${{ secrets.DatabasePassword }} with the secret defined in your repository or organization settings.
External secret operators
There are many external secret tools available, including:
- HashiCorp Vault
- Azure Key Vault
- AWS Secrets Manager
- Google Secret Manager
- 1Password
The first step to using external secrets is to add your secrets to the tool. The tool is the only place you should store your secrets where possible. You can use an external secret operator to access the secrets from your cluster. These operators let you reference a path to a secret, and use cluster credentials to log in to the external secret operator and retrieve the secrets for use. This is the gold standard for secret management.
As an example, if you stored the database password in HashiCorp Vault, you could use the external secrets operator that Vault provides. The configuration you'd deploy (or keep in Git) would look like this:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: database-password
spec:
vaultAuthRef: vaultauth-sample
mount: kvv2
type: kv-v2
path: secret
refreshAfter: 5s
destination:
name: database-password
Upon deploying this resource, the Vault operator creates the database-password secret and keeps the data up-to-date with the secret that's stored in Vault.
Environments and tenants
When configuring across environments and tenants, you should be careful that your configuration is properly isolated between each environment and tenant. The last thing you want is to have your development environment using a production database, only for a bug to accidentally wipe the database, causing data loss. CD tools like Octopus Deploy and dedicated secret management tools like HashiCorp Vault let you define scopes to prevent cross-contamination of configuration and secrets.
Seeing what's configured in each environment and tenant helps to prevent confusion as you add more environments and tenants to your CD process.
Configuration access management
Since your configuration can impact your environment as a code change, you should treat it with just as much deference. Just because an engineer can push configuration changes to a development environment doesn't mean they should also be able to push configuration changes to production. You should control access to pre-prod and production environments where appropriate.
Auditing
Code changes aren't the only cause of failures when deploying to an environment, so you should ensure you can easily track configuration changes in any environment. Configuration changes are just as important to an environment's overall health and performance as the version of the code that's running. As such, seeing the history of the environment's configuration is important. This is because being able to see that a configuration option was changed helps narrow down the causes of problems in any environment. GitOps excels at this, with Git being able to track revisions from the beginning without any issues.