Jump boxes or bastion hosts are a common networking strategy to expose a single secure entry point to the public internet, to access a private network. This single point of entry lets security teams closely monitor and control network access to the private network. Often the bastion host exposes a well known remote access service, like RDP or SSH, which teams can assume have been widely vetted and are trustworthy.
In this post, I explain how to host an OpenSSH server in a Kubernetes cluster to perform administrative tasks.
Deploying an SSH server
SSH servers have long been used to provide remote access to Linux servers, and it's relatively easy to host an SSH server as a Kubernetes pod.
The YAML file shown below creates a service account with a role and role-binding granting access to common resources in the current namespace. It then deploys an instance of the linuxserver/openssh-server image, inheriting the permissions of the service account, and exposes it via a load balancer service:
apiVersion: v1
kind: ServiceAccount
metadata:
name: k8s-admin
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: k8s-admin-role
rules:
- apiGroups: ["", "extensions", "apps", "networking.k8s.io"]
resources: ["deployments", "replicasets", "pods", "services", "ingresses", "secrets", "configmaps"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: k8s-admin-role-binding
subjects:
- kind: ServiceAccount
name: k8s-admin
apiGroup: ""
roleRef:
kind: Role
name: k8s-admin-role
apiGroup: ""
---
apiVersion: v1
kind: Service
metadata:
name: my-ssh-svc
labels:
app: ssh
spec:
type: LoadBalancer
ports:
- port: 2222
selector:
app: ssh
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-ssh
labels:
app: ssh
spec:
replicas: 1
selector:
matchLabels:
app: ssh
template:
metadata:
labels:
app: ssh
spec:
serviceAccountName: k8s-admin
containers:
- name: ssh
image: lscr.io/linuxserver/openssh-server:latest
ports:
- containerPort: 2222
env:
- name: PUID
value: "1000"
- name: PGID
value: "1000"
- name: TZ
value: "Australia/Brisbane"
- name: USER_NAME
value: "admin"
- name: USER_PASSWORD
value: "Password01!"
- name: PASSWORD_ACCESS
value: "true"
- name: SUDO_ACCESS
value: "true"
Note that, for convenience, this SSH server allows password access, the example YAML file embeds an insecure example password, and allows sudo access. A more robust solution is to use key files for authentication. Refer to the Docker Hub documentation for examples showing how to use key files for authentication.
Save the YAML above to a file called ssh.yaml and apply it with the command:
kubectl apply -f ssh.yaml
You can then find the IP address or hostname of the load balancer service with the command:
kubectl get service my-ssh-svc
On my local Kubernetes cluster, this command returned:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-ssh-svc LoadBalancer 10.96.164.169 172.21.255.202 2222:31628/TCP 29m
You can then SSH into the external IP address with the command:
ssh admin@172.21.255.202 -p 2222
You then have an interactive session inside the pod on the Kubernetes cluster.
Installing and configuring kubectl
To do anything useful with the cluster, you need to download kubectl and configure it to access the cluster from within the pod. Download and install kubectl with the commands:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
By default, pods have a number of files mounted under /var/run/secrets/kubernetes.io/serviceaccount that let the pod interact with the host cluster. To configure kubectl to use these files, save the following file to ~/.kube.config:
apiVersion: v1
clusters:
- cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://kubernetes.default
name: localk8s
contexts:
- context:
cluster: localk8s
user: user
name: localk8s
current-context: localk8s
kind: Config
preferences: {}
users:
- name: user
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
You can now run kubectl from your SSH session and interact with the parent cluster, providing a convenient and secure environment for cluster administration.
Building a custom OpenSSH Docker image
Downloading kubectl and copying the configuration file is easy enough, but the ephemeral nature of Kubernetes pods means eventually the container will be deleted and recreated, forcing you to download and configure kubectl again.
A better solution is baking kubectl and its configuration file into a custom Docker image. This ensures the files are available in the container when it's first started.
Save the following to a file called Dockerfile:
FROM lscr.io/linuxserver/openssh-server:latest
RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
RUN printf 'apiVersion: v1\n\
clusters:\n\
- cluster:\n\
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n\
server: https://kubernetes.default\n\
name: localk8s\n\
contexts:\n\
- context:\n\
cluster: localk8s\n\
user: user\n\
name: localk8s\n\
current-context: localk8s\n\
kind: Config\n\
preferences: {}\n\
users:\n\
- name: user\n\
user:\n\
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token' >> /opt/kubeconfig
RUN printf 'mkdir /config/.kube \n\
cp /opt/kubeconfig /config/.kube/config' >> /etc/cont-init.d/100-kubeconfig
You then build the custom Docker image with the following command, where yourdockerregistry is replaced with the name of a Docker registry you have the ability to push images to:
docker build . -t yourdockerregistry/openssh-server:latest
Replace the image property in the Kubernetes YAML file with:
image: yourdockerregistry/openssh-server:latest
After the new SSH server pods are created using your custom image, kubectl and its configuration file are ready to use without first downloading them.
Conclusion
A bastion host running OpenSSH on your Kubernetes cluster provides you with a single, secure entry point for administration and debugging tasks. By customizing the Docker image to include common tools like kubectl, DevOps teams can rely on the bastion host having any required tools for common administration tasks.
Happy deployments!
