Kubernetes Ingress resources provide a way of configuring incoming HTTP traffic and make it easy to expose multiple services through a single public IP address.
NGINX has long provided one of the most popular ingress controllers, but anything more than a proof of concept deployment inevitably meant customizing routing rules beyond the standard properties exposed by Ingress resources.
Until recently, the solution was to define these additional settings via annotations or providing configuration blocks in configmaps. But with version 1.5, the NGINX ingress controller provides two custom resource definitions (CRDs) defining more complex networking rules than the baseline Ingress resources.
In this post, we’ll explore some of the new functionality provided by the VirtualServer and VirtualServerRoute CRDs.
The sample Kubernetes cluster
For this blog, I used the Kubernetes distribution bundled with Docker Desktop:
I then deployed the sample application created for the Istio blog series, which can be installed with:
kubectl apply -f https://raw.githubusercontent.com/mcasperson/NodejsProxy/master/kubernetes/example.yaml
The resulting cluster looks like this:
I used Helm to install NGINX, but it’s not as easy as it could be. The GitHub docs point you to the Helm repo at https://helm.nginx.com/edge, which failed for me. The chart from the official Helm repository at https://kubernetes-charts.storage.googleapis.com/ did not include the CRDs.
The solution was to clone the NGINX Git repo and install the Helm chart from a local file. These commands worked with Helm 3:
git clone https://github.com/nginxinc/kubernetes-ingress/
cd kubernetes-ingress/deployments/helm-chart
helm install nginx-release .
A basic NGINX VirtualServer
We’ll start with a basic VirtualServer resource to expose the proxy.
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
upstreams:
- name: proxy
service: proxy
port: 80
routes:
- path: /
action:
pass: proxy
The upstreams property defines the services that traffic can be sent to. In this example, we direct traffic to the proxy service.
The routes match incoming requests and perform an action in response. Typically the action is to direct traffic to an upstream server, which we have done with the action.pass: proxy configuration.
This VirtualServer replicates the functionality we might otherwise have defined in an Ingress resource, and once deployed to the cluster, we can open http://localhost/whatever/you/want to view the proxy web application.
Custom actions
Things become interesting when we start digging into the new functionality the VirtualServer exposes.
Instead of passing requests to an upstream service, the VirtualServer can redirect the client to a new URL. Here we direct traffic back to the NGINX homepage:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
routes:
- path: /
action:
redirect:
url: http://www.nginx.com
code: 301
In this example, we define the content to be returned directly in the VirtualServer resource. This is great for testing:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
routes:
- path: /
action:
return:
code: 200
type: text/plain
body: "Hello World\n"
Traffic splitting
Traffic splitting can be used for canary deployments by directing a percentage of traffic to a new service. Here we configure the VirtualServer to pass traffic to the webserver services, splitting traffic between webserverv1 and webserverv2.
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
upstreams:
- name: webserverv1
service: webserverv1
port: 80
- name: webserverv2
service: webserverv2
port: 80
routes:
- path: /
splits:
- weight: 80
action:
pass: webserverv1
- weight: 20
action:
pass: webserverv2
Load balancing
In iptables proxy mode, endpoints available to a service are chosen at random. If you look at the diagram above, the webserver service directs traffic to both webserverv1 and webserverv2 deployments, so traffic to the webserver service would be randomly distributed between all pods.
NGINX allows us to specify the load balancing rules used to direct traffic to upstream services. In the example below, we set the lb-method property to the ip_hash load balancing algorithm, ensuring a client is always sent to the same backend pod:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
upstreams:
- name: webserver
service: webserver
port: 80
lb-method: ip_hash
routes:
- path: /
action:
pass: webserver
Timeouts, retries, and keepalives
Lower level configuration details like connection timeouts, retries, and keepalives used to be defined as annotations on Ingress resources. With a VirtualServer resource, these settings are now exposed as first-class properties:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
upstreams:
- name: webserver
service: webserver
port: 80
fail-timeout: 10s
max-fails: 1
max-conns: 32
keepalive: 32
connect-timeout: 30s
read-timeout: 30s
send-timeout: 30s
next-upstream: "error timeout non_idempotent"
next-upstream-timeout: 5s
next-upstream-tries: 10
client-max-body-size: 2m
routes:
- path: /
action:
pass: webserver
Adding VirtualServerRoutes
VirtualServer resources can delegate the request to a VirtualServerRoute. This allows a master VirtualServer resource to configure top-level traffic rules with VirtualServerRoute resources handling more specific routing rules.
In the example below, requests are sent to a VirtualServerRoute resource which selects one of three possible upstream services based on the path:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtualserver
spec:
host: localhost
routes:
- path: /
route: virtualserverroute
---
apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
name: virtualserverroute
spec:
host: localhost
upstreams:
- name: proxy
service: proxy
port: 80
- name: webserverv1
service: webserverv1
port: 80
- name: webserverv2
service: webserverv2
port: 80
subroutes:
- path: /webserverv1
action:
pass: webserverv1
- path: /webserverv2
action:
pass: webserverv2
- path: /
action:
pass: proxy
Conclusion
Ingress controllers started as a cottage industry in the Kubernetes ecosystem, but as each provider differentiated itself from the competition with new configuration options and features, the annotations added to Ingress resources made them unwieldy and unportable.
By implementing CRDs, NGINX exposes advanced functionality with verifiable properties. I expect these CRDs will be enriched even further as additional common use cases are identified.
