Facelift Kurun for Kubernetes Event Tunneling

What is Kurun?

Kurun is a multi-tool to help Kubernetes developers. We can summarize the features in 3 short sentences:

• Just like go run main.go but executed inside Kubernetes with one command.
• Just like kubectl port-forward ... but the other way around!
• Just like kubectl apply -f pod.yaml but images are built from local source code.

The second statement is especially handy during Kubernetes admission webhook developement. It's quite hard to configure an operator running on your local computer and receive admission webhook requests from within a remote Kubernetes cluster.

Recap of admission webhooks

Ideally, the operator runs inside the cluster, so the admission controllers send the admission requests to the operator directly. However, during development, it is more practical to keep your operator on your local machine (for debugging and other reasons), but unfortunately, admission controllers cannot access your operator this way. Applications running inside Kubernetes usually cannot open connections to your local workstation, partly for security reasons, but mostly because of IPv4 NAT.

Solution

The basic idea is to initiate a full-duplex, keepalive connection from the local machine (client side) into the Kubernetes cluster (server side) and use this connection to reverse-proxy the admission requests from the server to the client and deliver the admission responses.

First version

The first version of kurun was created in a strong need and followed the idea of "make it available as soon as possible". Initially Kurun was a bash script as well. Therefore we used third parties to solve problems, like:

• Tunneling: A practical third party named inlets was responsible for the tunneling. It was able to maintain the communication between the server and client side over a WebSocket (WS) connection.
• TLS termination: Since the inlets server did not provided an option for serving TLS connections, we embedded another third party named Ghostunnel. It was responsible for terminating TLS before the inlets server.

The initial version depended on kubectl exec calls to manage Kubernetes resources which was the quickest way of implementing Kubernetes communications.

Why fix it when it works?

The above concept worked well for a long time. Unfortunately the inlets stack became unavailable.

"I pivoted inlets this year from open source, with a separate commercial version, to a commercial version with open-source automation and tooling. Why? The Open Source model wasn't serving the needs of my business..." - Alex Ellis

You can read more details in this blog post. But as developers who are enthusiastic for open-source we took Alex advice:

"So I flipped this around. If you only used inlets because it was free, find an alternative (there are a bunch of them, with different tradeoffs) or pay for it." - Alex Ellis

We wrote our version of tuneling between Kubernetes and a local computer.

If you do something do it right!

The new version of kurun (0.6.0) upgrades several component:

• Kubernetes resource management: Uses the standard k8s.io Go modules to access the cluster and manage resources. Creates a deployment for the kurun server pod and also sets up a service for the pod.
• Tunneling: kurun client uses the official k8s.io Go modules to create an HTTP client that can access the cluster's API server. It relies on the API server proxy to open and maintain a secure WebSocket connection to the kurun server. This connection will be used as a tunnel between the local machine and the Kubernetes cluster.
• TLS termination: Now kurun has built-in TLS termination, there's no need to use any external tools.

What is under the hood?

Warning, the following paragraph is a deep technical description of kurun's low-level implementation. If you are not interested in this kind of information skip to the Webhook example. Proxying requests through a WebSocket tunnel might seem easy, but it's trickier than you think. Originally, WebSockets are handling bidirectional traffic without state. Hence, when you want to proxy HTTP calls over WebSockets you have two options. Create a new WebSocket connection for every single request, which is trivial but not efficient. The second is to track the parallel connection's packets. And this is what you see in the figure. Let's see this step-by-step:

1. The server receives the HTTP request from the Kubernetes API server.
2. It creates a goroutine to handle the request.
3. This routine encapsulates the packet with an id and sends it through the WebSocket.
4. The goroutine then waits for the response with the matching id.

On the client-side, similar things happen:

1. The client receives the encapsulated HTTP request along with its id.
2. It sends the request to the (webhook) handler and waits for the response.
3. The response is encapsulated with the same id and sent back on the WebSocket.

Finally, the server receives the response with the proper id and forwards the response to the Kubernetes API. With this solution, it is possible to do a non-blocking serialized tunnel between the Kubernetes API and your machine.

Webhook example

You can use the following example configuration with a sample application to try out kurun port-forward feature in a sandbox environment.

Requirements:

• Configuration available on your local machine for accessing a running Kubernetes cluster. It can also be kind/k3d/minikube
• git to clone repositories
• golang to run kurun and kube-service-annotate
• jq to read client TLS data from Kubernetes secrets

1. Install cert-manager

Probably the most convenient way to create/manage TLS certs in Kubernetes is using cert-manager. Our little example will also use it for obvious reasons. Install cert-manager into your Kubernetes cluster with the following command:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml

2. Generate certificates

Since Kubernetes' admissionregistration controllers use secure connections exclusively for webhooks, you will need some certificates to make things to operate. Please note we also added localhost to dnsNames of the certificate. This is required to be able to use the TLS cert on your local machine too.

Further information about Kubernetes webhooks: Dynamic Admission Control

Run the following command to create the following items:

• CA Issuer
• CA Cert
• CA Secret
• Server Cert Issuer
• Server Cert
• Server TLS Secret

Your output should be something like below:

issuer.cert-manager.io/kurun-ca-issuer created
certificate.cert-manager.io/kurun-ca-cert created
issuer.cert-manager.io/kurun-issuer created
certificate.cert-manager.io/kurun-cert created

3. Setup kube-service-annotate

kube-service-annotate is a simple sample application which handles admission requests and returns admission responses over HTTP and HTTPS. It basicly annotates Kubernetes services based on a ruleset. This application will run on your local machine this time, playing the role of an operator under development. You can clone the repository with the following command:

git clone https://github.com/banzaicloud/kube-service-annotate.git

kube-service-annotate will look for a config file named rules.yaml by default which contains the annotation rule set. Create a new file rules.yaml with the following example content:

- selector:
    app: my-service
  annotations:
    my-label-dependant-annotation: true
- annotations:
    always-annotate-this: true

4. Save TLS secrets to local machine

Now we have our certs in Kubernetes, but we will also need them on our local machine to run kube-service-annotate in secure mode. With the following bash commands you can extract and save the certificate and private key from the server secret to your local machine.

kubectl get secret kurun-secret -o json | jq -r '.data["tls.crt"]' | base64 -d > tls.crt
kubectl get secret kurun-secret -o json | jq -r '.data["tls.key"]' | base64 -d > tls.key

5. Run kube-service-annotate

The following line will start the kube-service-annotate on your local machine:

go run main.go --tls-key-file tls.key --tls-cert-file tls.crt --rules-file rules.yaml

The output should look something like this:

2022/03/01 11:27:56 [INFO] Reading rules from: "rules.yaml"
2022/03/01 11:27:56 [INFO] There are 2 active rules
2022/03/01 11:27:56 [WARN] no metrics recorder active
2022/03/01 11:27:56 [WARN] no tracer active
2022/03/01 11:27:56 [INFO] Listening TLS on :8080

6. Install kurun

https://github.com/banzaicloud/kurun

Install kurun using one of the following methods:

• with go

  go install github.com/banzaicloud/kurun

• with brew

  brew install kurun

• or clone the git repository

  git clone https://github.com/banzaicloud/kurun.git

7. Run kurun port-forward

kurun runs locally, but it has access to your Kubernetes cluster with the configuration stored in your KUBECONFIG. kurun performs the following actions on the cluster:

• Creates a deployment for kurun-server. The kurun-server listens (over HTTPS) inside the cluster and proxies requests to your local machine.
• A service will be generated which points to the kurun-server pod.

Run the following command to start kurun with the example parameter set:

kurun port-forward https://localhost:8080 --tlssecret kurun-secret -v

Explanation of parameters:

• https://localhost:8080 : the target where kurun client will proxy the admission requests to (this is where kube-service-annotate is listening)
• --tlssecret kurun-secret : reads the client TLS secret from the Kubernetes secret and uses it on the client side

You can exit from kurun client any time by hitting Ctrl+C. This automatically starts a cleanup process which removes all client-generated resources from your Kubernetes cluster, but see also the next step when using mutating webhooks.

7. Setup mutatingwebhookconfiguration

This step is essential when creating mutating webhooks. You need to have a mutating webhook configuration in your Kubernetes cluster, so admission controllers know where/how to send admission requests. Note that the service configuration now points to the kurun service.

kubectl apply -f -<<EOF
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: kurun-webhook
  annotations:
    cert-manager.io/inject-ca-from: "default/kurun-ca-cert"
webhooks:
  - name: kurun-webhook.banzaicloud.com
    clientConfig:
      service:
        name: kurun
        namespace: default
        path: "/mutate-secrets"
        port: 80
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["services"]
        scope: "*"
    admissionReviewVersions: ["v1beta1", "v1"]
    sideEffects: None
EOF

Don't forget to remove/edit this MutatingWebhookConfiguration after exiting kurun! Otherwise, it would point to a non-existing service because kurun removes its own resources on exit, and requests to the service would fail, resulting in your inability to create services (or resources set in the MutatingWebhookConfiguration).

8. Create service

Now everything is set up! By running the following script you can create a new dummy service that actually does nothing. On the other hand, the following things do happen (simplified):

• The Kubernetes API server receives the request, passes it to the admission controller.
• The admission controller checks mutationwebhookconfigurations and sends the admission request to the (kurun) service we set up.
• kurun service receives the admission request for my-service and proxies it to the pod running kurun-server.
• kurun server proxies the admission request through its WSS tunnel to kurun client on the local machine.
• kurun client forwards the admission request to kube-service-annotate.
• kube-service-annotate checks the admission request of my-service, and applies annotations to the service descriptor from matching rules in rules.yaml.
• kurun client receives the admission response from kube-service-annotate, proxies it through the WSS tunnel to the kurun server.
• kurun server receives the admission response and returns it to the admission controller.

• The admission controller passes on the extended my-service descriptor.

  kubectl apply -f -<<EOF
  apiVersion: v1
  kind: Service
  metadata:
    name: my-service
    namespace: default
    labels:
      app: my-service
  spec:
    clusterIP: None
    clusterIPs:
    - None
    type: ClusterIP
  EOF

kurun client logs the request:

2022/03/01 16:17:29 "level"=1 "msg"="handling request" "wsConn"={} "client"={} "id"=824634754304 "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Accept-Encoding":["gzip"],"Content-Type":["application/json"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"","RequestURI":"/mutate-secrets?timeout=10s","TLS":null,"Cancel":"<unhandled-chan>","Response":null}

kurun server also logs the events:

2022/03/01 16:17:29 "level"=1 "msg"="request received" "server"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"Content-Type":["application/json"],"Accept-Encoding":["gzip"],"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null}
2022/03/01 16:17:29 "level"=1 "msg"="request queued" "server"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null} "id"=824634754304
2022/03/01 16:17:29 writeLoop: "level"=1 "msg"="processing request" "server"={} "conn"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null}

9. Validating the results

Checking the newly created service to see the annotations added by kube-service-annotate.

kubectl describe svc my-service

The output should be similar to:

Name:              my-service
Namespace:         default
Labels:            app=my-service
Annotations:       always-annotate-this: true
                   my-label-dependant-annotation: true
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  RequireDualStack
IP Families:       IPv4,IPv6
IP:                None
IPs:               None
Session Affinity:  None
Events:            <none>

The complexity of example might be frightening for beginners, but starting to develop webhook based applications will make familiar all the things above. We think kurun is a useful developer tool which we actively use it in everyday work. We hope it will also make your work easier as a Kubernetes developer.