Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
INSIGHTS
10 min read
Published on 04/06/2022
Last updated on 08/19/2024
Facelift Kurun for Kubernetes Event Tunneling
Share
What is Kurun?
Kurun is a multi-tool to help Kubernetes developers. We can summarize the features in 3 short sentences:
- Just like go run main.go but executed inside Kubernetes with one command.
- Just like kubectl port-forward ... but the other way around!
- Just like kubectl apply -f pod.yaml but images are built from local source code.
The second statement is especially handy during Kubernetes admission webhook developement. It's quite hard to configure an operator running on your local computer and receive admission webhook requests from within a remote Kubernetes cluster.
Recap of admission webhooks
Ideally, the operator runs inside the cluster, so the admission controllers send the admission requests to the operator directly. 
However, during development, it is more practical to keep your operator on your local machine (for debugging and other reasons), but unfortunately, admission controllers cannot access your operator this way. Applications running inside Kubernetes usually cannot open connections to your local workstation, partly for security reasons, but mostly because of IPv4 NAT. 
Solution
The basic idea is to initiate a full-duplex, keepalive connection from the local machine (client side) into the Kubernetes cluster (server side) and use this connection to reverse-proxy the admission requests from the server to the client and deliver the admission responses. 
First version
The first version of kurun was created in a strong need and followed the idea of "make it available as soon as possible". Initially Kurun was a bash script as well. Therefore we used third parties to solve problems, like:
- Tunneling: A practical third party named
inletswas responsible for the tunneling. It was able to maintain the communication between the server and client side over a WebSocket (WS) connection. - TLS termination: Since the
inletsserver did not provided an option for serving TLS connections, we embedded another third party named Ghostunnel. It was responsible for terminating TLS before theinletsserver.
The initial version depended on kubectl exec calls to manage Kubernetes resources which was the quickest way of implementing Kubernetes communications. 
Why fix it when it works?
The above concept worked well for a long time. Unfortunately the inlets stack became unavailable.
"I pivoted inlets this year from open source, with a separate commercial version, to a commercial version with open-source automation and tooling. Why? The Open Source model wasn't serving the needs of my business..." - Alex Ellis
You can read more details in this blog post. But as developers who are enthusiastic for open-source we took Alex advice:
"So I flipped this around. If you only used inlets because it was free, find an alternative (there are a bunch of them, with different tradeoffs) or pay for it." - Alex Ellis
We wrote our version of tuneling between Kubernetes and a local computer.
If you do something do it right!
The new version of kurun (0.6.0) upgrades several component:
- Kubernetes resource management: Uses the standard k8s.io Go modules to access the cluster and manage resources. Creates a deployment for the
kurunserver pod and also sets up a service for the pod. - Tunneling:
kurunclient uses the official k8s.io Go modules to create an HTTP client that can access the cluster's API server. It relies on the API server proxy to open and maintain a secure WebSocket connection to thekurunserver. This connection will be used as a tunnel between the local machine and the Kubernetes cluster. - TLS termination: Now kurun has built-in TLS termination, there's no need to use any external tools.
| Task | First version | New version |
|---|---|---|
| Secure proxying | - | Secure WebSocket connection proxied by Kubernetes API server |
| Tunneling | Inlets (3rd party) | Native request/response proxy through WSS |
| TLS termination | Ghostunnel (3rd party) | Native go/http package |
| Accessing cluster resources | kubectl | Standard go k8s.io libraries |
What is under the hood?
Warning, the following paragraph is a deep technical description of kurun's low-level implementation. If you are not interested in this kind of information skip to the Webhook example. 
Proxying requests through a WebSocket tunnel might seem easy, but it's trickier than you think. Originally, WebSockets are handling bidirectional traffic without state. Hence, when you want to proxy HTTP calls over WebSockets you have two options. Create a new WebSocket connection for every single request, which is trivial but not efficient. The second is to track the parallel connection's packets. And this is what you see in the figure. Let's see this step-by-step:
- The server receives the HTTP request from the Kubernetes API server.
- It creates a goroutine to handle the request.
- This routine encapsulates the packet with an
idand sends it through the WebSocket. - The goroutine then waits for the response with the matching
id.
On the client-side, similar things happen:
- The client receives the encapsulated HTTP request along with its
id. - It sends the request to the (webhook) handler and waits for the response.
- The response is encapsulated with the same
idand sent back on the WebSocket.
Finally, the server receives the response with the proper id and forwards the response to the Kubernetes API. With this solution, it is possible to do a non-blocking serialized tunnel between the Kubernetes API and your machine.
Webhook example
You can use the following example configuration with a sample application to try out kurun port-forward feature in a sandbox environment.
| Component | Description |
|---|---|
| cert-manager | A great Kubernetes tool for managing TLS certs and secrets. |
| kube-service-annotate | A standalone tool which is responsible for the webhook logic. It annotates newly created services in your Kubernetes cluster. This will run on your local machine in this case. |
| kurun | Will set up a tunnel for reverse proxying the admission requests securely between your local machine and your Kubernetes cluster. |
Requirements:
- Configuration available on your local machine for accessing a running Kubernetes cluster. It can also be kind/k3d/minikube
gitto clone repositoriesgolangto run kurun and kube-service-annotatejqto read client TLS data from Kubernetes secrets
1. Install cert-manager
Probably the most convenient way to create/manage TLS certs in Kubernetes is using cert-manager. Our little example will also use it for obvious reasons. Install cert-manager into your Kubernetes cluster with the following command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml
2. Generate certificates
Since Kubernetes' admissionregistration controllers use secure connections exclusively for webhooks, you will need some certificates to make things to operate. Please note we also added localhost to dnsNames of the certificate. This is required to be able to use the TLS cert on your local machine too.
Further information about Kubernetes webhooks: Dynamic Admission Control
Run the following command to create the following items:
- CA Issuer
- CA Cert
- CA Secret
- Server Cert Issuer
- Server Cert
- Server TLS Secret
Your output should be something like below:
issuer.cert-manager.io/kurun-ca-issuer created
certificate.cert-manager.io/kurun-ca-cert created
issuer.cert-manager.io/kurun-issuer created
certificate.cert-manager.io/kurun-cert created3. Setup kube-service-annotate
kube-service-annotate is a simple sample application which handles admission requests and returns admission responses over HTTP and HTTPS. It basicly annotates Kubernetes services based on a ruleset. This application will run on your local machine this time, playing the role of an operator under development. You can clone the repository with the following command:
git clone https://github.com/banzaicloud/kube-service-annotate.gitkube-service-annotate will look for a config file named rules.yaml by default which contains the annotation rule set. Create a new file rules.yaml with the following example content:
- selector:
app: my-service
annotations:
my-label-dependant-annotation: true
- annotations:
always-annotate-this: true4. Save TLS secrets to local machine
Now we have our certs in Kubernetes, but we will also need them on our local machine to run kube-service-annotate in secure mode. With the following bash commands you can extract and save the certificate and private key from the server secret to your local machine.
kubectl get secret kurun-secret -o json | jq -r '.data["tls.crt"]' | base64 -d > tls.crt
kubectl get secret kurun-secret -o json | jq -r '.data["tls.key"]' | base64 -d > tls.key5. Run kube-service-annotate
The following line will start the kube-service-annotate on your local machine:
go run main.go --tls-key-file tls.key --tls-cert-file tls.crt --rules-file rules.yamlThe output should look something like this:
2022/03/01 11:27:56 [INFO] Reading rules from: "rules.yaml"
2022/03/01 11:27:56 [INFO] There are 2 active rules
2022/03/01 11:27:56 [WARN] no metrics recorder active
2022/03/01 11:27:56 [WARN] no tracer active
2022/03/01 11:27:56 [INFO] Listening TLS on :80806. Install kurun
Install kurun using one of the following methods:
with go
go install github.com/banzaicloud/kurunwith brew
brew install kurunor clone the git repository
git clone https://github.com/banzaicloud/kurun.git
7. Run kurun port-forward
kurun runs locally, but it has access to your Kubernetes cluster with the configuration stored in your KUBECONFIG. kurun performs the following actions on the cluster:
- Creates a deployment for kurun-server. The kurun-server listens (over HTTPS) inside the cluster and proxies requests to your local machine.
- A service will be generated which points to the kurun-server pod.
Run the following command to start kurun with the example parameter set:
kurun port-forward https://localhost:8080 --tlssecret kurun-secret -vExplanation of parameters:
- https://localhost:8080 : the target where kurun client will proxy the admission requests to (this is where
kube-service-annotateis listening) - --tlssecret kurun-secret : reads the client TLS secret from the Kubernetes secret and uses it on the client side
You can exit from kurun client any time by hitting Ctrl+C. This automatically starts a cleanup process which removes all client-generated resources from your Kubernetes cluster, but see also the next step when using mutating webhooks.
7. Setup mutatingwebhookconfiguration
This step is essential when creating mutating webhooks. You need to have a mutating webhook configuration in your Kubernetes cluster, so admission controllers know where/how to send admission requests. Note that the service configuration now points to the kurun service.
kubectl apply -f -<<EOF
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: kurun-webhook
annotations:
cert-manager.io/inject-ca-from: "default/kurun-ca-cert"
webhooks:
- name: kurun-webhook.banzaicloud.com
clientConfig:
service:
name: kurun
namespace: default
path: "/mutate-secrets"
port: 80
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["services"]
scope: "*"
admissionReviewVersions: ["v1beta1", "v1"]
sideEffects: None
EOFDon't forget to remove/edit this MutatingWebhookConfiguration after exiting
kurun! Otherwise, it would point to a non-existing service becausekurunremoves its own resources on exit, and requests to the service would fail, resulting in your inability to create services (or resources set in the MutatingWebhookConfiguration).
8. Create service
Now everything is set up! By running the following script you can create a new dummy service that actually does nothing. On the other hand, the following things do happen (simplified):
- The Kubernetes API server receives the request, passes it to the admission controller.
- The admission controller checks mutationwebhookconfigurations and sends the admission request to the (kurun) service we set up.
kurunservice receives the admission request formy-serviceand proxies it to the pod runningkurun-server.kurunserver proxies the admission request through its WSS tunnel tokurunclient on the local machine.kurunclient forwards the admission request tokube-service-annotate.kube-service-annotatechecks the admission request ofmy-service, and applies annotations to the service descriptor from matching rules inrules.yaml.kurunclient receives the admission response fromkube-service-annotate, proxies it through the WSS tunnel to thekurunserver.kurunserver receives the admission response and returns it to the admission controller.The admission controller passes on the extended
my-servicedescriptor.kubectl apply -f -<<EOF apiVersion: v1 kind: Service metadata: name: my-service namespace: default labels: app: my-service spec: clusterIP: None clusterIPs: - None type: ClusterIP EOF
kurun client logs the request:
2022/03/01 16:17:29 "level"=1 "msg"="handling request" "wsConn"={} "client"={} "id"=824634754304 "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Accept-Encoding":["gzip"],"Content-Type":["application/json"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"","RequestURI":"/mutate-secrets?timeout=10s","TLS":null,"Cancel":"<unhandled-chan>","Response":null}kurun server also logs the events:
2022/03/01 16:17:29 "level"=1 "msg"="request received" "server"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"Content-Type":["application/json"],"Accept-Encoding":["gzip"],"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null}
2022/03/01 16:17:29 "level"=1 "msg"="request queued" "server"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null} "id"=824634754304
2022/03/01 16:17:29 writeLoop: "level"=1 "msg"="processing request" "server"={} "conn"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null}9. Validating the results
Checking the newly created service to see the annotations added by kube-service-annotate.
kubectl describe svc my-serviceThe output should be similar to:
Name: my-service
Namespace: default
Labels: app=my-service
Annotations: always-annotate-this: true
my-label-dependant-annotation: true
Selector: <none>
Type: ClusterIP
IP Family Policy: RequireDualStack
IP Families: IPv4,IPv6
IP: None
IPs: None
Session Affinity: None
Events: <none>The complexity of example might be frightening for beginners, but starting to develop webhook based applications will make familiar all the things above. We think kurun is a useful developer tool which we actively use it in everyday work. We hope it will also make your work easier as a Kubernetes developer.
Subscribe to
the Shift!
Get emerging insights on innovative technology straight to your inbox.
