Kafka on Kubernetes with Local Persistent Volumes

If you are looking to try out an automated way to provision and manage Kafka on Kubernetes, please follow this Kafka on Kubernetes the easy way link.

At Banzai Cloud we use Kafka internally a lot. We have some internal systems and customer reporting deployments where we rely heavily on Kafka deployed to Kubernetes. We practice what we preach and all these deployments (not just the external ones) are done using our application platform, Pipeline. There is one difference between regular Kafka deployments and ours (though it is not relevant to this post): we have removed Zookeeper and use etcd instead.

Some of our older posts about Apache Kafka on Kuberbetes: Kafka on Kubernetes - using etcd Monitoring Apache Kafka with Prometheus Kafka on Kubernetes with Local Persistent Volumes Kafka on Kubernetes the easy way

We would like to contribute our changes back into the official Kafka codebase so we have opened a ticket KAFKA-6598 and a KIP KIP-273. If you want to chip in, follow along on the official Kafka mailing list.

For a quick overview of Kafka on Kubernetes - using etcd see the diagram below: In today's blog we're not going to go into detail about how we added support to etcd because we already have a post about that. This post will talk about how to deploy Kafka on top of Kubernetes with Pipeline using Local Persistent Volumes - a beta feature in Kubernetes 1.10.

If you are interested in our previous posts on the subject, read:

Kafka on Kubernetes - using etcd Monitoring Apache Kafka with Prometheus

Later on in this blog, we're going to talk about Kubernetes Persistent Volume related resources, so, if you're not familiar with those resources, please read our earlier blog post about it.

Kubernetes persistent volume options

Local Persistent Volume

Local Persistent Volume is a beta feature in Kubernetes 1.10. It was created to leverage local disks and it enables their use with Persistent Volume Claims, PVC. This type of storage is suitable for applications that handle data replication. Kafka is an application with its own data replication, so Local Persistent Volume is a perfect fit. Moreover, because of locally attached SSD's, we can hope for a better performance than from alternatively managed storage.

Note: there are some systems we use or deploy for our customers which already handle replications - like HDFS, Cassandra, etc - and Local Persistent Volume is a good fit for those.

To use Persistent Volume, there are some manual steps to go through first:

• Pre-partition, format and mount disks to nodes
• Create Persistent Volumes
  • Manually
  • Using a DaemonSet which handles the creation.
• Create Storage Class

In later Kubernetes releases, these steps will be omitted.

Because Kubernetes already has a feature called hostPath which allows us to use local disk as a storage for Pods, you may ask, 'Why should I use Local Persistent Volume instead?'' When using hostPath the storage path has to be set inside the Pod descriptor. On the other hand, when Local Persistent Volume is used, storage can be preserved through a Persistent Volume Claim, so the storage path is not encoded directly to the Pod spec.

Creating a Kubernetes Cluster

If you've chosen to use Pipeline to create a Kubernetes cluster on any of its supported cloud providers, you need a control plane. For a control plane you'll need a GitHub account and an OAuth App, the latter of which must be setup upfront, since OAuth is used for authentication. Once the prerequisites are satisfied, follow the instructions described in this blog to spin up a Kubernetes cluster.

Please note, Local Persistent Volume only works when Kubernetes is version 1.10+.

Determining and setting prerequisites

When using Pipeline there are two ways of provisioning a Kafka cluster.

• either use a deployment through Pipeline, which uses a production ready Kafka Helm chart behind the scenes
• or, if you're an enterprise customer that's opted for cluster management, we deploy Pipeline with an operator based on the new operator SDK

Below, we're going to use Pipeline to deploy Kafka - in this case with Helm over a RESTful API, using our open source Kafka charts. Since Pipeline only accepts requests from authenticated users, users must authenticate first, then acquire a token that will be used in subsequent curl requests.

If you want formatted results install/use jq

First, we need to determine the id of the cluster we intend to deploy Kafka to from a list of the clusters managed by Pipeline. The API call to the list of managed Kubernetes clusters is:

curl -g --request GET \
 --url 'http://{{url}}/api/v1/orgs/{{orgId}}/clusters' \
 --header 'Authorization: Bearer {{token}}' | jq

[ { "status": "RUNNING",
"status_message": "Cluster is running", "name":
"gkecluster-baluchicken-792", "location": "europe-west1-b",
"cloud": "google", "id": 1, "nodePools": { "pool1": {
"count": 3, "instance_type": "n1-standard-2" } } } ]

To interact with the Kubernetes Cluster with the standard kubectl, let's download the Kubernetes config. 

curl -g --request GET \
 --url
'http://{{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/config'
\
 --header 'Authorization: Bearer {{token}}'

Save the Kube config to a file in your computer then point the environment variable KUBECONFIG to that file. 

export
KUBECONFIG=/Users/baluchicken/Downloads/kubi.conf

Deploying requirements for Local Persistent Volume

As mentioned above, we need to do some manual configurations to get Local Persistent Volume working. We're going to use the local volume static provisioner, which will create the Persistent Volume. In a three node cluster, where each node has one SSD attached, this DaemonSet will create three individual Persistent Volumes. The code listed below also contains a ConfigMap, which tells the provisioner where to look for mounted disks. When using Google, all SSDs are mounted under /mnt/disks/ssdx.

kubectl create -f - <<EOF apiVersion:
v1 kind: ConfigMap metadata: name: local-provisioner-config
namespace: default data: storageClassMap: | local-scsi:
hostDir: /mnt/disks mountDir: /mnt/disks

---

apiVersion: extensions/v1beta1 kind: DaemonSet metadata:
name: local-volume-provisioner namespace: default labels:
app: local-volume-provisioner spec: selector: matchLabels:
app: local-volume-provisioner template: metadata: labels:
app: local-volume-provisioner spec: containers: - image:
"quay.io/external_storage/local-volume-provisioner:v2.1.0"
imagePullPolicy: "Always" name: provisioner securityContext:
privileged: true env: - name: MY_NODE_NAME valueFrom:
fieldRef: fieldPath: spec.nodeName volumeMounts: -
mountPath: /etc/provisioner/config name: provisioner-config
readOnly: true - mountPath: /mnt/disks name: local-scsi
volumes: - name: provisioner-config configMap: name:
local-provisioner-config - name: local-scsi hostPath: path:
/mnt/disks EOF

If everything's gone according to plan, there should be as many Persistent Volumes as nodes in your cluster. 

kubectl get pv NAME CAPACITY ACCESS
MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv-4ba80b32 368Gi RWO Delete Available local-scsi 19s
local-pv-7cca15bd 368Gi RWO Delete Available local-scsi 19s
local-pv-dbce8b03 368Gi RWO Delete Available local-scsi 19s

In our last step we will create a Storage Class, which will handle all requests coming from Persistent Volume Claims. This class will be unique insofar as it will delay PVC binding until the Pod is scheduled, so it can choose the appropriate local storage. To achieve this, StorageClass must contains this field: volumeBindingMode: "WaitForFirstConsumer"

<code class="language-bash hljs">kubectl create -f - &lt;&lt;<span class="hljs-string">EOF
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata:
name: "local-scsi" provisioner:
"kubernetes.io/no-provisioner" volumeBindingMode:
"WaitForFirstConsumer" EOF</span>
</code>

curl -g --request PUT \
 --url
'{{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/helm/repos/banzaicloud-stable'
\
 --header 'Authorization: Bearer {{token}}' \
 --header 'Content-Type: application/json' \
 -d '{ "name": "banzaicloud-stable", "cache":
"statestore/<your-cluster>/helm/repository/cache/banzaicloud-stable-index.yaml",
"url":
"kubernetes-charts-incubator.banzaicloud.com/branch/kafka-local-volume",
"username": "", "password": "", "certFile": "", "keyFile":
"", "caFile": "" }' | jq

{ "status": 200, "message": "resource
modified successfully", "name": "banzaicloud-stable" }

curl -g --request PUT \
 --url
'{{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/helm/repos/banzaicloud-stable/update'
\
 --header 'Authorization: Bearer {{token}}' \
 --header 'Content-Type: application/json'

{ "status": 200, "message": "repository updated
successfully", "name": "banzaicloud-stable" }

curl -g --request POST \
 --url
'http://{{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/deployments'
\
 --header 'Authorization: Bearer {{token}}' \
 --header 'Content-Type: application/json' \
 -d '{"name": "banzaicloud-stable/kafka"}' | jq

{ "release_name": "kindly-macaw",
"notes": "" }

kubectl get pods NAME
READY STATUS RESTARTS AGE etcd-cluster-0000 1/1 Running 0 6m
etcd-cluster-0001 1/1 Running 0 6m etcd-cluster-0002 1/1
Running 0 6m kafka-0 1/1 Running 0 7m kafka-1 1/1 Running 0
6m kafka-2 1/1 Running 0 5m
kindly-macaw-etcd-operator-65bdcfc5d7-mqvfm 1/1 Running 0 7m
pipeline-traefik-7c47dc7bd7-hntqp 1/1 Running 0 8m