Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
PRODUCT
9 min read
Share
It's no news that for quite a while our Apache Kafka on Kubernetes take, Supertubes has been happily running inside an Istio-based service mesh, in both single or multi-cluster setups across hybrid clouds. While we have touched on several aspects of the advantages Istio gave us, this post's aim is to collect some of the issues, cornerstones and benefits.
We see the service mesh as a key component of every modern Cloud Native stack. To make this a reality, we are on a mission to make Istio simple to use and manage for everyone. We have built a product called Backyards (now Cisco Service Mesh Manager), the Banzai Cloud operationalized and automated service mesh, which makes setting up and operating an Istio-based mesh a cinch.
Check out Supertubes in action on your own clusters: Register for an evaluation version and run a simple install command! As you might know Cisco has recently acquired Banzai Cloud. Currently we are in a transitional period and are moving our infrastructure. Contact us so we can discuss your needs and requirements, and organize a live demo. Evaluation downloads are temporarily suspended. Contact us to discuss your needs and requirements, and organize a live demo.
supertubes install -a --no-demo-cluster --kubeconfig <path-to-k8s-cluster-kubeconfig-file>
or read the documentation for details. Take a look at some of the Kafka features that we've automated and simplified through Supertubes and the Koperator, which we've already blogged about:
- Oh no! Yet another Kafka operator for Kubernetes
- Monitor and operate Kafka based on Prometheus metrics
- Kafka rack awareness on Kubernetes
- Running Apache Kafka over Istio - benchmark
- User authenticated and access controlled clusters with [Koperator]
- Kafka rolling upgrade and dynamic configuration on Kubernetes
- Envoy protocol filter for Kafka, meshed
- Right-sizing Kafka clusters on Kubernetes
- Kafka disaster recovery on Kubernetes with CSI
- Kafka disaster recovery on Kubernetes using MirrorMaker2
- The benefits of integrating Apache Kafka with Istio
- Kafka ACLs on Kubernetes over Istio mTLS
- Declarative deployment of Apache Kafka on Kubernetes
- Bringing Kafka ACLs to Kubernetes the declarative way
- Kafka Schema Registry on Kubernetes the declarative way
- Announcing Supertubes 1.0, with Kafka Connect and dashboard
The internet is full of questions and problems reported by people struggling with running Istio and Kafka alongside each other. Most problems are related to communication or bootstrap, and they all come from one single source: the sidecar. One of the major problems is that a sidecar is not yet a first class citizen in Kubernetes. It's been coming for a while and was announced in the 1.18 release, however it's been pushed back to 1.19. To understand the problem and the solution in more detail please check out our post about sidecars: Sidecar container lifecycle changes in Kubernetes. And why is this causing any issues, you wonder? Well, the main reason is that Kafka and it's metadata store, Zookeeper are designed to have all the required resources available at startup time:
Note that Kafka and Zookeeper were designed for physical on-prem datacenters and while it works fairly well in the cloud, out of the box it is not ready to run on a dynamic environment as Kubernetes.
While the above are some of the most common runtime problems you might face, there are different new problems as well. Let's assume that Kafka on Istio is already working and all of the typical Kafka and Zookeeper communication failures are fixed. What is one of the first areas of interest you would focus on? Yes, Security.
There are well defined ways of handling security on Kafka (proprietary) and on Kubernetes, and these don’t match. Getting them to just work without rewriting Kafka broker clients, persisting the existing ACLs and translating/enforcing them as K8s RBAC is an extremely hard challenge. There are several benefits in using Istio’s built-in security mechanism (more details in the next paragraph), because:
But coming back to the original question: How should you handle the fine-grained Kafka ACL's while clients access brokers using client certificates, the whole mesh is secured with mTLS, and Envoy does the SSL termination? The new (Istio 1.5) Envoy Kafka protocol filter comes to our rescue. That, with a KafkaPrincipalBuilder provided by Supertubes makes the whole process transparent to broker clients, and users are bypassing Envoy (instead of Envoy sending back a PLAINTEXT anonymous principal).
Now let's go through the benefits. I am not going to list all of them as we’ve blogged about several benefits (check out the Supertubes posts).
Developers and operators do not have to worry about implementing security features, they can rely on the transparent security features brought in by the service mesh:
The features above are already built and provided by Supertubes but this is not all. While setting up a production-ready Apache Kafka cluster on Kubernetes becomes as simple as registering for an evaluation version and running a simple command to install the CLI tool, there is way more Istio can provide. Evaluation downloads are temporarily suspended. Contact us to discuss your needs and requirements, and organize a live demo.
And finally, Istio has introduced WebAssembly extensibility support and this brings a totally new option (and additional languages, other than C++) to write different filters for the chain.
Supertubes was designed to be a best-of-class implementation of Kafka on Kubernetes leveraging Cloud Native technologies. As such, we opted to integrate tightly with the Istio service mesh, which - among other things - brings a layer of security, manageability, along with performance benefits to Kafka. This is a particularly compelling package if you are a SaaS provider who wants to run Kafka "as a service" on your own Kubernetes infrastructure, on your own terms. Supertubes installs, configures, and manages all the components that are required for Kafka success on Kubernetes. If you plan to run Kafka on Kubernetes, and interested to learn more about Supertubes, check out the product page or read the documentation.
Banzai Cloud Supertubes (Supertubes) is the automation tool for setting up and operating production-ready Kafka clusters on Kubernetes, leveraging a Cloud-Native technology stack. Supertubes includes Zookeeper, the Banzai Cloud Kafka operator, Envoy, Istio and many other components that are installed, configured, and managed to operate a production-ready Kafka cluster on Kubernetes. Some of the key features are fine-grained broker configuration, scaling with rebalancing, graceful rolling upgrades, alert-based graceful scaling, monitoring, out-of-the-box mTLS with automatic certificate renewal, Kubernetes RBAC integration with Kafka ACLs, and multiple options for disaster recovery.
Get emerging insights on innovative technology straight to your inbox.
Discover how AI assistants can revolutionize your business, from automating routine tasks and improving employee productivity to delivering personalized customer experiences and bridging the AI skills gap.
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.