Outshift Logo


7 min read

Blog thumbnail
Published on 03/05/2019
Last updated on 05/03/2024

Automate single mesh multi-clusters with the Istio operator


Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.
Last week we released our open source Istio operator designed to help ease the sometimes difficult task of managing Istio. One of the main feature of the operator is its ability to manage a single mesh multi-cluster Istio.

Multi cluster scenarios

Typical multi-cluster-based patterns are single mesh - combining multiple clusters into one unit managed by one Istio control plane - and mesh federation, wherein multiple clusters act as individual management domains and the service exposure between those domains is done selectively.

Single mesh multi-cluster

The single mesh scenario is most suitable in those use cases where clusters are configured together, sharing resources and typically being treated as one infrastructural component within an organization. Because the first release of the Istio operator supports Istio 1.0, our single mesh multi-cluster implementation is based on the features that version provides. It allows multiple clusters to be joined into a mesh, but all those clusters must be on a single shared network, all pod CIDRs must be unique and routable to each other in every cluster, and all API servers must also be routable to each other.
These requirements will be lowered with Istio 1.1
A single mesh multi-cluster is formed by enabling any number of Kubernetes control planes running a remote Istio configuration to connect to a single Istio control plane. Once one or more Kubernetes clusters is connected to the Istio control plane in that way, Envoy communicates with the Istio control plane in order to form a mesh network across those clusters.
Read on the roadmap section towards the end of the blog to learn about single mesh multi-cluster with lesser networking constraints and mesh federation for multi-clusters
Single mesh multi-cluster

How the Istio operator works

The Istio control plane must get information about service and pod states from remote clusters. To achieve this, the kubeconfig of each remote cluster must be added to the cluster where the control plane is running in the form of a k8s secret. The Istio operator uses a CRD called RemoteIstio to store the desired state of a given remote Istio configuration. Adding a new remote cluster to the mesh is as simple as defining a RemoteIstio resource with the same name as the secret which contains the kubeconfig for the cluster. The operator handles deploying Istio components to clusters and implements a sync mechanism, which provides constant reachability to the Istio central components in remote clusters. DNS entries are automatically updated whenever a pod restarts or any other failure occurs in the Istio control plane.

Try it out

For demo purposes, let's say there are two GKE clusters with flat networking, so their networking prerequisites are met. Let's also suppose there exists a running Istio control plane on the central cluster, deployed by our Istio operator.
❯ gcloud container clusters list NAME LOCATION MASTER_VERSION NODE_VERSION NUM_NODES STATUS k8s-central europe-west1-b 1.11.7-gke.4 1.11.7-gke.4 1 RUNNING k8s-remote-1 us-central1-a 1.11.7-gke.4 1.11.7-gke.4 1 RUNNING
There are some prerequisites each remote cluster must meet before it can be joined into the mesh.

Remote cluster prerequisites

Create istio-system namespace

[context:remote-cluster] ❯ kubectl create namespace istio-system
namespace/istio-system created

Create service account, cluster role and cluster role binding

[context:remote-cluster] ❯ kubectl create -f https://raw.githubusercontent.com/banzaicloud/istio-operator/master/docs/federation/example/rbac.yml
serviceaccount/istio-operator created
clusterrole.rbac.authorization.k8s.io/istio-operator created
clusterrolebinding.rbac.authorization.k8s.io/istio-operator created

Generate kubeconfig for the created service account

[context:remote-cluster] ❯ REMOTE_KUBECONFIG_FILE=$(https://raw.githubusercontent.com/banzaicloud/istio-operator/master/docs/federation/example/generate-kubeconfig.sh | bash)

Add kubeconfig for the remote cluster as a secret to the central cluster

[context:central-cluster] ❯ kubectl create secret generic remoteistio-sample --from-file ${REMOTE_KUBECONFIG_FILE} -n istio-system

Add remote cluster to the mesh

To add a remote cluster to the mesh, all you have to do is create a RemoteIstio resource on the central cluster where the operator resides. Once you've done this, the operator will begin the process of reconciling your changes, deploying the necessary components to the remote cluster and adding that cluster to the mesh.
The name of the secret that contains the kubeconfig must be the same as the name of the RemoteIstio resource
[context:central-cluster] ❯ cat <<EOF
| kubectl apply -n istio-system -f - apiVersion:
istio.banzaicloud.io/v1beta1 kind: RemoteIstio metadata:
labels: controller-tools.k8s.io: "1.0" name:
remoteistio-sample spec: autoInjectionNamespaces:

- "default" includeIPRanges: "\*" excludeIPRanges: ""
- name: "istio-pilot" labelSelector: "istio=pilot"
- name: "istio-policy" labelSelector:
- name: "istio-statsd" labelSelector: "statsd-prom-bridge"
- name: "istio-telemetry" labelSelector:
- name: "zipkin" labelSelector: "app=jaeger"
  controlPlaneSecurityEnabled: false citadel: replicaCount:
  1 sidecarInjector: replicaCount: 1 EOF
Here, the operator detects the creation of resources and begins the reconciliation process, creating the necessary Istio remote resources.
[context:central-cluster] ❯ kubectl logs -f istio-operator-controller-manager-0 -c manager
{"msg":"remoteconfig status updated","status":"Created"}
{"msg":"remoteconfig status updated","status":"Reconciling"}
{"msg":"reconciling remote istio","clusterName":"remote-cluster"}
{"msg":"remoteconfig status updated","status":"Available"}
{"msg":"remote istio reconciled","cluster":"remote-cluster"}

[context:remote-cluster]❯ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
istio-citadel-6cb84f4c47-86jp6            1/1     Running   0          2m
istio-sidecar-injector-68fdf88c87-69pr4   1/1     Running   0          2m
The operator keeps the DNS entries of Istio services in sync by managing services without selectors and corresponding endpoints on the remote cluster that hold the pod IPs of Istio services. These endpoints are automatically updated by the operator upon any failure or pod restart.
[context:remote-cluster] ❯ kubectl get svc -n istio-system
NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
istio-citadel            ClusterIP    <none>        8060/TCP,9093/TCP   2m
istio-pilot              ClusterIP   None           <none>        <none>              2m
istio-policy             ClusterIP   None           <none>        <none>              2m
istio-sidecar-injector   ClusterIP   <none>        443/TCP             2m
istio-statsd             ClusterIP   None           <none>        <none>              2m
istio-telemetry          ClusterIP   None           <none>        <none>              2m
zipkin                   ClusterIP   None           <none>        <none>              2m
❯ kubectl get endpoints
NAME                     ENDPOINTS                         AGE
istio-citadel  ,   2m
istio-pilot                      2m
istio-policy                     2m
istio-sidecar-injector                    2m
istio-telemetry                  2m
zipkin                            2m
Let's see what happens when the telemetry pod is removed from the central cluster.
[context:central-cluster] ❯ kubectl delete pods/istio-telemetry-77476d58c7-pmnnq
pod "istio-telemetry-77476d58c7-pmnnq" deleted

[context:central-cluster] ❯ kubectl logs -f istio-operator-controller-manager-0 -c manager
{"msg":"pod event detected","podName":"istio-telemetry-77476d58c7-pmnnq","podIP":"","podStatus":"Running"}
{"msg":"updating endpoints","cluster":"remote-cluster"}

[context:remote cluster] ❯ kubectl get endpoints/istio-telemetry
NAME              ENDPOINTS          AGE
istio-telemetry   4m
The operator detected that the pod was deleted and propagated the change to the remote cluster and updated the service endpoint to the current IP. Setting the autoInjectionNamespaces configuration in the RemoteIstio spec will give us automatic namespace labeling for namespaces on the remote side that need auto-sidecar injection.
[context:remote-cluster] ❯ kubectl get namespaces/default --show-labels
default   Active   1h    istio-injection=enabled,istio-operator-managed-injection=enabled
The complete example is available here.


As was mentioned at the beginning of this post, there are several multi-cluster scenarios. Our intention is that our operator should support those scenarios, as well as several mixed cluster scenarios.

1. Single mesh multi-cluster with lesser networking constraints

Istio 1.1 is going to introduce and implement Split Horizon EDS and SNI aware routing. Once implemented, it will no longer be necessary for a single mesh Istio setup to have a flat network. We plan to add that feature after re-releasing our operator with Istio 1.1 support.

2. Mesh federation multi-cluster

Our operator – with upcoming Istio 1.1 support – will also provide support for full mesh federation multi-clusters by connecting two or more k8s clusters running Istio control planes together via their Istio gateways. The alternative, setting up a mesh federation and simultaneously managing inter-connecting services between clusters without an operator, would require a significant amount of manual work.

About Banzai Cloud Pipeline

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.
Subscribe card background
Subscribe to
the Shift!

Get emerging insights on emerging technology straight to your inbox.

Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach

Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.

the Shift
emerging insights
on emerging technology straight to your inbox.

The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.

Outshift Background