Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
PRODUCT
10 min read
Published on 04/02/2019
Last updated on 08/19/2024
Seamless Istio control plane upgrade
Share
UPDATE: For a newer Istio control plane upgrade method using the canary upgrade flow see the Safe and sound canary upgrade for your Istio control plane post.
Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.
When we added support for Istio's service mesh in the Pipeline platform, we experienced first hand how the deployment and management of Istio can become increasingly complex. We realized that we weren't the only ones managing Istio with Helm dealing with these problems - that demand was emerging for an Istio operator (e.g. #9333). We decided to build an Istio operator of our own, and more than a month ago we open sourced it. Our goal was to simplify the deployment and management of Istio for everyone. The first version of our operator was capable of installing and managing Istio 1.0.x versions. Then, as soon as Istio 1.1 became available, we updated our operator to support that version as well (we also currently support Istio 1.1.1).
Motivation
More than simply facing the challenges of Istio deployment and management, many users had additional trouble with upgrades (e.g. #12154, #12658, #12659, #12664, #12755, #12801, #12855 and #12866). With Helm, issues even arose while upgrading between minor versions. However, since the most recent major release of Istio, upgrading between major versions has been the larger challenge. As of now, manual upgrade is the recommended approach. Since our operator was able to deploy and manage both Istio 1.0.x and 1.1.x versions, we have been working on making upgrades easier and more seamless between these versions. Today, we released a version of our operator which greatly simplifies the control plane upgrade between Istio versions.
Control plane upgrade with Istio operator
Let us suppose that we have a Kubernetes cluster with Istio 1.0.5, and we would like to upgrade our Istio components to Istio version 1.1.1. Here are the steps we need to perform to accomplish this with our operator:
- Deploy a version of our operator which supports Istio 1.1.x
- Apply a Custom Resource using Istio 1.1.1 components
It really is that easy! What happens is that once the operator discerns that the Custom Resource it's watching has changed, it reconciles all Istio-related components in order to perform a control plane upgrade. 
The upgrade process described above is between major Istio versions. Upgrading between minor Istio versions is even easier. You only need to perform the second of those two steps, i.e. apply a Custom Resource with the desired Istio version set for Istio's components, and the operator will do the rest for you.
Try it out!
Requirements:
- Minikube v0.33.1+ or Kubernetes 1.10.0+
KUBECONFIGset to an existing Kubernetes cluster
I created a Kubernetes cluster on GKE for this demo with our Pipeline platform. If you wish you could also create one on any of the six cloud providers we support or on-premise using Pipeline for free.
If you already have Istio 1.0.x installed on your cluster you can skip the next section and can jump right to Deploy sample BookInfo application.
Install Istio 1.0.5
We install Istio with our operator, so first we need to check out the release-1.0 branch of our operator (this branch supports Istio versions before 1.1.0):
$ git clone git@github.com:banzaicloud/istio-operator.git
$ git checkout release-1.0Install Istio Operator with make Simply run the following make goal from the project root to install the operator:
$ make deployThis command will install a Custom Resource Definition in the cluster, and will deploy the operator to the istio-system namespace. As is typical of operators, this will allow you to specify your Istio configurations to a Kubernetes Custom Resource. Install Istio Operator with Helm Alternatively, if you just can't let go of Helm completely, you can deploy the operator using a Helm chart, which is available in the Banzai Cloud stable Helm repo:
$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
$ helm install --name=istio-operator --namespace=istio-system --set-string operator.image.tag=0.0.10 banzaicloud-stable/istio-operatorNote: As of now, the 0.0.10 tag is the latest version of our operator to support Istio versions 1.0.x. Apply the Custom Resource Once you've applied the Custom Resource to your cluster, the operator will start reconciling all of Istio's components. There are some sample Custom Resource configurations in the config/samples folder. To deploy Istio 1.0.5 with its default configuration options, use the following command:
$ kubectl apply -n istio-system -f config/samples/istio_v1beta1_istio.yamlAfter some time, you should see that the Istio pods are running:
$ kubectl get pods -n istio-system --watch
NAME READY STATUS RESTARTS AGE
istio-citadel-5b69bd4749-h24xk 1/1 Running 0 1m
istio-egressgateway-bb8b48cf8-w65hm 1/1 Running 0 1m
istio-galley-5ddd798686-jpdlm 1/1 Running 0 1m
istio-ingressgateway-678ff4cc87-gkdzt 1/1 Running 0 1m
istio-operator-controller-manager-0 2/2 Running 0 9m
istio-pilot-fc664fcbd-kgl2k 2/2 Running 0 1m
istio-policy-5cf55ff648-xfhf7 2/2 Running 0 1m
istio-sidecar-injector-596f8dddbb-gvzk9 1/1 Running 0 1m
istio-telemetry-7cbf75c5cf-wk4v8 2/2 Running 0 1mThe Istio Custom Resource is showing Available in its status field and the Istio components are using 1.0.5 images :
$ kubectl describe istio -n istio-system istio
Name: istio-sample
Namespace: istio-system
Labels: controller-tools.k8s.io=1.0
Annotations: kubectl.kubernetes.io/last-applied-configuration={<span class="hljs-string">"apiVersion"</span>:<span class="hljs-string">"istio.banzaicloud.io/v1beta1"</span>,<span class="hljs-string">"kind"</span>:<span class="hljs-string">"Istio"</span>,<span class="hljs-string">"metadata"</span>:{<span class="hljs-string">"annotations"</span>:{},<span class="hljs-string">"labels"</span>:{<span class="hljs-string">"controller-tools.k8s.io"</span>:<span class="hljs-string">"1.0"</span>},<span class="hljs-string">"name"</span>:<span class="hljs-string">"istio-sampl...
API Version: istio.banzaicloud.io/v1beta1
Kind: Istio
Metadata:
Creation Timestamp: 2019-03-31T10:07:22Z
Finalizers:
istio-operator.finializer.banzaicloud.io
Generation: 2
Resource Version: 13101
Self Link: /apis/istio.banzaicloud.io/v1beta1/namespaces/istio-system/istios/istio-sample
UID: c6a095da-539c-11e9-9080-42010a9a0136
Spec:
Auto Injection Namespaces:
default
Citadel:
Image: istio/citadel:1.0.5
Replica Count: 1
Galley:
Image: istio/galley:1.0.5
Replica Count: 1
Gateways:
Egress:
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Ingress:
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Include IP Ranges: *
Mixer:
Image: istio/mixer:1.0.5
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Mtls: false
Pilot:
Image: istio/pilot:1.0.5
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Trace Sampling: 1
Proxy:
Image: istio/proxyv2:1.0.5
Sidecar Injector:
Image: istio/sidecar_injector:1.0.5
Replica Count: 1
Tracing:
Zipkin:
Address: zipkin.jaeger-system:9411
Status:
Error Message:
Status: Available
Events: <none>
</span>Deploy sample BookInfo application
Let's make sure that Istio 1.0.5 is properly installed with Istio's BookInfo application:
$ kubectl -n default apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml
service "details" created
deployment.extensions "details-v1" created
service "ratings" created
deployment.extensions "ratings-v1" created
service "reviews" created
deployment.extensions "reviews-v1" created
deployment.extensions "reviews-v2" created
deployment.extensions "reviews-v3" created
service "productpage" created
deployment.extensions "productpage-v1" created
$ kubectl -n default apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io "bookinfo-gateway" created
virtualservice.networking.istio.io "bookinfo" createdDetermine the external hostname of the ingress gateway and open productpage in a browser:
$ INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ open http://$INGRESS_HOST/productpageInstall Istio 1.1.1
To install Istio 1.1.1, first we need to check out the release-1.1 branch of our operator (this branch supports the Istio 1.1.x versions):
$ git clone git@github.com:banzaicloud/istio-operator.git
$ git checkout release-1.1If you installed Istio operator with
makein the previous section go to toInstall Istio Operator with make, if you installed it withhelmgo toInstall Istio Operator with helm. If you haven't installed Istio operator so far you can choose whichever install option you like.
Install Istio Operator with make Simply run the following make goal from the project root to install the operator:
$ make deployThis command will install a Custom Resource Definition in the cluster, and will deploy the operator to the istio-system namespace. Install Istio Operator with Helm Alternatively, you can deploy the operator using a Helm chart, which is available in the Banzai Cloud stable Helm repo:
$ kubectl apply -n istio-system -f config/samples/istio_v1beta1_istio.yamlNote: As of now, the 0.1.4 tag is the latest version of our operator to support Istio versions 1.1.x. Apply the new Custom Resource
If you've installed Istio 1.0.5 or earlier with the Istio operator, and if you check the logs of the operator pod at this point, you will see the following error message:
intended Istio version is unsupported by this version of the operator. We need to update the Istio Custom Resource with Istio 1.1's components for the operator to be reconciled with the Istio control plane.
To deploy Istio 1.1.1 with its default configuration options, use the following command:
$ kubectl apply -n istio-system -f config/samples/istio_v1beta1_istio.yamlAfter some time, you should see that new Istio pods are running:
$ kubectl get pods -n istio-system --watch
NAME READY STATUS RESTARTS AGE
istio-citadel-7664c58768-l8zgb 1/1 Running 0 7m
istio-egressgateway-8588c7c8d-wkpgk 1/1 Running 0 7m
istio-galley-78b8467b4d-b5dqs 1/1 Running 0 7m
istio-ingressgateway-5c48b96cb4-lnfsn 1/1 Running 0 7m
istio-operator-controller-manager-0 2/2 Running 0 16m
istio-pilot-84588fff4c-4lhq8 2/2 Running 0 7m
istio-policy-75f84689f5-78dxr 2/2 Running 0 7m
istio-sidecar-injector-66cd99d8c8-bp4j7 1/1 Running 0 7m
istio-telemetry-7b667c5fbb-2lfdc 2/2 Running 0 7mThe Istio Custom Resource is showing Available in its status field, and the Istio components are now using 1.1.1 images:
$ kubectl describe istio -n istio-system istio
Name: istio-sample
Namespace: istio-system
Labels: controller-tools.k8s.io=1.0
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"istio.banzaicloud.io/v1beta1","kind":"Istio","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":"istio-sampl...
API Version: istio.banzaicloud.io/v1beta1
Kind: Istio
Metadata:
Creation Timestamp: 2019-03-31T10:07:22Z
Finalizers:
istio-operator.finializer.banzaicloud.io
Generation: 3
Resource Version: 21904
Self Link: /apis/istio.banzaicloud.io/v1beta1/namespaces/istio-system/istios/istio-sample
UID: c6a095da-539c-11e9-9080-42010a9a0136
Spec:
Auto Injection Namespaces:
default
Citadel:
Image: docker.io/istio/citadel:1.1.1
Replica Count: 1
Control Plane Security Enabled: false
Default Pod Disruption Budget:
Enabled: true
Exclude IP Ranges:
Galley:
Image: docker.io/istio/galley:1.1.1
Replica Count: 1
Gateways:
Egress:
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Service Annotations:
Service Labels:
Service Type: ClusterIP
Ingress:
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Service Annotations:
Service Labels:
Service Type: LoadBalancer
K8s ingress:
Enabled: false
Include IP Ranges: *
Mixer:
Image: docker.io/istio/mixer:1.1.1
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Mtls: false
Node Agent:
Enabled: false
Image: docker.io/istio/node-agent-k8s:1.1.1
Outbound Traffic Policy:
Mode: ALLOW_ANY
Pilot:
Image: docker.io/istio/pilot:1.1.1
Max Replicas: 5
Min Replicas: 1
Replica Count: 1
Trace Sampling: 1
Proxy:
Enable Core Dump: false
Image: docker.io/istio/proxyv2:1.1.1
Proxy Init:
Image: docker.io/istio/proxy_init:1.1.
Sds:
Enabled: false
Sidecar Injector:
Image: docker.io/istio/sidecar_injector:1.1.1
Replica Count: 1
Rewrite App HTTP Probe: true
Tracing:
Zipkin:
Address: zipkin.istio-system:9411
Version: 1.1.1
Status:
Error Message:
Status: Available
Events: <none>At this point, your Istio control plane is upgraded to Istio 1.1.1 and your BookInfo application should still be available at:
$ open http://$INGRESS_HOST/productpageTakeaway
Upgrading the Istio control plane between Istio's major versions with our operator is as easy as deploying a new version of the operator, and then applying a new Custom Resource using the desired component versions.
Roadmap
1. Sidecar upgrades In order to change sidecars running older versions of the Istio proxy we need to perform a few manual steps. We're already working on providing a seamless upgrade path through the operator for sidecars. 2. In-place upgrades An in-place upgrade is impossible right now due to traffic disruptions. It is our plan to support this use-case as well, in the future.
Istio operator - contributing and development
If you're interested in this project, we happily accept contributions. You can support us by starring the repo, or if you want to help with development, feel free to read our contribution and development guidelines from our first blog post about the operator.
About Banzai Cloud Pipeline
Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.
Subscribe to
the Shift!
Get emerging insights on innovative technology straight to your inbox.
