Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
PRODUCT
7 min read
Published on 07/26/2020
Last updated on 08/19/2024
Certificate management on Istio
Share
Recently, we blogged about certificate management on Kubernetes. Today, we'll be returning to that topic, but we'll be focusing on the differences an Istio service mesh makes. The primary difference is the method of solving the ACME HTTP-01 challenge. Solving this challenge involves routing an HTTP request from the ACME server (the Certificate Authority) to the cert-manager challenge solver pod.
Cert management with Istio Ingress support
As we saw in our previous blog post, you can route such a challenge request by using a Kubernetes Ingress gateway. When using Istio, that same method is still viable, provided that Kubernetes Ingress support is enabled in your Istio config. If it's enabled, and you're using Kubernetes Ingress resources for routing the traffic entering your cluster, everything should work just fine. However, you will be limited to using one ingress gateway for all your ingress traffic. This can be a security issue, and it also prevents you from setting up separate ingress gateways for your different teams or products. Additionally, this will cause you to miss out on some Istio features. Considering these limitations, it usually makes sense to use native Istio routing.
Cert management using Istio
Using native Istio features for ingress gives you more freedom and flexibility:
- you can direct traffic for different domains to separate ingress gateways
- you can set up routing for HTTP and HTTPS traffic on non-standard ports
- you also get to use the familiar Istio features (circuit breaker, metrics, etc.)
While technically it's possible to use Kubernetes Ingress and Istio resources together to configure the same Istio ingress gateway, conflicts between them can easily cause difficult to diagnose problems; you should stick to using one or the other for a given ingress gateway. All in all, if you have Istio running in your cluster, it's a good idea to use it for your ingress gateways, too. So, what does this mean in terms of cert management automation? Well, cert-manager doesn't support Istio at the moment. There are solutions, though. When using an ACME issuer with challenge type HTTP-01, you will end up with an Ingress resource for routing the ACME challenge request. If you disabled Kubernetes Ingress support in Istio (or you specified a non-existent ingress class in the Issuer resource), then no ingress controller will react to the Ingress resource, therefore, no HTTP request will be routed to the ACME solver pod(s). What can we do? Two possible solutions: translate the Ingress resource into Gateway and VirtualService resources, or add Istio support to cert-manager. Obviously, adding Istio support to cert-manager would be the perfect solution, but it's a more involved process. Nonetheless, at Banzai Cloud, we're already working on just such an implementation. In fact, we have a functional solution, although it needs some more love before it's production ready (cert-manager#3011).
Translating Ingress resources
While we're waiting for direct Istio support in cert-manager to be released, we can make do with another solution: replicating the Ingress resource with Istio resources. It's possible to do this by hand, but usually the reason we use the ACME protocol is automation, so introducing an error prone and repetitive manual step into that process sort of defeats the point. When properly automated, this can serve as an effective solution, though. The good news is, if you're a Backyards (now Cisco Service Mesh Manager) user, you can try it out right now, because Backyards 1.3 includes this feature. If not, then feel free to use the evaluation version!
Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.
Try it out!
open the Backyards dashboard:
$ backyards dashboard- select
Gatewaysfrom the menu on the left - select the gateway you want secured. You can select whichever you like, but make sure the "Service Type" is "LoadBalancer". This selection determines the load balancer and, therefore, the IP address(es) that will be used for the ACME
HTTP-01challenge. In our case, it'sistio-ingressgateway.
- point your domain name to the IP address or DNS name found in the
ADDRESSfield. - in the
Ports & Hostssection, click oncreate newin the upper right corner - fill in the fields:
- you will probably want HTTPS protocol on the standard port
- enter your domain name(s) for
hosts. You can enter several by hitting enter after each one.You can also set up a domain under.banzaicloud.iofor testing purposes if you don't want to use your own domain name. Just click on the checkbox to the right ofuse .banzaicloud.ioand a new domain name will be generated for you, then add a subdomain under the generated domain name. - click on the checkbox
Use Let's Encrypt for TLSto get a certificate for your domain from Let's Encrypt - enter your email address. This address will be for ACME account management, so will be sent to Let's Encrypt.
- finally, click
Create
As you can see, two more items have appeared in the Ports & Hosts list, both for the host frontpage.qahqab.backyards.banzaicloud.io. One of them on port 443, and the other on port 80. The latter one is for solving the ACME HTTP-01 challenge created by translating the Ingress resource. 
While the certificate is being acquired, you can look up and see the contents of related Ingress, Gateway and VirtualService resources:
The
Ingressresource created by cert-manager:apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: istio labels: acme.cert-manager.io/http-domain: "1270878147" acme.cert-manager.io/http-token: "1101532839" acme.cert-manager.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2 namespace: istio-system spec: rules: - host: frontpage.qahqab.backyards.banzaicloud.io http: paths: - backend: serviceName: cm-acme-http-solver-rrg8c servicePort: 8089 path: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45sThe
GatewayandVirtualServicecreated by Backyards from the aboveIngressresource:apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: labels: acme.backyards.banzaicloud.io/http-domain: "1270878147" acme.backyards.banzaicloud.io/http-token: "1101532839" acme.backyards.banzaicloud.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2-czvnv namespace: istio-system spec: selector: app: istio-ingressgateway istio: ingressgateway servers: - hosts: - istio-system/frontpage.qahqab.backyards.banzaicloud.io port: name: http number: 80 protocol: HTTPapiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: labels: acme.backyards.banzaicloud.io/http-domain: "1270878147" acme.backyards.banzaicloud.io/http-token: "1101532839" acme.backyards.banzaicloud.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2-tv22w namespace: istio-system spec: exportTo: - "*" gateways: - istio-system/cm-acme-http-solver-nnpg2-czvnv hosts: - frontpage.qahqab.backyards.banzaicloud.io http: - match: - uri: exact: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s route: - destination: host: cm-acme-http-solver-rrg8c.istio-system.svc.cluster.local port: number: 8089
As you can see, these two resources describe the same routing path as the Ingress resource. After a short while the item with port 80 and protocol HTTP will disappear, and a green checkmark will appear next to HTTPS. This signals that the certificate was issued and is being used for securing your domain: 
Let's see if that's really the case:
$ curl -v https://frontpage.qahqab.backyards.banzaicloud.io
...
* Server certificate:
* subject: CN=frontpage.qahqab.backyards.banzaicloud.io
* start date: Jul 14 10:42:43 2020 GMT
* expire date: Oct 12 10:42:43 2020 GMT
* subjectAltName: host "frontpage.qahqab.backyards.banzaicloud.io" matched cert's "frontpage.qahqab.backyards.banzaicloud.io"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
...
< HTTP/2 404
< date: Tue, 14 Jul 2020 11:44:09 GMT
< server: istio-envoyAs you can see, the connection is secured with a certificate from Let's Encrypt. Did you notice that 404, though? That's because we haven't set up any routing yet. We can do that now, so head over to the routes tab, click create new in the upper right corner and fill in the fields. At the very least, the gateway, host and port number need to be filled in, but, depending on your requirements, you might want to specify other values and settings, too. 
When you're ready, click Create. Now that routing is set up for our demo service, let's see if it works:
$ curl https://frontpage.qahqab.backyards.banzaicloud.io
frontpageYou can also try it in your browser to inspect the certificate more easily.
Wrap up
As we have seen in this blog post, while cert-manager doesn't directly support Istio, it's possible to make it work. And with Backyards, it's not only possible but seamless, too.
Subscribe to
the Shift!
Get emerging insights on innovative technology straight to your inbox.
