Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
INSIGHTS
9 min read
Published on 07/19/2020
Last updated on 06/18/2024
Certificate management on Kubernetes: How to use the ACME protocol
Share
When exposing services it's generally a good idea to follow the industry standard and use HTTPS protocol. HTTPS requires a certificate issued by a trusted third party, called a Certificate Authority (or CA for short). There are several ways to acquire one, but a simple and effective method is to use Let's Encrypt (a CA) by way of the ACME protocol.
The ACME protocol is a communication protocol for interacting with CAs that makes it possible to automate the request and issuance of certificates. The idea is that manual certificate management can easily result in expired certificates, which usually translate to a non-working website and/or services. For this reason, you should be able to configure your infrastructure once and let it handle certificate renewals automatically.
To ensure that you automate this process, certificates issued by Let's Encrypt are valid for only 90 days, since it's widely believed that 90 days is frequent enough for users to automate the handling of renewals. Let’s further explore Kubernetes certificate management—and what you can run to make the process easier on yourself.
Kubernetes certificate management: Using the ACME protocol
What you need to know about the ACME protocol is that it involves proving that you control the domains present in the Certificate Signing Request (CSR). This is done by solving challenges (one for each domain). You can usually choose between several challenge types, which vary depending on the CA and the domains involved. The two most common challenge types are HTTP-01 and DNS-01.
HTTP-01 challenge
This challenge type is solved by replying to a specific HTTP request with an appropriate response. The request is a GET request to a url of the form http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> and the response is a combination of the token and your ACME account key. This is probably the simplest challenge to automate, but it can't be used for wildcard domains (for example *.example.com). For wildcard domains you have to solve the DNS-01 challenge.
DNS-01 challenge
Solving the DNS-01 challenge can be done by putting a specific value in a TXT record for the given domain. As the main idea behind the ACME protocol is automation, this challenge type only makes sense if your DNS provider has an API. There are several ACME clients which can handle the submitting of CSRs as well as solving the required challenges. One such client is certbot which can handle "legacy" environments (Apache, Nginx, etc.). If you are running your services in a Kubernetes cluster, your best bet is to use cert-manager. Let's take a look at how it works.
cert-manager
Cert-manager is the complete package when it comes to handling multiple certificate issuer types (ACME, self-signed, CA among others). It can acquire and automatically renew certificates before expiry. If you are using Kubernetes Ingress to route your ingress traffic, cert-manager can automatically solve HTTP-01 challenges. It does this by spinning up a pod for each challenge and then applying the necessary routing changes to your Ingress config.
When you interact with cert-manager you'll be using a couple of custom resource types: Issuer, ClusterIssuer, and Certificate. There are a few other custom resource types involved, which aren't necessary to know about but which can be helpful in tracking down errors. These are the CertificateRequest, Order and Challenge custom resources. Let's see what each one is used for. 
User-facing custom resources
Issuer, ClusterIssuer resources
An issuer is an entity that can generate signed certificates. There are several supported issuers built into cert-manager, and it can be extended with new ones if necessary. An Issuer or ClusterIssuer resource describes one issuer entity. You will need at least one such resource in your cluster. We will be focusing on the ACME Issuer type. ACME is the protocol implemented by Let's Encrypt. The difference between Issuer and ClusterIssuer is that the Issuer's use is restricted to the namespace it's created in; it's not possible to reference it from a Certificate resource in another namespace. A ClusterIssuer, however, is global, usable from any Certificate resource in the cluster.
Certificate resource
A Certificate resource describes the intention to acquire a certificate for one or more of your domains. Each certificate must reference an issuer resource. This will be the issuer used for acquiring the desired certificate.
When cert-manager notices a new Certificate resource it proceeds with the creation of a CertificateRequest.
Internal custom resources
CertificateRequest resource
This resource represents one request for a certificate. It contains the certificate signing request, which is itself encoded in PEM format, and the certificate if it was already received, in which case Ready will be set to True.
The absence of this resource doesn't mean the certificate hasn't yet been requested. Cert-manager will not recreate this resource if the certificate has already been acquired.
Order resource
This resource type comes into play when a certificate is requested from an ACME issuer. An Order resource represents the order for a certificate to be issued.
In its status you can find the challenges to be solved. There can be several challenge types for a domain, but only one of those need be solved. Cert-manager will select one challenge type for each of the domains, then proceed with the creation of Challenge resources.
Challenge resource
Not unlike the Order resource type, this resource type plays a role when the certificate is requested from an ACME issuer. These resources describe the challenges cert-manager selected to solve.
Now that we've covered the parts of cert-manager, let's see it in action!
Try it out with a Kubernetes cluster!
You will need a Kubernetes cluster and domain to play with.
Create cluster
Create a Kubernetes cluster.
If you need a hand with that, you can create a cluster with the Banzai Clouds Pipeline platform on five different clouds or on-premise. Pipeline is available online, for free at Try Pipeline.
- Point
KUBECONFIGat your cluster. Make sure you have an
Ingresscontroller in your cluster. If you used Banzai Cloud's Pipeline platform to create the cluster, you already have one along with aLoadBalancer-typeService:$ kubectl get -n pipeline-system deploy/ingress-traefik svc/ingress-traefik NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-traefik 1/1 1 1 3m19s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-traefik LoadBalancer 10.10.231.18 a015671f691794291bd991eb635ed907-1982155256.us-east-2.elb.amazonaws.com 443:32271/TCP,80:30543/TCP 3m20sThe service should have port 80 and 443 open. Port 80 is for the ACME
HTTP-01challenge, and port 443 is for HTTPS traffic.Install cert-managerIt can be as simple as:
kubectl apply -f https://github.com/jetstack/cert- manager/releases/download/v0.15.1/cert-manager.yamlThere are other options and more details at https://cert-manager.io/docs/installation/kubernetes/
Point your domain to the external IP of your
LoadBalancertypedServiceon which you want external traffic to enter your cluster. "External IP" might be a DNS name (as it is in the example above), in which case you will have to create aCNAMErecord instead of anArecord.Set theDOMAINenvironment variable to your domain (example.comwill not work):$ DOMAIN=example.comDeploy httpbin for testing purposes
$ kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: httpbin namespace: default spec: selector: app: httpbin ports: - port: 8080 protocol: TCP targetPort: 80 --- apiVersion: v1 kind: Pod metadata: name: httpbin namespace: default labels: app: httpbin spec: containers: - image: kennethreitz/httpbin:latest name: httpbin ports: - containerPort: 80 protocol: TCP EOFYou can check if it's up and running through port-forwarding and curl:
$ kubectl port-forward svc/httpbin 8080 & $ curl localhost:8080/getCreate an issuer
Note: we will be using the staging provider to avoid hitting rate limits.
First, set the
EMAILenvironment variable to your email address. Let's Encrypt will use this to contact you about expiring certificates and other issues related to your account.$ EMAIL=some@email.addressNow, you're ready to create the issuer for Let's Encrypt:
$ kubectl apply -f - <<EOF apiVersion: cert-manager.io/v1alpha2 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: email: ${EMAIL} server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: # Secret resource that will be used to store the account's private key. name: example-issuer-account-key solvers: - http01: ingress: name: test-ingress EOFCreate an ingress
$ kubectl apply -f - <<EOF apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: test-ingress namespace: default spec: backend: serviceName: httpbin servicePort: 8080 tls: - hosts: - ${DOMAIN} secretName: test-ingress-cert EOFThis ingress is not functional yet, because the secret being referenced does not exist.
If you remove the tls config, the service should be reachable on HTTP protocol:
$ curl ${DOMAIN}/getIf it's not working, try the external IP of your ingress service to see if the problem is in your cluster or with the DNS resolution. If it works with the external IP, you might need to wait a couple of minutes for the DNS changes to propagate.
Acquire a certificateAs I have mentioned, we have a non-working ingress because of the missing secret. We have a couple options to acquire a certificate and make it work:
- Create the ingress without tls config. This way, the ingress can serve HTTP traffic to complete the ACME
HTTP-01challenge. After the certificate is issued, thetlsconfig can be added back in. - Create a self-signed certificate initially, and then acquire a properly signed one:
- Create a self-signed
Issuerand use it in yourCertificateresource - Wait until the certificate is ready (check it using
curl --insecure https://${DOMAIN}/get) - Edit your
Certificateresource and change the issuer to the Let's EncryptIssuer - The self-signed certificate will then be replaced by a newly acquired, properly signed certificate
- Create a self-signed
- The easiest solution: annotate the
Certificateresource to instruct cert-manager to create a self signed certificate before attempting to request the properly signed one. This is basically the same as method 2, but automated.
Let's go with method 3 and annotate the
Certificateresource withcert-manager.io/issue-temporary-certificate: "true".$ kubectl apply -f - <<EOF apiVersion: cert-manager.io/v1alpha2 kind: Certificate metadata: name: test-ingress namespace: default labels: cert: test-ingress annotations: cert-manager.io/issue-temporary-certificate: "true" spec: secretName: test-ingress-cert dnsNames: - ${DOMAIN} issuerRef: name: letsencrypt-staging kind: ClusterIssuer EOFIf everything is set up correctly, you should receive a certificate in a few minutes. You can check its progress by inspecting the
Certificate,CertificateRequest,OrderandChallengeresources. Take a look at the status and events sections.$ kubectl describe certificate -l cert=test-ingress $ kubectl describe certificaterequest -l cert=test-ingress $ kubectl describe order -l cert=test-ingress $ kubectl describe challengeNote: as of cert-manager v0.15.1,
Challengeresources don't inherit the labels the way other resources do, so it's not as easy to select the challenges that correspond to yourCertificateresource.Try adding one or more entries (for example foo.< your-domain>) to
dnsNamesin yourCertificateresource, which you have not pointed to your ingress. Challenges for these domains won't be solvable, so you'll have time to inspect theChallengeresources. You can also see the changes it made to yourIngressresource.The received certificate will be placed in the secret with key
tls.crtand the corresponding private key with keytls.key. You can check it with the following command:$ kubectl describe secret test-ingress-certNow we are ready to see the issued certificate in action (the
--insecureflag is necessary when using the staging provider, which we did):$ curl -v --insecure https://${DOMAIN}/get ... * Server certificate: * subject: CN=<your-domain> * start date: Jul 3 19:57:58 2020 GMT * expire date: Oct 1 19:57:58 2020 GMT * issuer: CN=Fake LE Intermediate X1 ...Keeping cert-manager and your
Certificateresources in your cluster will ensure that all your certificates are renewed before expiry.- Create the ingress without tls config. This way, the ingress can serve HTTP traffic to complete the ACME
Wrap-up: Certificate management on Kubernetes
It's important to secure your services with TLS; cert-manager makes it easy to do that in a Kubernetes environment. You should use it! If you are interested in certificate management on Istio, check out our next post, Certificate management on Istio as well.
Subscribe to
the Shift!
Get emerging insights on innovative technology straight to your inbox.
