SPIFFE/SPIRE Federation on Kind clusters

Spiffe.io is a universal identity control plane for distributed systems. SPIFFE is a set of standards to help us achieve zero trust identity framework and SPIRE is a reference implementation of SPIFFE. SPIRE can authenticate and authorize workloads in a distributed system. Essentially SPIRE maps workloads to SPIFFE identities and distributes credentials for secure micro-services communication between Kubernetes workloads. SPIRE servers can be deployed in different architectures. SPIRE Federation (Federating SPIRE Servers with Different Trust Domains) and Nested SPIRE (Chaining SPIRE Servers to Use the Same Trust Domain). SPIFFE/SPIRE Architectures In this blog, we will demonstrate how to authenticate two SPIFFE-identified workloads that are identified by two different SPIRE Servers. The first part of this tutorial demonstrates how to set up SPIRE federation by showing the SPIRE configuration changes and spire-server commands used to set up a stock quote webapp frontend and service backend. The second part of this document lists the steps to show the scenario in action using Helm charts and kubectl commands. In this demonstration you will learn to:

• Configure SPIRE Server to expose its SPIFFE Federation bundle endpoint using SPIFFE authentication.
• Configure SPIRE Servers to exchange trust bundles from each other.
• Create one-time initial Bootstrap federation bundles between two SPIRE Servers using different trust domains.
• Create registration entries for the workloads so that they can federate with other trust domain.

To demonstrate SPIRE Federation, we will set up two Kubernetes clusters using Kind. Then deploy SPIRE server-agent workloads using SPIRE Helm chart provided in the example for distributing SPIFFE ID and trust credentials. Next, we will deploy MetalLB using the MetalLB Helm cart provided in the example to auto-assign external load balancer IP addresses to our clusters. Lastly, we will configure SPIRE servers to federate with its neighboring cluster using the cluster's external IP. The stock quote webapp frontend and service backend we deploy in our Kubernetes cluster leverages the SPIFFE workload API to get mTLS credentials. In the first cluster server app servers client endpoint with stocks values to the client, server fetches x509 certificate from SPIRE server and sets up a listening port for the client. Client app in the second cluster with SPIRE workloads fetches TLS bundles from all the trusted domains and selects the right certificate to connect to the server. The server-client application is built to use grpc.dial from gRPC library with spiffe-api the API provides functionality to use x509 trust bundle for setting up gRPC channel with its targets. A port forwarded to your localhost on port 8080 can be used to access the client-server stockbroker webpage and initiate a mTLS request from client to server, which can be verified from the client-server pods logs.

SPIRE Federation using SPIFFE Authentication

To demonstrate SPIRE Federation, we'll use a simple example from the official SPIRE user guide for the Federation model. Source available here spire-tutorials/docker-compose/federation I have modified the server-client example that can be deployed in the Kubernetes cluster with SPIRE server-agent model. spire-federation-kind This example demonstrates a stock quote web app frontend and service backend by setting up two SPIFFE-identified workloads that are identified by two different SPIRE Servers. Let's say we have a stockbroker's web app that displays stock quotes periodically from a stock market web service provider. We can see the scenario as below:

1. The user enters the broker's web app stock quotes URL in a browser.
2. The web app workload receives the request and makes an HTTP request for quotes to the stock market service using mTLS.
3. The stock market service receives the request and sends the quotes in the response.
4. The web app renders the stock quotes page using the returned quotes and sends it to the browser.
5. The browser displays the quotes to the user. The web app includes JavaScript to refresh the page every 1 second, so every second these steps are executed again.

The server holds a set of secure information (stock quotes) and the client exposes HTTP endpoint to display quotes via a web browser. The communication between server-client is via gRPC protocol endpoints, and security for this connection is mTLS using SPIFFE identity framework. Each workload has been implemented to use the SPIFFE API to authenticate with SPIRE server and obtain trust bundle credentials. The following describes the details of the server-client application authenticated by SPIFFE and secured by mTLS connection.

Setting It All Up

First, let's clone spire-federation-kind repository

git clone https://github.com/nishantapatil3/spire-federation-kind

Compile and build modified stock quotes server and broker webapp client Docker images

# Replace docker push with your public docker repository
./1-build.sh

Create two clusters using Kind, by following the instructions given in below URL

https://kind.sigs.k8s.io/docs/user/quick-start/#creating-a-cluster

Let's name the clusters creates as kind-1 and kind-2 for our deployment

# Create Kind clusters
kind create cluster --name kind-1
kind create cluster --name kind-2
# Export kubeconfig to desired location
mkdir -p ~/kubeconfigs
kind get kubeconfig --name=kind-1 > ~/kubeconfigs/kind-1.kubeconfig
kind get kubeconfig --name=kind-2 > ~/kubeconfigs/kind-2.kubeconfig

Once we have two clusters running, deploy MetalLB load-balancer and SPIRE components(server-agent). MetalLB load balancer is required to allow ingress traffic to the Kubernetes cluster and redirect traffic to target pods in our cluster. globalPrefix refers to xx.xx.<globalPrefix>.xx IP address that MetallLB can be configured to allow subnet range for the external IP address in Kubernetes. Example: If globalPrefix=255, external IP address configuration range is 172.17.255.1 to 172.17.255.255

# Apply the Helm chart
helm template helm/metallb-system --set globalPrefix="255" | kubectl apply --kubeconfig $cluster1 -f -
helm template helm/metallb-system --set globalPrefix="254" | kubectl apply --kubeconfig $cluster2 -f -

After we deploy load balancers in our clusters, let's deploy spire-server and spire-agent To do that we will apply below Helm charts with SPIRE values as shown below. We can set trustDomain with any alphanumeric combination and provide this trust name to federatesWith argument that the cluster federates with. federatesWith.address should match the external IP of SPIRE service that it federates with, you can check this IP address using kubectl get svc -n spire. Also, port=8443 refers to the federation-endpoint of spire-server service configuration.

helm template helm/spire --set trustDomain=cluster1.com --set federatesWith[0].trustDomain=cluster2.com --set federatesWith[0].address=172.17.254.1 --set federatesWith[0].port=8443 | kubectl apply --kubeconfig $cluster1 -f -
helm template helm/spire --set trustDomain=cluster2.com --set federatesWith[0].trustDomain=cluster1.com --set federatesWith[0].address=172.17.255.1 --set federatesWith[0].port=8443 | kubectl apply --kubeconfig $cluster2 -f -

Next, bootstrap SPIRE certificates with each other such that the workloads (server and client) can receive the certificate from either trust domains to establish the connection The script 2-bootstrap.sh uses spire-server's tool bin/spire-server bundle show to get trust bundle from one trust domain and using that it sets the trust bundle in the other clusters spire-server using bin/spire-server bundle set

$ ./2-bootstrap.sh
Setting clusters kubeconfig /home/ubuntu/spire-federation-kind/lab_clusters.sh
Bootstrap certificate from cluster1 to cluster2
bundle set.
Bootstrap certificate from cluster2 to cluster1
bundle set.

Run the following command to create SPIRE entries such that spire-server can fetch the right certificate from a list of registered trust domains to connect to its target. This script 3-register.sh uses spire-server's container tool to register workloads stockbroker server and client in their respective clusters with SPIRE. We set the FederatesWith field to authenticate itself with other cluster's spire-server in order to connect to workloads deployed in that cluster.

$ ./3-register.sh
Setting clusters kubeconfig /home/ubuntu/spire-federation-kind/lab_clusters.sh
-------------------------
Registering workload: server
Entry ID      : 12c07f5f-8629-4654-9961-dd6dbeba577e
SPIFFE ID     : spiffe://cluster1.com/server
Parent ID     : spiffe://cluster1.com/spire-agent
Revision      : 0
TTL           : default
Selector      : k8s:sa:server-service-account
FederatesWith : spiffe://cluster2.com
-------------------------
-------------------------
Registering workload: client
Entry ID      : 8082ce59-4bca-43a5-83d5-006890f822b5
SPIFFE ID     : spiffe://cluster2.com/client
Parent ID     : spiffe://cluster2.com/spire-agent
Revision      : 0
TTL           : default
Selector      : k8s:sa:client-service-account
FederatesWith : spiffe://cluster1.com
-------------------------

Deploy stockbroker server in cluster1 and web app client in cluster2

$ kubectl apply -f helm/server.yaml --kubeconfig $cluster1
serviceaccount/server-service-account created
service/stock-quotes-service created
deployment.apps/stock-quotes-service created
$ kubectl apply -f helm/client.yaml --kubeconfig $cluster2
serviceaccount/client-service-account created
service/broker-webapp created
deployment.apps/broker-webapp created

Stockbroker server utilizes SPIRE API to create a trust bundle and initiates a listening port with x509src security. SPIRE servers exchange this trust bundle and distribute it to authenticated workloads that ask for bundles to connect to its target. The client gets the trust bundle for a given domain (Trust domains that the client is registered to federate with) using bundleSrc. Client gets two certificates from two trust domains and iterates over them to find the right certificate to connect to the stockbroker server workload and establishes mTLS secure connection to it. Client calls SPIRE API for all the trust domain that spire-server is configured to federate with and gets the trust bundle in an iterable-list Each bundle is picked up to connect to the target and if the connection succeeds, that bundle is channeled to caller function via a goroutine. After a secure channel is established between server-client, the server provides stocks information to the client which can be processed to display information to the user. Now we can Port forward the client pod to localhost:8080 using kubectl. Replace the below command with client-pod's name and run the command

Example:
kubectl port-forward broker-webapp-85574f4585-cxvxg 8080:8080 - kubeconfig $cluster2
$ kubectl port-forward broker-webapp-85574f4585-q92x4 8080:8080 --kubeconfig $cluster2
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080

Open any browser and navigate to http://localhost:8080/quotes, you should see a grid of randomly generated phony stock quotes that are updated every 1 second. Server (left k9s) and client (right k9s), you can see that client fetched two certificated from cluster1.com and cluster2.com, iterated over these certificates, and picked the right certificate to connect to stockbroker server. Source By demonstrating the above steps we have successfully created SPIRE Federated clusters and secured server-client communication.

What did we cover?

1. Deploy SPIRE in Kubernetes clusters and configure SPIFFE Federation bundle endpoints.
2. Set up SPIRE Federation configuration and exchange trust bundles between two clusters.
3. Create Bootstrap Federation bundles between two Trust Domains.
4. Demonstrate an example to show SPIRE Federation in action using SPIFFE's identity API via the golang spiffe/v2 library.

Wrap-up: What SPIRE can do?

1. Secure microservices communication.
2. Validates cryptographic service identities.
3. Eliminates the need for secret management.
4. Automatically issues, distributes and renews short-lived credentials.
5. Reduces operational overhead associated with credential management.