PRODUCT
8 min read
Published on 02/25/2020
Last updated on 03/21/2024
Running the Vault secret webhook alongside Istio
Share
One of the most popular feature of Bank-Vaults, the Vault swiss-army knife for Kubernetes is the secret injection webhook. With the growing popularity of Istio, recently the most requested feature was to support for running Bank-Vaults alongside Istio. We are big fans of Istio (a year ago we open sourced an Istio operator) and we have built an automated and operationalized service mesh, Banzai Cloud Backyards. As both components (Bank-Vaults and Backyards (now Cisco Service Mesh Manager)) are part of our hybrid cloud container management plaform, Pipeline, we went ahead and made them work together smoothly.
We support the following three scenarios:
- Scenario 1: Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled
- Scenario 2: The namespace where Vault is running has Istio sidecar injection enabled
- Scenario 3: Both namespaces have Istio sidecar injection enabled
Prerequisites
Install the Banzai Cloud Istio operator with the Backyards CLI
- Register for an evaluation version and run the following command to install the CLI tool (KUBECONFIG must be set for your cluster):
- Install the Istio operator using Backyards. For this example we need only the Istio operator, however feel free to experiment with the nice Backyards UI/CLI and the large collection of automated features like observability, traffic routing, canary, circuit breakers, and so on - check out this long list of features.
backyards install ? Install istio-operator (recommended). Press enter to accept Yes ? Install canary-operator (recommended). Press enter to accept No ? Install and run demo application (optional). Press enter to skip No
- Make sure you have mTLS enabled in the Istio mesh through the operator with the following command:Enable mTLS if it is not set to
STRICT
:
After this, we can check that mesh is configured with❯ backyards mtls require mesh INFO[0000] switched global mTLS to STRICT successfully
mTLS
turned on which applies to all applications in the cluster in Istio-enabled namespaces. You can change this if you would like to use another policy.$ backyards mtls get mesh mTLS rule for /mesh Policy Targets MtlsMode /default [] STRICT
Install the Bank-Vaults components
- You are recommended to create a separate namespace for Bank-Vaults called
vault-system
. You can enable Istio sidecar injection here as well, but Kubernetes won't be able to call back the webhook properly since mTLS is enabled (and Kubernetes is outside of the Istio mesh). To overcome this, apply aPERMISSIVE
Istio authentication policy to thevault-secrets-webhook
Service itself, so Kubernetes can call it back without Istio mutual TLS authentication.$ kubectl create namespace vault-system namespace/vault-system created $ backyards sidecar-proxy auto-inject on vault-system INFO[0002] auto sidecar injection successfully set to namespace default $ backyards mtls allow vault-system/vault-secrets-webhook INFO[0001] policy peers for vault-system/vault-secrets-webhook set successfully mTLS rule for vault-system/vault-secrets-webhook Policy Targets MtlsMode vault-system/vault-secrets-webhook-rw6mc [vault-secrets-webhook] PERMISSIVE
- Now you can install the operator and the webhook to the prepared namespace:
Soon the webhook and the operator become up and running. Check that thehelm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com helm upgrade --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --namespace vault-system helm upgrade --install vault-operator banzaicloud-stable/vault-operator --namespace vault-system
istio-proxy
got injected into all Pods invault-system
.
Scenario 1: Vault runs outside, the application inside the mesh {#scenario1}
To recap Scenario 1: Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled. First, install Vault outside the mesh, then install an application within the mesh.Install Vault outside the mesh {#install-vault-outside-mesh}
- Provision a Vault instance with the Bank-Vaults operator in a separate namespace:
kubectl create namespace vault
- Apply the RBAC and CR files to the cluster to create a Vault instance in the vault namespace with the operator:
kubectl apply -f rbac.yaml -f cr-istio.yaml
If you are writing your own Vault CR make sure that$ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE vault-0 3/3 Running 0 22h vault-configurer-6458cc4bf-6tpkz 1/1 Running 0 22h
istioEnabled: true
is configured, this influences port naming so the Vault service port protocols are detected by Istio correctly. - The
vault-secrets-webhook
can't inject Vault secrets intoinitContainers
in an Istio-enabled namespace when theSTRICT
authentication policy is applied to the Vault service, because Istio needs a sidecar container to do mTLS properly, and in the phase wheninitContainers
are running the Pod doesn't have a sidecar yet. If you wish to inject intoinitContainers
as well, you need to apply aPERMISSIVE
authentication policy in thevault
namespace, since it has its own TLS certificate outside of Istio scope (so this is safe to do from networking security point of view).$ backyards mtls allow vault INFO[0001] policy peers for vault/ set successfully mTLS rule for vault/ Policy Targets MtlsMode vault/default [] PERMISSIVE
Install the application inside a mesh {#install-application-inside-mesh}
In this scenario Vault is running outside the Istio mesh (as we have installed it in the previous steps) and our demo application runs within the Istio mesh. To install the demo application inside the mesh, complete the following steps:- Create a namespace first for the application and enable Istio sidecar injection:
kubectl create namespace app
backyards sidecar-proxy auto-inject on app
- Install the application manifest to the cluster:
kubectl apply -f app.yaml
- Check that the application is up and running. It should have two containers, the
app
itself and theistio-proxy
:$ kubectl get pods -n app NAME READY STATUS RESTARTS AGE app-5df5686c4-sl6dz 2/2 Running 0 119s
$ kubectl logs -f -n app deployment/app app time="2020-02-18T14:26:01Z" level=info msg="Received new Vault token" time="2020-02-18T14:26:01Z" level=info msg="Initial Vault token arrived" s3cr3t going to sleep...
Scenario 2: Running Vault inside the mesh {#scenario2}
To run Vault inside the mesh, complete the following steps. Note that these instructions assume that you have Scenario 1 up and running, and modifying it to run Vault inside the mesh.- Turn off Istio in the
app
namespace by removing theistio-injection
label:backyards sidecar-proxy auto-inject off app backyards sidecar-proxy auto-inject on vault
- Delete the Vault pods in the
vault
namespace, so they will get recreated with theistio-proxy
sidecar:kubectl delete pods --all -n vault
- Check that they both come back with an extra container (4/4 and 2/2 now):
$ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE vault-0 4/4 Running 0 1m vault-configurer-6d9b98c856-l4flc 2/2 Running 0 1m
- Delete the application pods in the app namespace, so they will get recreated without the istio-proxy sidecar:
kubectl delete pods --all -n app
app
container (1/1) and Vault access still works:
$ kubectl get pods -n app
NAME READY STATUS RESTARTS AGE
app-5df5686c4-4n6r7 1/1 Running 0 71s
$ kubectl logs -f -n app deployment/app
time="2020-02-18T14:41:20Z" level=info msg="Received new Vault token"
time="2020-02-18T14:41:20Z" level=info msg="Initial Vault token arrived"
s3cr3t
going to sleep...
Scenario 3: both Vault and the app are running inside the mesh
In this scenario, both Vault and the app are running inside the mesh. You can configure this scenario right after completing the Prerequisites.- Enable sidecar auto-injection for both namespaces:
backyards sidecar-proxy auto-inject on app backyards sidecar-proxy auto-inject on vault
- Delete all pods so they are getting injected with the proxy:
kubectl delete pods --all -n app kubectl delete pods --all -n vault
- Check the logs in the app container. It should sill show success:
$ kubectl logs -f -n app deployment/app time="2020-02-18T15:04:03Z" level=info msg="Initial Vault token arrived" time="2020-02-18T15:04:03Z" level=info msg="Renewed Vault Token" s3cr3t going to sleep...
Conclusion
The Bank-Vaults alongside Istio feature, Backing up Vault with Velero, Vault replication across multiple datacenters and HSM support with the Bank-Vaults operator are three major features in the upcoming Bank-Vaults release, so stay tuned.Update: since releasingBank-Vaults 1.0, we also providecommercial support for Bank-Vaults. If you're interested in commercial support, or anything else from our suite of products, make sure you get in touch with us,here.If you're interested in contributing, check out the Bank-Vaults repository, or give us a GitHub star.
Learn more about Bank-Vaults:
- Secret injection webhook improvements
- Backing up Vault with Velero
- Vault replication across multiple datacenters
- Vault secret injection webhook and Istio
- HSM support
- Injecting dynamic configuration with templates
- OIDC issuer discovery for Kubernetes service accounts
- Show all posts related to Bank-Vaults
About Backyards
Banzai Cloud’s Backyards (now Cisco Service Mesh Manager) is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes and our Istio operator, it gives you flexibility, portability, and consistency across on-premise datacenters and cloud environments. Use our simple, yet extremely powerful UI and CLI, and experience automated canary releases, traffic shifting, routing, secure service communication, in-depth observability and more, for yourself.Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
Subscribe
to
the Shift
!Get on emerging technology straight to your inbox.
emerging insights
The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.