The Banzai Cloud Vault Operator

At Banzai Cloud we're building a feature rich platform, Pipeline, on top of Kubernetes. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers - AWS, GCP, Azure and BYOC - and deploy all kinds of predefined or ad-hoc workloads to these clusters. We wanted to set the industry standard for the way in which our users log in and interact with secure endpoints, and, at the same time, we wanted to provide dynamic secret management for each application we support. With those goals in mind, and while taking into account our preference for native Kubernetes support, we selected Vault. This operator (and the Bank-Vaults project itself) are all part of the Pipeline Platform. We strive to offer enterprise grade security to our users, and to applications deployed to Kubernetes through Pipeline. In fact, it's a tier zero feature of our Pipeline Platform.

Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

We've already described what Operators for Kubernetes are in a previous blog post. In this post we'll explain how our Vault Operator was born out of the new Operator SDK and our Bank-Vaults project.

Banzai Cloud Vault Operator

There already exists a Vault Operator developed by CoreOS.

The Vault Operator makes it easier to install, manage, and maintain instances of Vault – a tool designed for storing, managing, and controlling access to secrets, such as tokens, passwords, certificates, and API keys – on Kubernetes clusters.

This is a great tool for installing a self-contained instance of Vault and etcd on top of Kubernetes. Using their work as inspiration, we've begun working toward our own solution. Although both ideas are similar, we started over and based our project on the brand new Operators SDK, which has really sped up development. Here are a few features that we sorely missed from existing operators, and which we found as opensource projects on GitHub:

• Automatic Vault initialization

• Root Token and Unseal Keys encrypted and stored in cloud KMS systems (Azure Key Vault, AWS KMS, GCP KMS)

• And in Kubernetes Secrets (however this should not be used in production, due to the current limitations of Kubernetes Secrets, see this doc for more details)

• Automated unsealing

• Automated re/configuration of Vault based on a YAML/JSON file like: Auth backends, Secret backends and policies

• And which is not tied to etcd at all so that you can choose your own storage backend (e.g. cloud provider storages)

These features are already embedded in our Bank-Vaults project (where the heavy lifting is done for us), which is the most comprehensive open source project built on top of Vault. This means that the whole Vault experience is still cloud-agnostic, just like it was before - except the operator, which is extended with event handling. All of the above configurations are done in a single YAML file. An example Vault Custom Resource Definition looks like this:

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
 name: "vault"
spec:
 size: 1
 image: vault:0.10.1
 bankVaultsImage: banzaicloud/bank-vaults:latest

 # Describe where you'd like to store the Vault unseal keys and root token.
 unsealConfig:
   # In this case we are storing the root token and the unseal keys in Kubernetes secrets.
   kubernetes:
     secretNamespace: default

 # A YAML representation of the final vault config file.
 # See https://www.vaultproject.io/docs/configuration/ for more information.
 config:
   storage:
     file:
       path: "/vault/file"
   listener:
     tcp:
       address: "0.0.0.0:8200"
       tls_cert_file: /vault/tls/server.crt
       tls_key_file: /vault/tls/server.key
   ui: true

 # See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details.
 externalConfig:
   policies:
   - name: allow_secrets
     rules: path "secret/*" {
             capabilities = ["create", "read", "update", "delete", "list"]
           }

   auth:
   - type: kubernetes
     roles:
       # Allow every pod in the default namespace to use the secret kv store
       - name: default
         bound_service_account_names: default
         bound_service_account_namespaces: default
         policies: allow_secrets
         ttl: 1h

You may have noticed where the TLS certificates come from in this configuration and found it strange. Don't worry, they're automatically generated by the operator via Sprig functions, which are also used in Helm. The main benefit of this operator, in contrast to our Vault Helm chart, is that we can react to certain Vault events in a standardized way (via the operator's framework). Here, Vault is represented as a Kubernetes resource, and its state is stored in the Vault cluster.

Local Vault Operator

This is the simplest approach. The following commands will install a single node Vault instance that stores unseal and root tokens in Kubernetes secrets:

# Install the operator
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm upgrade --install vault-operator banzaicloud-stable/vault-operator

# Create a Vault instance with the Vault CR
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr.yaml

After a few seconds, you can check the existing operator and vault pods:

kubectl get pods

NAME                                                        READY     STATUS    RESTARTS   AGE
vault-66f484898d-lbltm                                      2/2       Running   0          10s
vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s

Vault HA in Google Cloud

To demonstrate the HA setup, let's deploy Vault on Google Cloud. For this example we'll assume you have a GKE Cluster and have configured the Service Account with the following IAM Roles:

• Cloud KMS Admin

• Cloud KMS CryptoKey Encrypter/Decrypter

• Storage Admin

If you don't have a running GKE Cluster provision one with one click using Pipeline.

We'll assume that your kubectl is pointed to the GKE cluster, so you can continue deploying the operator:

git clone git@github.com:banzaicloud/bank-vaults.git
cd bank-vaults

# You won't be able to change RBAC if this command isn't executed (you'll need to get the cluster-admin role):
kubectl create clusterrolebinding ${YOUR-NAME}-cluster-admin-binding --clusterrole=cluster-admin --user=${YOUR_EMAIL}

# Install the operator
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm upgrade --install vault-operator banzaicloud-stable/vault-operator

# First, edit this file to change the project settings, see below:
kubectl apply -f deploy/rbac.yaml
kubectl apply -f deploy/cr-gcs-ha.yaml

Here you can see a sample HA configuration:

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
 name: "vault"
spec:
 size: 3
 image: vault:0.10.1
 bankVaultsImage: banzaicloud/bank-vaults:master

 # Describe where you would like to store the Vault unseal keys and root token
 # in GCS encrypted with KMS.
 # NOTE: please use your own project values, otherwise this example won't work:
 unsealConfig:
   google:
     kmsKeyRing: "vault" # CHANGEME
     kmsCryptoKey: "vault-unsealer" # CHANGEME
     kmsLocation: "global" # CHANGEME
     kmsProject: "continual-air-196513" # CHANGEME
     storageBucket: "vault-ha" # CHANGEME

 # A YAML representation of a final vault config file, this config represents
 # a HA config in Google Cloud.
 # See https://www.vaultproject.io/docs/configuration/ for more information.
 config:
   storage:
     gcs:
       bucket: "vault-ha"
       ha_enabled: "true"
   listener:
     tcp:
       address: "0.0.0.0:8200"
       tls_cert_file: /vault/tls/server.crt
       tls_key_file: /vault/tls/server.key
   api_addr: https://vault.default:8200
   ui: true

Prometheus metrics are also in the making

We also plan to add Vault Prometheus metrics to the operator's feature set (a feature that the CoreOS Operator already has), but first we'd like to carefully analyze our available options and decide what's right for us and our users. For more details, please see the attached issue. We place a lot of emphasis on observability and do some pretty advanced monitoring of our federated clusters, so stay tuned as this will also land soon.

Unseal from mobile, through a quorum

There are a few ways of unsealing Vault that are already built into Bank-Vaults, but we weren't entirely satisfied with those options. Since the operator is capable of listening to Kubernetes events - thus Vault restarts and statuses as well - we can notify alerting systems and notify operators to unseal Vault if, for example, Vaults has crashed. This allows us to unseal Vault with human intervention, using unseal keys distributed amongst a group of people, with keys stored on mobile devices (or laptops). Such a quorum would receive notifications on their mobile devices and, pending approval, the application would unseal Vault right from their phones. The Vault Operator would open up an unseal port for a limited time in order to minimize the attack window, during which Vault is sealed.

We will soon be releasing an iOS and Android app, and a Golang binary for different OSs

Learn through code

This project is open source, of course, and its code is tightly integrated into the Bank-Vaults project and can be found, here, in our GitHub repository. We put a great deal of effort into securing users of Pipeline, since security is one of the platform's core building blocks.

Learn more about Bank-Vaults:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• HSM support
• Injecting dynamic configuration with templates
• OIDC issuer discovery for Kubernetes service accounts
• Show all posts related to Bank-Vaults