Vault unseal flow with KMS

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers such as AWS, GCP, Azure and BYOC, on-premise and hybrid, and deploy all kinds of predefined or ad-hoc workloads to these clusters. For us and our enterprise users, Kubernetes secret management (Base64) was woefully inadequate, so we chose Vault with native Kubernetes support to manage our secrets.

Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

The Vault problem

Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt data, allowing access to Vault. (https://www.vaultproject.io/docs/concepts/seal.html)

Vault starts in an uninitialized state, which means it has to be initialized with an initial set of parameters. The response to the init request is the root token and unseal keys. After that, Vault becomes initialized but remains in a sealed state. A sealed state is a state in which no secrets can reach or leave Vault until a person, possibly more people than one, unseals it with the required number of unseal keys. As stated in the official documentation:

Unsealing makes the process of automating a Vault install difficult. Automated tools can easily install, configure, and start Vault, but unsealing it is a very manual process. We have plans in the future to make it easier. For the time being, the best method is to manually unseal multiple Vault servers in HA mode. Use a tool such as Consul to make sure you only query Vault servers that are unsealed.

This is one of the things that we've been trying to make easier here at Banzai Cloud. It's why the Bank-Vaults project was born, to make using Vault smoother.

The Bank-Vaults Init and Unseal process

Bank-Vaults runs in an endless loop and does the following:

1. Bank-Vaults checks if Vault is initialized, if yes it continues to step 2, otherwise:
   • first it calls Vault init, which returns the root token and the configured number of unseal keys
   • it encrypts the above token and keys with the configured KMS key
   • it stores the encrypted token and keys in the cloud provider's object storage
   • the root token and keys are flushed from Bank-Vaults memory with explicit GC as soon as possible

• Bank-Vaults checks if Vault is sealed, if no it continues to step 3, otherwise:
  • it reads the encrypted unseal keys from the cloud provider's object storage
  • it decrypts the unseal keys with the configured KMS key
  • it unseals Vault with decrypted unseal keys
  • the keys are flushed from Bank-Vaults memory with explicit GC as soon as possible
• Repeats the second step after the configured time period

The optional configure step looks like this:

1. OS signal is received that the external configuration file has changed

• parse the configuration file
• it reads the encrypted root token from the cloud provider's object storage
• it decrypts the root token with the configured KMS key
• apply the parsed configuration on the Vault API
• the root token is flushed from Bank-Vaults memory with explicit GC as soon as possible

These features are already embedded in our Bank-Vaults project (we've already done the heavy lifting), which is the most comprehensive open source project built on top of Vault. This means that the whole Vault experience is still cloud-agnostic.

Alibaba Cloud support

Until now we've had init, unseal and configuration support on Kubernetes, AWS, Azure and Google Cloud, but a few days ago we added Alibaba Cloud support to Bank-Vaults as well. With this feature Bank-Vaults can init, unseal and configure Hashicorp Vault in the Alibaba Cloud in a cloud agnostic way. The integration was carried out with the official Go SDK for Alibaba Cloud. We have created a basic tutorial on how to setup Bank-Vaults on Alibaba Cloud (note: the tutorial assumes that you already have an account).

The setup

Install the Alibaba Cloud CLI by following the documentation. Define some basic values in the form of environment variables that will be global during this tutorial:

export ALIBABA_ACCESS_KEY_ID="YOUR ALIBABA ACCESS KEY ID HERE"
export ALIBABA_ACCESS_KEY_SECRET="YOUR ALIBABA ACCESS KEY SECRET HERE"
export ALIBABA_REGION="eu-central-1"
export ALIBABA_BUCKET="vault-test" 

Create the Alibaba Object Storage Service bucket. This will be the home of the encrypted Vault root token and unseal keys:

aliyun oss mb oss://${ALIBABA_BUCKET}
    --acl private
    --access-key-id ${ALIBABA_ACCESS_KEY_ID} \
    --access-key-secret ${ALIBABA_ACCESS_KEY_SECRET} \
    --region ${ALIBABA_REGION}

Create the Alibaba KMS key for token, and unseal key encryption. Unfortunately there is no CLI way to do this, so you'll have to log into the web console and create a KMS key in the same region where you created the OSS bucket. Record the KMS Key's ID, which is in a UUIDv4 format. This documentation might help: https://www.alibabacloud.com/help/doc-detail/28935.htm Startup a Vault instance on your laptop with the following configuration:

storage "file" {
  path = "/tmp/vault"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = true
}

Start Vault with the above config in the HCL file: 

export VAULT_ADDR="http://127.0.0.1:8200"
vault server -config vault.hcl

Install Bank-Vaults CLI (this assumes you have a working Golang installation with $GOPATH setup): 

go get github.com/banzaicloud/bank-vaults/cmd/bank-vaults

Run Bank-Vaults in a way that uses previous Alibaba resources: 

export ALIBABA_ACCESS_KEY_ID="YOUR ALIBABA ACCESS KEY ID HERE"
export ALIBABA_ACCESS_KEY_SECRET="YOUR ALIBABA ACCESS KEY SECRET HERE"
export ALIBABA_KMS_KEY_ID="7c8063eb-f9dc-421b-ae81-15d195c9f147"

bank-vaults unseal --init --mode alibaba-kms-oss \
    --alibaba-access-key-id ${ALIBABA_ACCESS_KEY_ID} \
    --alibaba-access-key-secret ${ALIBABA_ACCESS_KEY_SECRET} \
    --alibaba-kms-region ${ALIBABA_REGION} \
    --alibaba-kms-key-id ${ALIBABA_KMS_KEY_ID} \
    --alibaba-oss-endpoint oss-eu-central-1.aliyuncs.com \
    --alibaba-oss-bucket ${ALIBABA_BUCKET}

And the output should be the following: 

INFO[0009] initializing vault
INFO[0009] unseal key stored in key store                key=vault-unseal-0
INFO[0009] unseal key stored in key store                key=vault-unseal-1
INFO[0009] unseal key stored in key store                key=vault-unseal-2
INFO[0009] unseal key stored in key store                key=vault-unseal-3
INFO[0009] unseal key stored in key store                key=vault-unseal-4
INFO[0009] root token stored in key store                key=vault-root
INFO[0009] checking if vault is sealed...
INFO[0009] vault sealed: true
INFO[0010] successfully unsealed vault

Alibaba support in terms of cloud storage

Also, we are in the process of implementing an Alibaba storage backend for Vault, based on the Alibaba Object Storage Service. We have a working version already; the code is located in this branch - some tweaks and tests are needed, before opening a PR - and we have opened a PR into the main Vault repository. Thus, our users will have the same level of support for AWS, Azure, Google and Alibaba not just for init, unseal, configuration (supported right now by Bank-Vaults), but also in terms of storage. There is no Alibaba storage backend in Vault right now, and - as we have already mentioned in previous posts - we like to use the available storage options offered by cloud providers. Follow this issue for more details. We have started to work on supporting Alibaba Cloud in Pipeline as well, and aspire to perfect user experience on that cloud provider when using Vault.

Prometheus metrics

Prometheus metrics became part of our Vault Helm chart. We use the standard Statsd Prometheus Exporter setup, because Vault already has integrated statsd reporting. This endpoint integrates well with our existing Prometheus monitoring. Metrics will be integrated into the Vault Operator as well: follow that issue, here. It's worth mentioning that Vault exposes interesting metrics about seal/unseal state changes, behavior that can also be observed through Prometheus.

Learn through the code

This project is open source, of course, and its code can be found here, in our GitHub repository. We put a lot of effort into securing users of the Pipeline, since it's one of the core building blocks of the platform.

Learn more about Bank-Vaults:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• HSM support
• Injecting dynamic configuration with templates
• OIDC issuer discovery for Kubernetes service accounts
• Show all posts related to Bank-Vaults