Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
PRODUCT
8 min read
Share
At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers such as AWS, GCP, Azure and BYOC, on-premise and hybrid, and deploy all kinds of predefined or ad-hoc workloads to these clusters. For us and our enterprise users, Kubernetes secret management (Base64) was woefully inadequate, so we chose Vault with native
Kubernetes support to manage our secrets.
Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline
Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt data, allowing access to Vault. (https://www.vaultproject.io/docs/concepts/seal.html)
Vault starts in an uninitialized state, which means it has to be initialized with an initial set of parameters. The response to the init request is the root token and unseal keys. After that, Vault becomes initialized but remains in a sealed state. A sealed state is a state in which no secrets can reach or leave Vault until a person, possibly more people than one, unseals it with the required number of unseal keys. As stated in the official documentation:
Unsealing makes the process of automating a Vault install difficult. Automated tools can easily install, configure, and start Vault, but unsealing it is a very manual process. We have plans in the future to make it easier. For the time being, the best method is to manually unseal multiple Vault servers in HA mode. Use a tool such as Consul to make sure you only query Vault servers that are unsealed.
This is one of the things that we've been trying to make easier here at Banzai Cloud. It's why the Bank-Vaults project was born, to make using Vault smoother.
Bank-Vaults runs in an endless loop and does the following:
The optional configure step looks like this:
These features are already embedded in our Bank-Vaults project (we've already done the heavy lifting), which is the most comprehensive open source project built on top of Vault. This means that the whole Vault experience is still cloud-agnostic.
Until now we've had init, unseal and configuration support on Kubernetes, AWS, Azure and Google Cloud, but a few days ago we added Alibaba Cloud support to Bank-Vaults as well. With this feature Bank-Vaults can init, unseal and configure Hashicorp Vault in the Alibaba Cloud in a cloud agnostic way. The integration was carried out with the official Go SDK for Alibaba Cloud. We have created a basic tutorial on how to setup Bank-Vaults on Alibaba Cloud (note: the tutorial assumes that you already have an account).
Install the Alibaba Cloud CLI by following the documentation. Define some basic values in the form of environment variables that will be global during this tutorial:
export ALIBABA_ACCESS_KEY_ID="YOUR ALIBABA ACCESS KEY ID HERE"
export ALIBABA_ACCESS_KEY_SECRET="YOUR ALIBABA ACCESS KEY SECRET HERE"
export ALIBABA_REGION="eu-central-1"
export ALIBABA_BUCKET="vault-test"
Create the Alibaba Object Storage Service bucket. This will be the home of the encrypted Vault root token and unseal keys:
aliyun oss mb oss://${ALIBABA_BUCKET}
--acl private
--access-key-id ${ALIBABA_ACCESS_KEY_ID} \
--access-key-secret ${ALIBABA_ACCESS_KEY_SECRET} \
--region ${ALIBABA_REGION}
Create the Alibaba KMS key for token, and unseal key encryption. Unfortunately there is no CLI way to do this, so you'll have to log into the web console and create a KMS key in the same region where you created the OSS bucket. Record the KMS Key's ID, which is in a UUIDv4 format. This documentation might help: https://www.alibabacloud.com/help/doc-detail/28935.htm Startup a Vault instance on your laptop with the following configuration:
storage "file" {
path = "/tmp/vault"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
Start Vault with the above config in the HCL file:
export VAULT_ADDR="http://127.0.0.1:8200"
vault server -config vault.hcl
Install Bank-Vaults CLI (this assumes you have a working Golang installation with $GOPATH
setup):
go get github.com/banzaicloud/bank-vaults/cmd/bank-vaults
Run Bank-Vaults in a way that uses previous Alibaba resources:
export ALIBABA_ACCESS_KEY_ID="YOUR ALIBABA ACCESS KEY ID HERE"
export ALIBABA_ACCESS_KEY_SECRET="YOUR ALIBABA ACCESS KEY SECRET HERE"
export ALIBABA_KMS_KEY_ID="7c8063eb-f9dc-421b-ae81-15d195c9f147"
bank-vaults unseal --init --mode alibaba-kms-oss \
--alibaba-access-key-id ${ALIBABA_ACCESS_KEY_ID} \
--alibaba-access-key-secret ${ALIBABA_ACCESS_KEY_SECRET} \
--alibaba-kms-region ${ALIBABA_REGION} \
--alibaba-kms-key-id ${ALIBABA_KMS_KEY_ID} \
--alibaba-oss-endpoint oss-eu-central-1.aliyuncs.com \
--alibaba-oss-bucket ${ALIBABA_BUCKET}
And the output should be the following:
INFO[0009] initializing vault
INFO[0009] unseal key stored in key store key=vault-unseal-0
INFO[0009] unseal key stored in key store key=vault-unseal-1
INFO[0009] unseal key stored in key store key=vault-unseal-2
INFO[0009] unseal key stored in key store key=vault-unseal-3
INFO[0009] unseal key stored in key store key=vault-unseal-4
INFO[0009] root token stored in key store key=vault-root
INFO[0009] checking if vault is sealed...
INFO[0009] vault sealed: true
INFO[0010] successfully unsealed vault
Also, we are in the process of implementing an Alibaba storage backend for Vault, based on the Alibaba Object Storage Service. We have a working version already; the code is located in this branch - some tweaks and tests are needed, before opening a PR - and we have opened a PR into the main Vault repository. Thus, our users will have the same level of support for AWS, Azure, Google and Alibaba not just for init, unseal, configuration (supported right now by Bank-Vaults), but also in terms of storage. There is no Alibaba storage backend in Vault right now, and - as we have already mentioned in previous posts - we like to use the available storage options offered by cloud providers. Follow this issue for more details. We have started to work on supporting Alibaba Cloud in Pipeline as well, and aspire to perfect user experience on that cloud provider when using Vault.
Prometheus metrics became part of our Vault Helm chart. We use the standard Statsd Prometheus Exporter setup, because Vault already has integrated statsd
reporting. This endpoint integrates well with our existing Prometheus monitoring. Metrics will be integrated into the Vault Operator as well: follow that issue, here. It's worth mentioning that Vault exposes interesting metrics about seal/unseal state changes, behavior that can also be observed through Prometheus.
This project is open source, of course, and its code can be found here, in our GitHub repository. We put a lot of effort into securing users of the Pipeline, since it's one of the core building blocks of the platform.
Learn more about Bank-Vaults:
Get emerging insights on innovative technology straight to your inbox.
Outshift is leading the way in building an open, interoperable, agent-first, quantum-safe infrastructure for the future of artificial intelligence.
* No email required
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.