Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
PRODUCT
7 min read
Share
The following is a guest blog post from Jürgen Weber, Bank-Vaults user and contributor extraordinaire Here at hipages, we have a legacy approach to how we keep and maintain our ‘secrets’. The login details for some of our primary application resources are easy to obtain and with this carries great risk.. So to solve this we decided to embark on a ‘secrets’ project and implement Hashicorps Vault. As a part of this project, we looked at a variety of solutions. Being a small operations team handling 5+ product teams we are always looking for ways to make the big ‘feel’ in terms of time and resources small. This includes realising that you can not be an expert in every technology, all the way to ensuring that what you make available to your teams is quick and easy to understand and implement. The Kubernetes Operator method is one big win in this area. Companies and owners of products can turn their products (like Vault or Prometheus) into a Kubernetes resource that an administrator can just submit a Resource Definition and voila it exists. Enter the Banzai Cloud vault-operator, bank-vaults. Once we had our Vault installation up and running with the help of the Banzai Cloud team we then needed to make our applications ‘talk’ to it somehow. As a guide, we had been using various Howtos and YouTube videos we found by Seth Vargo as our frame of reference on how to achieve this. His examples use the Vault Kubernetes Auth plugin and consul-template to maintain a configuration file for your application on a memory resident disk inside a Kubernetes Pod. Consul-template will then maintain the credentials and the file, it also has the ability to run an arbitrary command to HUP your application or ask it to read the template file after changes are made to it by consult-template. I was able to get this working relatively easily but the problem we were seeing with the approach is that it is incredibly manual. I started the Kubernetes implementation here at hipages many a year ago and this was before helm
was the norm. This means our suite of applications that are running in Kubernetes are applied using generic Kubernetes YAML manifest files, with each manifest for each application residing in that applications code base. It would mean running around to our 20+ applications/Git repositories and adding all the bits and pieces to every app. How droll! How can we avoid doing this? We found the answer in the form of a ‘mutating-webhook’ that is part of the bank-vaults project. A mutating WebHook is the production of a Kubernetes API resource where you can ask the Kubernetes API to submit resources to a web server or piece of code of your choosing. In this case, when the Kubernetes API wants to ‘CREATE’ a pod, send it to the bank-vaults mutating webhook first. At the time of the investigation, the webhook only supported the KV store and PKI backends (using the bank-vaults vault-env CLI) but for our use case, we would need support for the AWS and database secrets backends. I then had a very close look at the code for the webhook and vault-env and considered how consul-template could fit into this ecosystem. The general workflow for vault-env:
vault:
or >>vault:
.value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
).The general workflow for consul-template:
A break down of the two approaches: vault-env:
consul-template:
I now have this super neat way to implement what I needed... but it did not support what I needed. So the solution? Add consul-template support to the bank-vaults mutating webhook and have the best of both worlds. So with the help of the BanzaiCloud team, I got down to adding consul-template support in the webhook, with some very efficient code reviews it was all accepted and merged within a week or so. Here is how you use this new feature (note; this assumes you have a working installation of Vault): Install the webhook:
helm init -c
helm repo add banzaicloud-stable http://kubernetes-charts.banzaicloud.com/branch/master
helm upgrade --namespace my-namespace --install my-vault
banzaicloud-stable/vault-secrets-webhook
Setup your consule-template configuration via a ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app: my-app
name: my-app-consul-template
data:
config.hcl: |
vault {
ssl {
ca_cert = "/vault/tls/ca.crt"
}
retry {
backoff = "1s"
}
}
template {
contents = <<EOH
{{- with secret "database/creds/readonly" }}
username: {{ .Data.username }}
password: {{ .Data.password }}
{{ end }}
EOH
destination = "/vault/secrets/config.yaml"
command = "/bin/sh -c \"kill -HUP $(pidof vault-demo-app) || true\""
}
Add annotations to your Deployment or PodSpec:
template:
metadata:
annotations:
vault.security.banzaicloud.io/vault-ct-configmap: "my-app-consul-template"
vault.security.banzaicloud.io/vault-role: my-app
labels:
app: my-app
name: my-app
namespace: my-namespace
spec:
volumes:
- name: app-volume
emptyDir: {}
…
You can now see our injected pods:
$ kubectl get pod mypod-svhjh -n my-namespace -o json | jq '.spec.initContainers[].name'
"vault-agent"
This pod starts up and logs into Vault and authenticates against it using the PodSpec’s defined serviceAccountName. It will write out a token into a memory resident shared file system.
$ kubectl get pod mypod-svhjh -n my-namespace -o json | jq '.spec.containers[].name'
"consul-template"
"..."
If you decide to add the consul-template webhook annotations to a Job Spec, do be aware the default will maintain consul-template as a daemon. This is not desired as the job will never terminate. I recently added another annotation vault.security.banzaicloud.io/vault-ct-once
. You can set this to true and consul-template will run once write the template and shutdown. Meaning your Job can run to completion and the pod will die.
You are now editing Pods on the fly, so maybe some users may find it confusing or odd when they install a deployment/pod/helm chart in Kubernetes and all of a sudden it has all of these 'extra containers'.
Jürgen Weber is a technology enthusiast and a lover of all things shiny and new. Having extensive experience in AWS and automation technologies like Puppet. Now the proud maintainer of a successful greenfield Kubernetes implementation at hipages.com.au.
Learn more about Bank-Vaults:
Get emerging insights on innovative technology straight to your inbox.
Discover how AI assistants can revolutionize your business, from automating routine tasks and improving employee productivity to delivering personalized customer experiences and bridging the AI skills gap.
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.