Dynamic SSH with Vault and Pipeline

At Banzai Cloud, we're building a feature rich platform as a service on Kubernetes, called Pipeline. With Pipeline, we provision large, multi-tenant Kubernetes clusters on all major cloud providers, such as AWS, GCP, Azure and BYOC, and deploy all kinds of predefined or ad-hoc workloads to these clusters. When we needed a way for our users to login and interact with protected endpoints and, at the same time, provide dynamic secrets management support, while simultaneously providing native Kubernetes support for all our applications, we turned to Vault.

Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

In this blog post we'll explore one of the many workflows we support: authentication via GitHub using OAuth2 tokens, organization mapping to a Vault team, and seamless SSH in cloud instances belonging to that organization. The cloud instances/VMs become trusted by connecting to Vault via AppRole and signing host keys. This way we neither have to exchange keys with our end users, nor concern ourselves with manually managing or revoking them, all of which is automated by Pipeline and secured by Vault.

All of the following operations are automated by Pipeline, thus, for clusters/control plains launched by Pipeline, these steps may not necessarily need to be taken.

Dynamic SSH credentials

Once we provision a Kubernetes cluster in the cloud or on-prem, our customers typically need to access the hosts, so we need a very dynamic method of distribution access. Static SSH keys that access these clusters (especially so that they can’t be dynamically revoked) are not an option for us and our customers. For this purpose, we've decided to use Vault's SSH Secret backend, which does dynamic Client Key Signing and Host Key Signing for accessing our remote VMs.

Requirements:

• A working Vault server accessible inside your infrastructure (automated by Pipeline)

• A GitHub organization (can be GitHub Enterprise as well, but this demo uses the public version)

• A VM (the examples below are based on Ubuntu)

• An SSH keypair on your machine

Client Key Signing

With this method a user signs (requests a certificate for) his/her SSH public key with a CA key which is stored in Vault. The target machine has to be configured to trust users signed with this key. This is automated by Pipeline as well.

Administrative steps

First and foremost you have to log in as an administrator into Vault:

# Point your Vault CLI to the Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

vault login

Let's configure a policy for the users who can SSH into the developer VMs. Configure the dev policy like this:

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }'

Configure the GitHub Auth method for the users. Everyone who logs in through the GitHub Auth method and is part of the dev team in the specified GitHub organization will be mapped to the dev policy in Vault:

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }'

Enable the ssh secret engine for client key signing on the ssh-client-signer path (can be something else, but make it consistent):

vault secrets enable -path=ssh-client-signer ssh
vault write ssh-client-signer/config/ca generate_signing_key=true
# This policy makes the CA signings valid for 24h, and is acceptable for daily workflow
vault write ssh-client-signer/roles/dev -<<EOH
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "default_extensions": [
    {
      "permit-pty": ""
    }
  ],
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "24h"
}
EOH

Download the public key from Vault onto the machine you would like to SSH into:

curl https://${VAULT_ADDR}/v1/ssh-client-signer/public_key > /etc/ssh/trusted-user-ca-keys.pem

Change /etc/ssh/sshd_config and add the following line so it will accept users with signed keys:

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Restart sshd to realize changes:

service ssh restart

User steps

Using the following steps, a user mapped to the dev policy can request Vault to sign his/her SSH public key with the trusted CA key:

# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...

vault login -method github $GITHUB_TOKEN

vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/id_rsa-cert.pub

Now you can log in to the VM:

ssh ubuntu@xy.xyz.xyz.xyz

Host Key Signing

How many times have you seen this message?

The authenticity of host 'xy.xyz.xyz.xyz (xy.xyz.xyz.xyz)' can't be established.
ECDSA key fingerprint is SHA256:wu5UbyRRuKw+JwCotA2QzQa7dsgs627pzwIYpXhprqA.
Are you sure you want to continue connecting (yes/no)?

That means you're unable to confirm that the machine you're SSHing into is a machine you can trust. With an additional configuration mechanism, and with the help of Vault, you can make sure those VMs are trusted from the get go. This reduces the probability of a user accidentally SSHing into an unmanaged or malicious machine. This is a method wherein the VMs have to authenticate themselves and sign their own keys with Vault. The VMs will use Vault's AppRole Auth Method which is an auth method oriented to automated workflows (machines and services), and is less useful for human operators.

Administrative steps

Enable the ssh secret engine for host key signing on the ssh-host-signer path (can be something else, but make it consistent):

vault secrets enable -path=ssh-host-signer ssh
vault write ssh-host-signer/config/ca generate_signing_key=true
vault secrets tune -max-lease-ttl=87600h ssh-host-signer

Configure the hostsigning policy and hostrole role:

vault write ssh-host-signer/roles/hostrole \
    key_type=ca \
    ttl=87600h \
    allow_host_certificates=true \
    allowed_domains="*" \
    allow_subdomains=true

vault write sys/policy/hostsigning policy='
    path "ssh-host-signer/sign/hostrole" {
        capabilities = ["update"]
    }'

Let's update the dev policy with two additional paths. The two additional AppRole based paths are needed so developers can request AppRole leased temporary tokens, and pass them to a newly started VM via User Data, for example:

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }
    path "auth/approle/role/hostrole/role-id" {
        capabilities = ["read"]
    }
    path "auth/approle/role/hostrole/secret-id" {
        capabilities = ["update"]
    }'

Setup AppRole auth method so that hosts will be able to lease Vault tokens for host key (self) signing:

vault write auth/approle/role/hostrole policies=hostsigning \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40

Sign the key on the host (log in with AppRole auth method first):

# Get a token with Vault AppRole
VAULT_TOKEN=$(vault write -field token auth/approle/login role_id=${VAULT_ROLE_ID} 
secret_id=${VAULT_SECRET_ID})

# Log in with that token
vault login ${VAULT_TOKEN}

# Sign the key of this host
vault write -field=signed_key ssh-host-signer/sign/hostrole \
    cert_type=host \
    public_key=@/etc/ssh/ssh_host_rsa_key.pub > /etc/ssh/ssh_host_rsa_key-cert.pub

# Set permissions on the certificate to be 0640:
chmod 0640 /etc/ssh/ssh_host_rsa_key-cert.pub

Change /etc/ssh/sshd_config and add the host certificate path to it:

# For host keys
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

Restart SSH to realize the changes:

service ssh restart

User steps

When starting a machine:

# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...

vault login -method github $GITHUB_TOKEN

VAULT_ROLE_ID=$(vault read -field role_id auth/approle/role/hostrole/role-id)
VAULT_SECRET_ID=$(vault write -f -field secret_id auth/approle/role/hostrole/secret-id)
# Pass VAULT_ROLE_ID and VAULT_SECRET_ID to the newly started VM.

When logging into that machine:

# Setup your local ssh configuration to allow hosts signed by this CA (coming from Vault)
CERT_AUTHORITY=$(vault read -field=public_key ssh-host-signer/config/ca)
echo "@cert-authority * $CERT_AUTHORITY" >> ~/.ssh/known_hosts

# If you are using client-side signed keys, sign your key temporarily
vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > 
~/.ssh/id_rsa-cert.pub

# SSH into target machines as usual
ssh ubuntu@xy.xyz.xyz.xyz

Learn by the code

We already start our Pipeline Control Plane instances with this method, which you can check out, here, in our GitHub repository.

As we've just demonstrated, Vault has a superb engine for Dynamic SSH credentials and Host Key signing, and, with simple configurations and some code, most security features required by enterprises can be quickly and effectively implemented. The whole workflow can be automated with the help of Vault's dynamic nature and the help of AppRole. For simplicity's sake we'll end this post here. However, we'll keep posting about how we seal and unseal Vault and continue to explore other advanced scenarios in our forthcoming posts. Meanwhile, please check out the open source code on our GitHub and make sure to go through the excellent tutorials and use cases made available by Vault.

Learn more about Bank-Vaults:

• Secret injection webhook improvements

• Backing up Vault with Velero

• Vault replication across multiple datacenters

• Vault secret injection webhook and Istio

• HSM support

• Injecting dynamic configuration with templates

• OIDC issuer discovery for Kubernetes service accounts

• Show all posts related to Bank-Vaults