Secure logging on Kubernetes with Fluentd and Fluent Bit

Update: Logging operator v3 (released March, 2020)

We're constantly improving the logging-operator based on feature requests of our ops team and our customers. The main features of version 3.0 are:

• Log routing based on namespaces
• Excluding logs
• Select (or exclude) logs based on hosts and container names
• Logging operator documentation is now available on the Banzai Cloud site.

Check The Kubernetes logging operator reloaded post for details.

As we eluded to in the last post in this series, we'll be continuing our discussion of centralized and secure Kubernetes logging/log collection. Log messages can contain sensitive information, so it's important to secure transport between distributed parts of the log flow. This post will describe how we've secured moving log messages on our Kubernetes clusters provisioned by Pipeline.

Logging series: Centralized logging under Kubernetes Secure logging on Kubernetes with Fluentd and Fluent Bit Advanced logging on Kubernetes

Client Certificate Authentication

Most of us are familiar with the TLS protocol that secures connections like HTTPS. We will briefly discuss the basics of TLS handshakes and certificate management, just to make sure we're not glossing over any details about securing the transportation of log messages. In a simple use-case the server owns its private key and a certificate that is signed by a trusted third-party. When the client connects and receives the server certificate, the server verifies the signature. After that, it opens an encrypted channel that is secured by one of the symmetric cipher protocols. In order to provide client authentication, the protocol is extended with mTLS. You can read more about mTLS here. The most common way of enabling mutual authentication is to use client certificates. In this scenario as well, the client has a private key and certificate (signed by the third-party CA), which the server verifies. For a more detailed overview, read this great article by CloudFlare.

Setting up the Certificates

By following these easy steps you can set-up your own environment to secure your logging infrastructure. If you want a detailed how-to, check this OpenSSL Certificate Authority article. We will follow three easy steps:

• Generate the Root CA (for self-signed certificates)
• Generate the Intermediate Server Cert (Singed with CA)
• Generate the Client certificate (Signed with CA)

Prepare the working directory

mkdir certs csr private touch
index.txt echo "1000" > serial

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.crt.pem

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 365
preserve          = no
policy            = policy_strict

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name (required)
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = US
stateOrProvinceName_default     = CA
#localityName_default           = Mountain View
0.organizationName_default      = Your company name
#organizationalUnitName_default =
emailAddress_default            = foo@example.com

[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer
basicConstraints = critical,CA:true
keyUsage = critical, cRLSign, digitalSignature, keyCertSign

[ client_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

Generate files for Certification Authority (CA)

Before we start, we need a CA private key file, which we'll use this to sign the certificates: ca.key.pem.

openssl genrsa -aes256 -out
private/ca.key.pem 4096

It's highly recommended that you protect keys with passwords, although you can skip that step by removing the -aes256 option.

Generate cert for Certification Authority (CA)

Using the key, we'll generate the CA certificate: ca.crt.pem.

openssl req -config openssl.cnf \
 -key private/ca.key.pem \
 -new -x509 -days 365 -sha256 -extensions v3_ca \
 -out certs/ca.crt.pem

Generate Server Private Key

We'll need a private key for the server component, as well: server.key.pem

openssl genrsa -aes256 -out
private/server.key.pem 4096

Create Server Certificate Sign Request (csr)

To produce a signed certificate, we first need to generate a request: server.csr

openssl req -config openssl.cnf \
 -key private/server.key.pem \
 -new -sha256 -out csr/server.csr.pem

Self Sign Server CSR with CA

As soon as we get the CSR, we can sign it with our CA: server.crt.pem

openssl ca -config openssl.cnf
-outdir certs \
 -cert certs/ca.crt.pem \
 -keyfile private/ca.key.pem \
 -extensions server_cert -days 365 -notext -md sha256 \
 -in csr/server.csr.pem \
 -out certs/server.crt.pem

Fill out Common Name as hostname or fqdn, ie: server

Create the Client certificates

We'll follow similar steps to create the server certificate.

• Create a private key: client.key.pem

  openssl genrsa -aes256 -out
  private/client.key.pem 4096

• Create a certification request: <code>client.csr.pem

  openssl req -config openssl.cnf \
   -key private/client.key.pem \
   -new -sha256 \
   -out csr/client.csr.pem

• Sign the Client CSR with CA: client.crt.pem

  openssl ca -config openssl.cnf
  -outdir certs \ 
  -cert certs/ca.crt.pem \ 
  -keyfile private/ca.key.pem \ 
  -extensions client_cert -days 365 -notext -md sha256 \ 
  -in csr/client.csr.pem \ 
  -out certs/client.crt.pem

Fill out Common Name as hostname or fqdn, ie: client

Check the certificates

You're ready to use your fresh certificates. However, it's highly advised that you test them to make sure they were created correctly. Please run the following OpenSSL commands in two different shells to check two-way authentication.

Secure logging in action

To use the above, we need to enable TLS auth for Fluent-bit and Fluentd

Enable TLS on Fluent-bit

According to the documentation, "Each output plugin that requires to perform Network I/O can optionally enable TLS...". Great, we've already generated the necessary certificates, so now we just need to modify the configuration. For further information please read the official manual. This is a snippet from our Fluent-bit Helm chart, where the configuration looks like this:

[OUTPUT]
    Name          forward
    Match         *
    Host          {{ .Values.backend.forward.host }}
    Port          {{ .Values.backend.forward.port }}
    Retry_Limit False
    tls               On
    tls.verify        {{ .Values.backend.forward.tls.verify }}
    tls.ca_file       /fluent-bit/ssl/ca.crt.pem
    tls.crt_file      /fluent-bit/ssl/client.crt.pem
    tls.key_file      /fluent-bit/ssl/client.key.pem
    tls.key_passwd    ${TLS_PRIVATE_KEY_PASSPHRASE}
    Shared_Key        ${SHARED_KEY}

The ${VARABLE_NAME} values indicate the values that come from Environment variables.

To use the certificates as Secrets in Kubernetes, we have to upload them to the cluster.

kubectl create secret generic
fluentbit-tls \
 --from-file=ca.crt.pem=./certs/ca.crt.pem \
 --from-file=client.crt.pem=./certs/client.crt.pem \
 --from-file=client.key.pem=./private/client.key.pem \
 --from-literal 'server.key.passphrase=1111' \
 --from-literal 'fluent.shared.key=shared1234'

After this, we only need to attach the secret to the pod with the /fluent-bit/ssl/ path.

Enable TLS on Fluentd

Luckily, with the latest Fluentd we don't need the secure_input plugin. as it's already bundled with the core. For more information, check the official documentation. This is a snippet from our custom Fluentd chart:

<source>
  @type   forward
  port    24231
  @log_level debug
  <security>
    self_hostname fluentd
    shared_key fluentd
  </security>
  <transport tls>
    version                TLSv1_2
    ca_path                /fluentd/etc/ssl/ca.crt.pem
    cert_path              /fluentd/etc/ssl/server.crt.pem
    private_key_path       /fluentd/etc/ssl/server.key.pem
    private_key_passphrase "#{ENV["TLS_PRIVATE_KEY_PASSPHRASE"]}"
    client_cert_auth       true
  </transport>
</source>

The #{ENV["VARABLE_NAME"]} values indicate that the values come from Environment variables.

We need to upload the server secret to the cluster.

kubectl create secret generic
fluentd-tls \
 --from-file=ca.crt.pem=./certs/ca.crt.pem \
 --from-file=server.crt.pem=./certs/server.crt.pem \
 --from-file=server.key.pem=./private/server.key.pem \
 --from-literal 'server.key.passphrase=1234' \
 --from-literal 'fluent.shared.key=shared1234'

Now put it together

In our first post, we familiarized ourselves with the tools used for collecting, aggregating and storing  logs. In this article we extended those solutions with secure communication channels and authentication. In the next post in this series, we'll simplify all of this in a Pipeline spotguide (publishing the Helm charts as well), and demonstrate some advanced log parsing and monitoring techniques, too.