Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
PRODUCT
7 min read
Share
Update: Logging operator v3 (released March, 2020)
We're constantly improving the logging-operator based on feature requests of our ops team and our customers. The main features of version 3.0 are:
- Log routing based on namespaces
- Excluding logs
- Select (or exclude) logs based on hosts and container names
- Logging operator documentation is now available on the Banzai Cloud site.
Check The Kubernetes logging operator reloaded post for details.
As we eluded to in the last post in this series, we'll be continuing our discussion of centralized and secure Kubernetes logging/log collection. Log messages can contain sensitive information, so it's important to secure transport between distributed parts of the log flow. This post will describe how we've secured moving log messages on our Kubernetes clusters provisioned by Pipeline.
Logging series: Centralized logging under Kubernetes Secure logging on Kubernetes with Fluentd and Fluent Bit Advanced logging on Kubernetes
Most of us are familiar with the TLS protocol that secures connections like HTTPS
. We will briefly discuss the basics of TLS handshakes and certificate management, just to make sure we're not glossing over any details about securing the transportation of log messages. In a simple use-case the server owns its private key and a certificate that is signed by a trusted third-party. When the client connects and receives the server certificate, the server verifies the signature. After that, it opens an encrypted channel that is secured by one of the symmetric cipher protocols. In order to provide client authentication, the protocol is extended with mTLS. You can read more about mTLS here. The most common way of enabling mutual authentication is to use client certificates. In this scenario as well, the client has a private key and certificate (signed by the third-party CA), which the server verifies. For a more detailed overview, read this great article by CloudFlare.
By following these easy steps you can set-up your own environment to secure your logging infrastructure. If you want a detailed how-to, check this OpenSSL Certificate Authority article. We will follow three easy steps:
mkdir certs csr private touch
index.txt echo "1000" > serial
You need a specific openssl configuration file openssl.cnf
To eliminate problems that arise from different OpenSSL settings, we should make sure to use a custom configuration with explicit settings (and not fall back on defaults).
[ ca ]
# `man ca`
default_ca = CA_default
[ CA_default ]
# Directory and file locations.
dir = .
certs = $dir/certs
crl_dir = $dir/crl
new_certs_dir = $dir/newcerts
database = $dir/index.txt
serial = $dir/serial
RANDFILE = $dir/private/.rand
# The root key and root certificate.
private_key = $dir/private/ca.key.pem
certificate = $dir/certs/ca.crt.pem
# SHA-1 is deprecated, so use SHA-2 instead.
default_md = sha256
name_opt = ca_default
cert_opt = ca_default
default_days = 365
preserve = no
policy = policy_strict
[ req ]
# Options for the `req` tool (`man req`).
default_bits = 4096
distinguished_name = req_distinguished_name
string_mask = utf8only
# SHA-1 is deprecated, so use SHA-2 instead.
default_md = sha256
# Extension to add when the -x509 option is used.
x509_extensions = v3_ca
[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name
localityName = Locality Name
0.organizationName = Organization Name
organizationalUnitName = Organizational Unit Name
commonName = Common Name (required)
emailAddress = Email Address
# Optionally, specify some defaults.
countryName_default = US
stateOrProvinceName_default = CA
#localityName_default = Mountain View
0.organizationName_default = Your company name
#organizationalUnitName_default =
emailAddress_default = foo@example.com
[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer
basicConstraints = critical,CA:true
keyUsage = critical, cRLSign, digitalSignature, keyCertSign
[ client_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection
[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
Before we start, we need a CA private key file, which we'll use this to sign the certificates: ca.key.pem
.
openssl genrsa -aes256 -out
private/ca.key.pem 4096
It's highly recommended that you protect keys with passwords, although you can skip that step by removing the
-aes256
option.
Using the key, we'll generate the CA certificate: ca.crt.pem
.
openssl req -config openssl.cnf \
-key private/ca.key.pem \
-new -x509 -days 365 -sha256 -extensions v3_ca \
-out certs/ca.crt.pem
We'll need a private key for the server component, as well: server.key.pem
openssl genrsa -aes256 -out
private/server.key.pem 4096
To produce a signed certificate, we first need to generate a request: server.csr
openssl req -config openssl.cnf \
-key private/server.key.pem \
-new -sha256 -out csr/server.csr.pem
As soon as we get the CSR, we can sign it with our CA: server.crt.pem
openssl ca -config openssl.cnf
-outdir certs \
-cert certs/ca.crt.pem \
-keyfile private/ca.key.pem \
-extensions server_cert -days 365 -notext -md sha256 \
-in csr/server.csr.pem \
-out certs/server.crt.pem
Fill out
Common Name
as hostname or fqdn, ie: server
We'll follow similar steps to create the server certificate.
Create a private key: client.key.pem
openssl genrsa -aes256 -out
private/client.key.pem 4096
Create a certification request: <code>client.csr.pem
openssl req -config openssl.cnf \
-key private/client.key.pem \
-new -sha256 \
-out csr/client.csr.pem
Sign the Client CSR with CA: client.crt.pem
openssl ca -config openssl.cnf
-outdir certs \
-cert certs/ca.crt.pem \
-keyfile private/ca.key.pem \
-extensions client_cert -days 365 -notext -md sha256 \
-in csr/client.csr.pem \
-out certs/client.crt.pem
Fill out
Common Name
as hostname or fqdn, ie: client
You're ready to use your fresh certificates. However, it's highly advised that you test them to make sure they were created correctly. Please run the following OpenSSL commands in two different shells to check two-way authentication.
To use the above, we need to enable TLS auth for Fluent-bit
and Fluentd
According to the documentation, "Each output plugin that requires to perform Network I/O can optionally enable TLS..."
. Great, we've already generated the necessary certificates, so now we just need to modify the configuration. For further information please read the official manual. This is a snippet from our Fluent-bit Helm chart, where the configuration looks like this:
[OUTPUT]
Name forward
Match *
Host {{ .Values.backend.forward.host }}
Port {{ .Values.backend.forward.port }}
Retry_Limit False
tls On
tls.verify {{ .Values.backend.forward.tls.verify }}
tls.ca_file /fluent-bit/ssl/ca.crt.pem
tls.crt_file /fluent-bit/ssl/client.crt.pem
tls.key_file /fluent-bit/ssl/client.key.pem
tls.key_passwd ${TLS_PRIVATE_KEY_PASSPHRASE}
Shared_Key ${SHARED_KEY}
The
${VARABLE_NAME}
values indicate the values that come from Environment variables.
To use the certificates as Secrets in Kubernetes, we have to upload them to the cluster.
kubectl create secret generic
fluentbit-tls \
--from-file=ca.crt.pem=./certs/ca.crt.pem \
--from-file=client.crt.pem=./certs/client.crt.pem \
--from-file=client.key.pem=./private/client.key.pem \
--from-literal 'server.key.passphrase=1111' \
--from-literal 'fluent.shared.key=shared1234'
After this, we only need to attach the secret to the pod with the /fluent-bit/ssl/
path.
Luckily, with the latest Fluentd we don't need the secure_input
plugin. as it's already bundled with the core. For more information, check the official documentation. This is a snippet from our custom Fluentd chart:
<source>
@type forward
port 24231
@log_level debug
<security>
self_hostname fluentd
shared_key fluentd
</security>
<transport tls>
version TLSv1_2
ca_path /fluentd/etc/ssl/ca.crt.pem
cert_path /fluentd/etc/ssl/server.crt.pem
private_key_path /fluentd/etc/ssl/server.key.pem
private_key_passphrase "#{ENV["TLS_PRIVATE_KEY_PASSPHRASE"]}"
client_cert_auth true
</transport>
</source>
The
#{ENV["VARABLE_NAME"]}
values indicate that the values come from Environment variables.
We need to upload the server secret to the cluster.
kubectl create secret generic
fluentd-tls \
--from-file=ca.crt.pem=./certs/ca.crt.pem \
--from-file=server.crt.pem=./certs/server.crt.pem \
--from-file=server.key.pem=./private/server.key.pem \
--from-literal 'server.key.passphrase=1234' \
--from-literal 'fluent.shared.key=shared1234'
In our first post, we familiarized ourselves with the tools used for collecting, aggregating and storing logs. In this article we extended those solutions with secure communication channels and authentication. In the next post in this series, we'll simplify all of this in a Pipeline spotguide (publishing the Helm charts as well), and demonstrate some advanced log parsing and monitoring techniques, too.
Get emerging insights on innovative technology straight to your inbox.
Discover how AI assistants can revolutionize your business, from automating routine tasks and improving employee productivity to delivering personalized customer experiences and bridging the AI skills gap.
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.