The Kubernetes logging operator reloaded

About a year ago we published the first release of our popular logging-operator. The initial version of that operator was designed to fit Pipeline, the Banzai Cloud hybrid cloud container management platform. However, since then, all kinds of people have found it to be an extremely useful tool that helps them manage their logs on Kubernetes. Initially, Fluent ecosystem automation was enough to support the disparate needs of our userbase, but, as the popularity of the logging-operator grew, different setups were put in place by our community that revealed some of its limitations. Over the last few months we've been collecting user feedback, wishlists and feature requests, and have decided to redesign the operator from scratch. Today, we're happy to announce that we're releasing the beta version of our brand new logging-operator v2.

Design principles of v2

Tailored to, and driven by, our community, as well as the requirements of the Pipeline platform and those of a new logging product for Kubernetes forthcoming from Banzai Cloud, the main design principles of our operator were:

• Make logging more Kubernetes native and provide namespace isolation
• Allow users to filter logs based on Kubernetes labels
• Make it easy to write new plugins by reusing as much code as possible
• Provide multiple outputs for the same logs
• Provide as much help as possible when debugging complex scenarios
• Implement support for multiple log flows

Now let's dig into these in more detail.

Namespace separation

One of the key problems with logging was separating different namespaces. Because Kubernetes logging solutions tail logs on a per node bases, they arrive in bulk from a variety of namespaces. For that reason, the operator guards the Fluentd configuration and checks permissions before adding new flows. In this way, the logging-operator adheres to namespace boundaries and denies prohibited rules.

Labels vs Fluentd tags

The second problem we faced was identifying logs. Kubernetes distinguishes resources based on their name and labels, while Fluentd handles log sources as flows. The log metadata consists of a timestamp and tag, which becomes attached to a log's record when it enters the Fluent-bit pipeline. To fix this problem we opensourced a Fluentd plugin called label-router, which is able to route logs via namespaces and labels.

Single log, multiple flows

It's common practice to archive all your logs in object stores, then send out a filtered version for analytics (e.g. ElasticSearch). Our operator seamlessly multiplies logs for different outputs, so you can set up multiple flows with different filters and different outputs.

Easily extendable and self documented

Several plugins share the same configurable component, similar to an output plugin’s buffer section, and these are now reusable across different plugins. To maintain order and a clear structure, we designed them with their documentation in mind; the operator generates documentation based on docstring comments and attribute tags. You can read more about this in our developers guide.

Configuration checks

When Fluentd aggregates logs, it shares the configurations of different log flows. To prevent a bad configuration from failing the Fluentd process, a configuration check validates these first. The new settings only go live after a successful check.

Component overview

Now that we're through covering the design principles, let's do a detailed component overview and take a walk through the new resources we've added.

Logging

The Logging resource represents the logging system, and contains configurations for Fluentd and Fluent-bit. It also establishes the controlNamespace, the administrative namespace of the logging-operator. The Fluentd statefulset and Fluent-bit daemonset will be deployed in this namespace, and non namespaced resources like ClusterOutput and ClusterFlow are only effective in this namespace - are ignored in any other namespace. For more detailed information please visit the documentation of our CRD. example Logging 

apiVersion:
logging.banzaicloud.io/v1beta1 kind: Logging metadata: name:
default-logging-simple namespace: logging spec: fluentd: {}
fluentbit: {} controlNamespace: logging

Output

Output defines an output for a logging flow. This is a namespaced resource which means only a Flow within the same namespace can access it. You can use secrets in these definitions, but they must also be in the same namespace. You can find a list of supported plugins, here. We at Banzai Cloud and our open source community members are constantly working on new plugins, so if you can't find what you need, you should feel free to get in touch with us over the Banzai Cloud Slack channel or through GitHub. example Output 

apiVersion:
logging.banzaicloud.io/v1beta1 kind: Output metadata: name:
gcs-output-sample spec: gcs: credentials_json: valueFrom:
secretKeyRef: name: gcs-secret key: credentials.json
project: logging-example bucket: banzai-log-test path:
logs/${tag}/%Y/%m/%d/ buffer: path: /tmp/buffer timekey: 1m
timekey_wait: 10s timekey_use_utc: true

Flow

Flow defines a logging flow with filters and outputs. This is a namespaced resource as well, so only logs from the same namespaces are collected. You can specify selectors to filter logs according to Kubernetes labels, and can define one or more filters within a Flow. These filters are applied in the order in the definition. You can find the supported filters list, here. At the end of the Flow, you can attach one or more outputs, which may also be Output or ClusterOutput resources. example Flow 

apiVersion:
logging.banzaicloud.io/v1beta1 kind: Flow metadata: name:
flow-sample namespace: default spec: filters: -
tag_normaliser: format:
${namespace_name}.${pod_name}.${container_name} - parser:
key_name: message parsers: - type: nginx outputRefs: -
gcs-output-sample - elasticsearch-output selectors: app:
nginx

ClusterOutput

ClusterOutput defines an Output without namespace restrictions. It is only effective in the controlNamespace.

ClusterFlow

ClusterFlow defines a Flow without namespace restrictions. It is also only effective in the controlNamespace.

Showtime

Let’s take a look at the new operator at work, as it saves all logs to S3. In the coming weeks, and before launching our new logging product, we will be sharing posts about the operator's capabilities and our/customer usage patterns, so stay tuned.

Install the logging-operator

Install the logging-operator using the Helm chart. This chart only deploys the CRDs and the operator itself. First we create the controlNamespace 

kubectl create namespace
logging-system

Note: If you have auto namespace create enabled in Helm you can skip namespace creation

$ helm repo add banzaicloud-stable
https://kubernetes-charts.banzaicloud.com $ helm repo update
$ helm install banzaicloud-stable/logging-operator
--namespace logging-system

Create default logging

Now we can spin up the logging components. The fluentd statefulset and fluent-bit daemonset will be created as a result of creating the Logging custom resource. For ease of use, we created a Helm chart to set the Logging resource up with TLS.

helm install
banzaicloud-stable/logging-operator-logging --set
controlNamespace=logging-system

kubectl apply -f logging.yaml

logging.yaml

apiVersion: logging.banzaicloud.io/v1beta1 kind: Logging
metadata: name: default-logging-simple spec: fluentd: {}
fluentbit: {} controlNamespace: logging-system

Note: ClusterOutput and ClusterFlow resources will only be accepted in the controlNamespace

Create an AWS secret

If you have your $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY set, you can use the following snippet to create a Kubernetes secret from your Amazon credentials.

kubectl create secret generic
logging-s3 --namespace logging-system --from-literal
"awsAccessKeyId=$AWS_ACCESS_KEY_ID" --from-literal "awsSecretAccesKey=$AWS_SECRET_ACCESS_KEY"

Or you can set the secret up manually:

kubectl apply -f secret.yaml

secret.yaml

apiVersion: v1 kind: Secret metadata: name: logging-s3
namespace: logging-system type: Opaque data: awsAccessKeyId:

<base64encoded> awsSecretAccesKey: <base64encoded>

It is ESSENTIAL that you install the secret and the ClusterOutput definition in the same namespace.

Create an S3 output definition

kubectl apply -f clusteroutput.yaml

clusteroutput.yaml

apiVersion:
logging.banzaicloud.io/v1beta1 kind: ClusterOutput metadata:
name: s3-output namespace: logging-system spec: s3:
aws_key_id: valueFrom: secretKeyRef: name: logging-s3 key:
awsAccessKeyId aws_sec_key: valueFrom: secretKeyRef: name:
logging-s3 key: awsSecretAccesKey s3_bucket:
logging-amazon-s3 s3_region: eu-central-1 path:
logs/${tag}/%Y/%m/%d/ buffer: path: /tmp/buffer timekey: 10m
timekey_wait: 30s timekey_use_utc: true

Note: For a production setup we recommend using a longer timekey interval to avoid generating too many objects.

Configure a ClusterFlow

The following snippet uses tag_normaliser to re-tag logs and, afterward, push them to S3.

kubectl apply -f clusterflow.yaml

clusterflow.yaml

apiVersion: logging.banzaicloud.io/v1beta1 kind: ClusterFlow
metadata: name: all-log-to-s3 namespace: logging-system
spec: filters: - tag_normaliser: {} selectors: {}
outputRefs: - s3-output

The logs then become available in the bucket of a path like: /logs/default.default-logging-simple-fluentbit-lsdp5.fluent-bit/2019/09/11/201909111432_0.gz And that's that. By now, we hope you're as excited as we are about the new version of our logging-operator. We are actively working on more filters and outputs in order to cover all of our and our customers' use cases. We'll be releasing more examples, and some of the more advanced situations our customers found themselves in, as part of the operator's documentation. That way you'll be able to design your own Kubernetes logging infrastructure based on real life examples and proven working models. If you need help and support utilizing the logging-operator, please be sure you contact us on Slack or subscribe to one of our commercial packages. Happy logging!