Introducing the Thanos Operator

At Banzai Cloud we support and manage hybrid Kubernetes clusters for our customers across five clouds and on-prem (bare metal, VMware). Therefore, the ability and fluency required to observe these clusters is an absolute must. Very frequently, the Pipeline control plane is tasked with managing multiple Kubernetes clusters, which it does through our own CNCF certified Kubernetes distribution, PKE, or a cloud provider-managed distribution. When that happens, it's important that we federate metrics, collect them into a single place for querying, analysis and long term storage. As discussed in a previous post, we have standardized multi-cluster-monitoring on Thanos, and we also maintain an open source and very popular Thanos Helm chart. For awhile, this Thanos Helm chart based approach worked well. However, the high number of nodes and clusters we were managing eventually began to push the limits of what was feasible; because the number of clusters that we manage from single Pipeline control planes is so high, each cluster frequently has a different layout, configuration and spread across clouds and datacenters (note that we support four different ways of building hybrid clouds on Kubernetes), so configuring them and keeping them working properly was quite a challenge. To make matters worse, Thanos has several different components that can be deployed in any number of ways. These components must work properly and securely in conjunction with Prometheus, which requires discovery (or hardcoded endpoints). Our next step, then, was to create a Thanos operator. We have considerable experience in writing operators, and have produced quite a few popular ones, such as:

• Istio operator,
• Kafka operator,
• Logging operator,
• Vault operator and webhook,
• HPA operator.

For more operators, please visit our GitHub page. As you might expect, during the development of different operators we began to recognize similar patterns and snippets of code. In accordance with the DRY (Don't Repeat Yourself) principle, we refactored these common parts into libraries like the Kubernetes object matcher library and our operator tools library. Today, we're very excited to announce the alpha release of the Banzai Cloud Thanos operator. It's our hope that, as with our other operators, a healthy community will grow up around this release and help to drive innovation in the future. Also, we'll be introducing a preview of the One Eye project, the ultimate observability tool for Kubernetes. One Eye brings observability to multi-cluster Kubernetes infrastructures and automates the collection, correlation, and storage of logs and metrics. Built on the Thanos operator, the Fluent ecosystem logging operator, Prometheus operator, Grafana, Loki and more, One Eye is a project we've been developing for quite some time, and is the only tool you'll ever need to effortlessly observe clusters. Subscribe here if you're interested in a demo, or willing to become an early adopter.

Thanos operator, showtime

There are some prerequisites to kickstarting your Thanos operator experience, such as a Kubernetes cluster and making sure you have all your dependencies set up (like Prometheus and Grafana). Don't worry though, you can easily create a Kubernetes cluster across any number of supported clouds and datacenters with the free, hosted version of Pipeline; the One Eye CLI takes care of everything else (dependencies and configurations). For simplicity's sake, we'll demo the Thanos operator for the first time in a single cluster environment (where both the produced metrics and all of Thanos' components will be deployed). In another post we'll showcase how we automated multi-cluster monitoring with Thanos, wherein metrics are produced in multiple clusters. It should be noted that the storing, compacting, querying and analyzing of these metrics is done on a separate Kubernetes cluster.

Install prerequisites

First, we need to create an object store. Thanos supports many types of object storage, as documented here. For our demo we'll be using Amazon S3.

Install with the One Eye CLI - interactive way

During the interactive mode, the One Eye CLI will ask for the name of the bucket and the associated AWS secrets, and will install Thanos and all dependencies (e.g. Prometheus). Register for an evaluation version and run the following command to install the CLI tool:

As you might know, Cisco has recently acquired Banzai Cloud. Currently we are in a transitional period and are moving our infrastructure. Contact us so we can discuss your needs and requirements, and organize a live demo.

install the One Eye CLI and deploy the Thanos operator Evaluation downloads are temporarily suspended. Contact us to discuss your needs and requirements, and organize a live demo.

Install with the One Eye CLI - non-interactive way

If you've already exported your AWS credentials as environment variables, you can create a Bucket configuration with the following commands: create object store configuration 

cat << EOF > object-store.yaml
type: S3
config:
  endpoint: "s3.eu-west-1.amazonaws.com"
  bucket: "test-bucket"
  region: "eu-west-1"
  access_key: "${AWS_ACCESS_KEY_ID}"
  secret_key: "${AWS_SECRET_ACCESS_KEY}"
EOF

create the Kubernetes secret from the object store configuration 

kubectl create secret generic one-eye --from-file=object-store.yaml=object-store.yaml

install the One Eye CLI and deploy the Thanos operator Evaluation downloads are temporarily suspended. Contact us to discuss your needs and requirements, and organize a live demo. verify the installation by opening the thanos query page 

one-eye thanos connect

Install with Helm

This will require several manual steps and configurations, so please follow along with the manual, here. Note: we use Helm 3

Benefits of an operator

While we (and our customers) use Helm quite a lot, managing applications with Helm charts has its drawbacks. We also love automation, and are constantly looking for ways to improve and automate steps that were previously manual. This helps us to properly integrate Thanos into the stack that we're trying to build. Specifically, the operator helps us:

• Auto discover endpoints
• Manage persistent volumes
• Configure metrics
• Configure simple TLS

This is not a complete list, and there is already work being done to add the following:

• Tracing configuration
• Endpoint validation
• Certificate management
• Advanced secret configuration

Now let's dig in.

Thanos Operator CRDs explained

As is typical, the Thanos operator is instrumented via custom resources, so let's take a look at the CRDs it uses.

ObjectStore

This custom resource is responsible for bucket management. Thanos compacts and downsamples on storage buckets, and it is highly recommended that you run only one compactor per bucket.

Generally speaking, more than one is not semantically concurrency safe and must be deployed as a singleton against a bucket.

Moreover, there is a great bucket inspection tool, called bucket, that provides a simple web interface for chunks stored in buckets.

Thanos & StoreEndpoint

Thanos is the main custom resource responsible for Query, Store and Rule configurations. It has a special relationship with StoreEndpoint, which represents distinct StoreAPI endpoints. These can be Sidecar, Store, Rule or any other Store API provider. Thanos resources instantiate per StoreEndpoint. This means that you can use a single Thanos configuration for several Prometheus instances or clusters. Now let's look at some deployment examples to help us understand the CRDs a little better.

Single cluster deployment

In this scenario, we'll be using Thanos for its long term storage capabilities. This means we'll install a Prometheus operator with a Thanos sidecar and a full Thanos stack. Let's see an example of a minimalist setup. objectstore 

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: ObjectStore
metadata:
  name: objectstore-sample
spec:
  config:
    mountFrom:
      secretKeyRef:
        name: thanos
        key: object-store.yaml
  bucketWeb:
    label: cluster
  compactor: {}

storeendpoint 

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: StoreEndpoint
metadata:
  name: storeendpoint-sample
spec:
  # Add fields here
  thanos: thanos-sample
  config:
    mountFrom:
      secretKeyRef:
        name: thanos
        key: object-store.yaml
  selector: {}

 thanos

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: thanos-sample
spec:
  query: {}
  rule: {}
  storeGateway: {}

Multi cluster deployment

So now we have a single cluster deployment, but what if we have several clusters and need a single means of viewing them? There are several different approaches we can take. For now, we'll explore the most basic. We have multiple clusters with Thanos already installed. We need to configure a Thanos customresource with endpoints that point to these clusters. Thanos with only query definition 

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: thanos-multi
spec:
  query: {}

 store-endpoints per cluster

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: StoreEndpoint
metadata:
  name: remote-cluster-n
spec:
  thanos: thanos-multi
  url: http://remote-cluster-n.com

Observer cluster deployment

This deployment model subtracts the additional workload from the cluster, and moves Thanos components to a dedicated observer cluster. thanos with queryDiscovery definition 

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: query-master
spec:
  query: {}
  queryDiscovery: true

thanos definition for the remote clusters 

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: thanos-generic-n
spec:
  query: {}
  rule: {}
  storeGateway: {}

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: StoreEndpoint
metadata:
  name: remote-cluster-n
spec:
  thanos: thanos-generic-n
  url: http://remote-cluster-n.com
  config:
    mountFrom:
      secretKeyRef:
        name: thanos
        key: object-store-n.yaml

Note: this deployment model is applicable to single cluster multi-namespace deployments as well

Conclusions

Thanos is a powerful tool that allows you to build highly available, multi-cluster monitoring systems. However, there are several difficulties that naturally arise when creating a production-ready version of such a system:

• Configuring Thanos service discovery for multiple clusters
• Sharding Prometheus instances
• Configuration of transport layer TLS and certificates
• Proper scaling of resources (how many metrics, and what kind of resources are needed, etc)
• Setting up Label/Time range shards
• Keeping your configuration up to date and propagating cluster changes
• Adding a caching layer

As a rule of thumb, we automate the setup of all the observability tools that our customers need to use our Pipeline hybrid-cloud container management platform. That goes for Thanos, Prometheus and the Fluent ecosystem as well. All these features land in the Thanos operator, and, with One Eye, we'll soon be making the setup and management of these tools extensible and incredibly easy. For those familiar with the Banzai Cloud Logging operator, One Eye brings with it the following extensions: Host Logs, K8s Event logs, Cloud Watch and Webhook Logs. While this project is still very much under way (we'll be releasing it late Q1), feel free to set up your Thanos infrastructure using the popular Thanos Helm chart, the Thanos operator or the One Eye CLI. If you are interested in a One Eye demo, contact us here. For the One Eye documentation click here.

View One Eye related blogposts