Spotguides revisited

Note: The Spotguides feature mentioned in this post is outdated and not available anymore. In case you are interested in a similar feature, contact us for details.

About a year ago we published a blog post on Spotguides, a core feature of the Banzai Cloud Pipeline platform. We spent a lot of time using and refining the original ideas, and as a result, many things changed since we first introduced the concept. In this blog post we'll learn about how Spotguides are used to easily deploy and manage complex cloud-native application stacks.

Note: The Pipeline CI/CD module mentioned in this post is outdated and not available anymore. You can integrate Pipeline to your CI/CD solution using the Pipeline API. Contact us for details.

What are Spotguides?

At Banzai Cloud most of the project names and terminologies are borrowed from surfing (yes, few of us are eager surfers). In surfing, a “spotguide” is a description of the surf spot with all the information a surfer needs to know about the wave, tide, required swell direction, etc. in order to get the most out of a given situation. Banzai Cloud Spotguides have the same intent: they deliver a lot of operations know-how for different environments in an easy-to-use package. The shortest definition is that a Spotguide is a CI/CD driven cloud-native application template which makes it possible to painlessly transition from code to a production ready Kubernetes deployment. In our Banzai Cloud Pipeline world everything can be a Spotguide that can be expressed with a CI/CD workflow --- which makes this a very versatile and powerful tool. When fully utilized, Spotguides can leverage all features of Banzai Cloud Pipeline --- including cluster provisioning, centralized logging and monitoring, storage and secret management, continuous deployment, among others --- which frees developers to focus on the applications they are developing, not the support structures around them. Each Spotguide lives in a GitHub repository, which contains a template for an application environment (along with cluster resources), and a description of how to make a working copy of itself. When a Spotguide is launched, a new GitHub repository is created for the user. The repository will contain a skeleton application with placeholder values filled in and CI/CD turned on. Launching a Spotguide is as simple as completing a questionnaire.

Why Spotguides?

Building code for Kubernetes often requires one to wear multiple hats: at a minimum, there is a developer and an operations side one has to be knowledgeable about. An application developer might have to know quite a bit about the --- often subtle --- details of a logging framework just to be able to get on with development. For example, an annoying issue developers have to deal with is that several cloud native frameworks (including Prometheus and the Fluent ecosystem) expect logs in a certain format or expect metrics to be exposed in a different way than what would be natural for developers and their tools. At other times, when an application is transitioned into production, the application logic can easily become intertwined with the operational support logic that is required to scale it. Spotguides facilitate a smoother interaction between the developer and operator worlds by enabling better separation of concerns and externalization of operational logic from the application and deployment codebases. A Spotguide embodies the knowledge of the engineers that write it and is not just a tool to increase productivity but it is also a means of facilitating knowledge transfer and standardization in an organization.

How do Spotguides work?

Spotguides embrace the concept of maintaining a desired state in the source code. This declarative, singular definition ensures that CI/CD workflows have a predictable result; it depends exclusively on the source tree. All the information is held in version control to:

• Launch or reuse Kubernetes clusters
• Test and bundle applications
• Deploy Kubernetes components
• Install monitoring dashboards

Spotguides not only lower the barriers to entry for deploying and scaling applications on Kubernetes, but they are also fully customizable. Spotguides use standard, widely adopted tools (like Helm, Prometheus, Grafana, FluentBit, Fluentd), different Banzai Cloud service tools and best practices encapsulated in a questionnaire-driven blueprint. In the following example, we will launch our Node.js application with MongoDB Spotguide. The UI wizard for creating an instance looks like this:

1. Choose a Spotguide from the menu
2. Answer the questions in the form that will be mapped to values and secrets for the CI/CD pipeline
3. Specify a new or select an existing Kubernetes cluster. To specify the cluster you will get a ready-to-use recommendation for your preferred provider based on the Spotguide's resource requirements
4. Create the GitHub repository and optionally start it by triggering the first CI/CD run
5. The CI/CD pipeline will coordinate all the details:
   1. Ensures that the desired cluster is running (and creates it if it doesn't already exist)
   2. Clones the repository
   3. Tests the application
   4. Builds a Docker image and pushes it to Docker Hub
   5. Packages the application's Helm chart
   6. Installs the necessary secrets in the Kubernetes cluster
   7. Deploys the application
6. Check the Spotguide summary for easy access to the resources that were created.

The GitHub repository contains everything you need for a production ready Node.js application. The example application exposes a simple REST API, which can be accessed using the provided HTTP endpoint. Whenever a change is made to the source code and the master branch is updated, the CI/CD pipeline will be triggered to test, validate and update the deployment of the application. After deployment, the application state and health can be monitored using the application's Grafana dashboard.

What's next?

Banzai Cloud maintains its own library of Spotguides:

• Node.js with MongoDB database
• Spark with History Server
• Spring Boot with SQL
• Zeppelin with Spark
• MySQL
• Tensorflow

With more to come soon. All this is great, but the real power of Spotguides lies in the ability to create private templates. By creating your own Spotguides (perhaps derived from ones provided by Banzai Cloud), you can provide a consistent and unified operating environment across your entire organization. This technology can be used to share best practices and also to enforce enterprise-wide operating conventions.

About Banzai Cloud Pipeline

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.