Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
PRODUCT
7 min read
Share
Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified
Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive
Apache Kafka on Kubernetes series: Kafka on Kubernetes - using etcd
Note: The Pipeline CI/CD module mentioned in this post is outdated and not available anymore. You can integrate Pipeline to your CI/CD solution using the Pipeline API. Contact us for details.
There is a newer version of Pipeline, 0.3.0 available. Accordingly, the value of this blogpost may have depreciated, so we highly recommend that you check Pipeline's documentation for the latest howto. This blog post will quickly and concisely present how to a perform a basic CI/CD workflow setup using our platform. We have also prepared a few projects in our GitHub repo that can be used as starting points (see the links at the end of this post).
These example projects make use of resources (Spark clusters on k8s on AWS) provisioned on-demand during their CI/CD workflow. However, some workflow steps vary considerably (build details, run details).
This example will focus on Spark, but you should note that Pipeline is a generic microservice platform that's not exclusively tied to use with big data workloads - is able to run any containerized, distributed workflow. Our next example in this series will involve databases, as we are moving towards JEE support, a key component of which is a persistent datastore.
To hook a Spark project into the CI/CD workflow of Banzai Cloud Pipeline, watch the video we've provided or follow the instructions below.
It's easy to make the assumption that the source of a given Spark application is stored in GitHub and its user has an AWS account, since example CI flows run on EC2 instances provisioned on demand. Nevertheless, Pipeline is not tied to a particular cloud provider, but supports a variety of cloud and managed K8S providers (see Pipeline's documentation).
Register an OAuth application on GitHub for the Pipeline CI/CD workflow. For now, fill in the Authorization callback URL
with a dummy value. This field will be updated once the Control Plane is up and running, using an IP address or DNS name. Take note of the Client ID
and Client Secret
, as these are required to launch the Pipeline Control Plane.
The easiest way to run a Pipeline Control Plane is to use a Cloudformation template.
Specify an Amazon S3 template URL
and add the URL to our template https://s3-eu-west-1.amazonaws.com/cf-templates-grr4ysncvcdl-eu-west-1/2017340oCy-new.templatei5xlidcwt4p
AWS Credentials
Control Plane Instance Config
Pipeline Credentials
{" "}
Banzai-Ci Credentials
Client Id
Client Secret
{" "}
Grafana Dashboard
{" "}
Prometheus Dashboard
Advanced Pipeline Options
0.3.0
to use the current stable Pipeline release.{" "}
Control Plane
instance.Control Plane
.{" "}Authorization callback URL
field to http://{control_plane_public_ip}/authorize
.pipeline.yml
pipeline workflow configuration for your Spark applicationThe steps of the workflow executed by the CI/CD flow are described in the .pipeline.yml
file, which must be placed in the root directory of the Spark application's source code. The file has to be pushed to the GitHub repo along with the source files of the application. Here's an example Spark application, spark-pi-example, which can be used to try out the CI/CD pipeline.
Note: To accomplish this, fork that repository into your own!
In order to set up your own spark application for the workflow, start by customizing the .pipeline.yml
configuration file in spark-pi-example
. The following sections need to be modified:
remote_build:
...
original_commands:
- mvn clean package -s settings.xml
run:
...
spark_class: banzaicloud.SparkPi
the name of your application
run:
...
spark_app_name: sparkpi
This is the relative path to your Spark application's jar
. The jar
is generated by this build command.
run:
...
spark_app_source: target/spark-pi-1.0-SNAPSHOT.jar
the application arguments
run:
...
spark_app_args: 1000
Navigate to http://{control_plane_public_ip}
in your web browser and grant access for the organizations that contain the GitHub repositories that you want to hook into the CI/CD workflow. Then click authorize access. It may take some time for all of Pipeline's services to fully initialize, and so the page may not load at first. If this happens, please wait a little and retry.
Navigate to http://{control_plane_public_ip}
, bringing you to the CI/CD user interface. Select Repositories
from the top left menu. This lists all the repositories that the Pipeline has access to. Select the repositories you want to be hooked into the CI/CD flow.
For the hooked repositories set the following secrets:
plugin_endpoint
- specify http://{control_plane_public_ip}/pipeline/api/v1
plugin_username
- specify the same user name you used for Pipeline Credentialsplugin_password
- specify the same password you used for Pipeline CredentialsModify the source code of your Spark application. Commit changes and push them to the repository on GitHub. Pipeline is notified about commits through GitHub webhooks, and will trigger the flow described in the watched repositories' .pipeline.yml
file.
Any running CI/CD jobs can be monitored and managed at http://{control_plane_public_ip}/account/repos
In order to check the logs of the different steps in the CI/CD workflow, click on the desired commit message in the UI. That's it! Once configured, the Spark application will be built, deployed and executed for every commit pushed to the project's repository. You can check on the workflow's progress by clicking on the small orange dot beside the commit in the GitHub UI. Here are some of our git repos that contain example projects with pipeline workflow configurations:
Get emerging insights on innovative technology straight to your inbox.
Discover how AI assistants can revolutionize your business, from automating routine tasks and improving employee productivity to delivering personalized customer experiences and bridging the AI skills gap.
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.