6 min read
by Nandor Kracser
Published on 07/25/2018
Last updated on 02/05/2024
Published on 07/25/2018
Last updated on 02/05/2024
Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline
Kubernetes API proxy with PipelinePipeline is an application centric platform; our end users push deployments to Kubernetes (clusters we install or manage) through our RESTful API, or CLI. In many cases they don't care about the logistics of how they get from code to deployment, as long as that process fulfills their enterprise standards (e.g. static code analysis, vulnerability scans, autoscaling deployments, etc). However, administrators might want to have deeper interactions with the system - for example, they'll likely want to use/access the K8s API server or interactive dashboard. All Pipeline deployments are secured using OAuth2 tokens and fine grained ACLs. While we automate all of these, this post describes the low level code details and provides guidance to those who would like to implement something similar. The Kubernetes API server is a very versatile daemon, it also offers proxy capabilities towards the system and user defined Kubernetes Services, also it allows you to:
- Manage existing cluster resources
- Visualize pod CPU and memory utilization collected by heapster
- Configure an Ingress to publicly expose running applications
- Provision new deployments from JSON or YAML manifests
The workflowThis workflow assumes that you have already created a working Pipeline installation on your local machine, based on this developer documentation.
After the successful GitHub OAuth login let's get a Pipeline API token:
# Opens up this URL in your default browser
"token" part of this response and export it as an environment variable:
Create a cluster with Pipeline, already documented and augmented with Postman.
Let's assume that you have created a cluster in organization ID
export TOKEN= ...
1 and the cluster ID is
1 as well, to proxy the Kubernetes API's root path open up this URL in your browser:
Open up the Kubernetes Dashboard in your browser through the proxy:
You can also call the API with cURL, but only with the Authorization header containing the
curl -f -s -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
http://localhost:9090/pipeline/api/v1/orgs/1/clusters/1/proxy/ | jq
BenefitsBecause the whole Kubernetes API is proxied through Pipeline, the authentication mechanism can be reused, since Pipeline already handles the OAuth login so you don't have to setup an OAuth proxy. The default is GitHub, but we use the pluggable qor/auth backend, since it's easy to add Google or other OAuth providers to Pipeline. For our enterprise users we provide LDAP/AD support as well. Since users are part of organizations, and have certain roles handled by the Casbin backend, the Kubernetes API can be restricted to certain roles. For example, administrators have full CRUD access on the Dashboard, but members may only read the API.
Implementation detailsNaturally, this project is open source, and its code is available on Pipeline's GitHub repository. The driving force as it were, the proxy itself, is based on a slightly modified version of the code that makes the
kubectl proxy CLI command work.
When proxy requests arrive for a cluster, the Gin handler method checks to see if there is an existing proxy instance created for the cluster, if yes, it simply returns it and passes on the HTTP context. Otherwise a proxy is created towards the cluster based on the K8s configuration already stored in Vault. This proxy is then stored in a thread-safe sync.Map for later reuse. When the cluster is deleted the proxy gets deleted as well if there is one.
We have already talked about Gin Handlers, we install this
ProxyToCluster Handler with the Any directive to catch all HTTP methods. This handler intercepts at least three other handlers before it can start serving proxy requests. The first is the
Audit Handler, which audits all requests sent to the API (user, timestamp, path and method are saved, sensitive data is masked out). The second is the
Auth Handler, which parses the Authorization header or the session cookies sent by the user to authenticate the request. Last, the Casbin based
Authorization handler checks Casbin's role and policy storage to see if the current user has the rights to access the requested path using the corresponding method.
We take our Pipeline users security and their trust very seriously - as usual, the code is open source. If you believe you have found a security issue, please contact us at firstname.lastname@example.org. Thank you.
Get emerging insights on emerging technology straight to your inbox.
Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.