Policy enforcements on K8s with Pipeline

In the past few weeks we've been blogging about the advanced, enterprise-grade security features we are building into our open source PaaS, Pipeline. If you'd like to review these features, please read this series:

Security series: 
Authentication and authorization of Pipeline users with OAuth2 and Vault 
Dynamic credentials with Vault using Kubernetes Service Accounts 
Dynamic SSH with Vault and Pipeline 
Secure Kubernetes Deployments with Vault and Pipeline 
Policy enforcement on K8s with Pipeline 
The Vault swiss-army knife 
The Banzai Cloud Vault Operator 
Vault unseal flow with KMS 
Kubernetes secret management with Pipeline 
Container vulnerability scans with Pipeline 
Kubernetes API proxy with Pipeline

As you can see from the posts above, security is extremely important to us and to our enterprise users. However, we'd also like to simplify and automate it. In this post we'll be introducing policy based access control in Pipeline, and highlighting how we enforce authorization of Kubernetes deployments.

Fine grained authz for Pipeline - requirements

To recap, Pipeline is an application written in Golang. It exposes a RESTful API using the Gin web framework to:

• Create Kubernetes clusters on all major cloud providers
• Provide an end-to-end language agnostic CI/CD solution
• Manage application (Helm) repositories
• Manage cluster profiles
• Deploy applications using Helm, and manage app lifecycles
• Deploy spotguides
• Provide out of the box observability (log collection, tracing, monitoring)

For the full list of endpoints, please check the Postman collection or our OpenAPI Swagger specifications.

These endpoints are available to anyone who's been authenticated with one of the supported OAuth2 providers. However, our end users needed more fine grained access controls to interact with those endpoints (e.g. developers could use list clusters (GET), but only operators could create or delete (PUT and DELETE) them).

Enterprises came to us with all kinds of requirements and standards they already used, and custom policy lists they wanted automatically enforced in Pipeline. We needed a solution that supported various access control models, such as:

• ACL (access control lists)
• RBAC (role-based access control)
• ABAC (attribute-based access control)
• RESTful (with path support and all HTTP verbs)

Also, all of the above needed to be dynamically configurable, to work in a highly dynamic clustered environment and to work with/on Kubernetes. We went ahead and standardized the PERM metamodel (Policy, Effect, Request, Matchers) and built our authz features on the Casbin Golang project.

Fine grained access to Pipeline API

The examples below are valid and can be used in any Golang application (policy as code), or can be enforced through the Gin router

As mentioned before, we use Gin to serve our REST endpoints, and we have integrated Casbin with Gin in this package. Once the account admin is authenticated, it receives a default access policy which allows it to interact with all user specific endpoints and to query the organization-based endpoints:

username := "kellyslater"
enforcer.AddPolicy(username, "/api/v1/orgs", "*")
enforcer.AddPolicy(username, "/api/v1/tokens", "*")
for _, orgid := range orgids {
  enforcer.AddPolicy(username, fmt.Sprintf("/api/v1/orgs/%d", orgid), "GET")
  enforcer.AddPolicy(username, fmt.Sprintf("/api/v1/orgs/%d/*", orgid), "GET")
}
enforcer.SavePolicy()

The policy is saved to a supported backend, and the user can now interact with Pipeline. As part of their interactions they will be able to set different access policies for other people in the same organization through the API and UI.

At this point we save policies to a database, however, we are working on a Vault plugin to push access policies to a sealed Vault service, using our Bank-Vaults project

Please find below a typical Pipeline authorization service deployment. Besides persistent static policies, which can be changed through our UI, API or via a policy file, our end users can make permission management changes at runtime - hello, policy as code.

Policy enforcements to Kubernetes deployments

Kubernetes has a pretty comprehensive authorization model when interacting with the API server. If your application is deployed as a custom resource definition, it can leverage the RBAC and other authz features, though most deployments are not CRDs.

Imagine a very simple Spring Boot or Wordpress application deployed to Kubernetes with Pipeline. The application itself might, or might not, contain authentication and authorization. However, a Pipeline operator wants to dynamically attach authz to said application, and set up specific rules as to who can access and do what. Since such applications are accessed from outside the cluster, they require a Kubernetes ingress service. These ingresses rely on reverse proxies like nginx, or traefik - which is what Pipeline uses.

Ever since Golang 1.8, writing plugins has been easier, which is exactly what we're doing. Based on the registered ingress service in the reverse proxy, we allow operators to set fine grained access policies on top of REST verbs to control who can do what. This allows them to attach fine grained access policies to old or new deployments, through ingress controllers. Those policy configuration mechanisms and automations are the same ones we use internally for Pipeline.

Below is a typical Pipeline authorization enforcement flow.

Note that the plugin mechanism for traefik is not completely finished, so, at this point, we're using a custom built traefik. Also, note that we are set to standardize the Istio service mesh, and this might change in Pipeline in the near future