Pod level external HTTP(S) proxy configuration with Istio

Companies frequently use proxies to act as a link between an internal network and the Internet. This is often frustrating for employees, even non-IT ones, when they can't access a specific site from the company network. For engineers it's even more obnoxious, since they have to configure all kinds of compute infrastructure to connect to external networks via these proxies. It's debatable if this is the best way to harden corporate network security, but it's still the most widely spread method to restrict outgoing traffic. The intentions behind introducing a proxy are usually good, as the main reason is almost always to prevent data breaches by adding an additional layer of security. Proxies come with other useful features, however. They can audit network traffic, filter undesired content or even act as a cache. This post will talk about corporate proxies from an infrastructure engineer's standpoint, when they have to deal with the configuration in an environment including Kubernetes and Istio.

tl;dr:

When utilizing dynamic services that run on Kubernetes and Istio, a proxy requires an increased level of configuration automation for all the components running behind it. We're going to give a brief overview of how Backyards (now Cisco Service Mesh Manager) helps with the pod-level configuration of these settings in the Kubernetes cluster.

Backyards (now Cisco Service Mesh Manager) is the Banzai Cloud Istio distribution that just works. Backyards (now Cisco Service Mesh Manager) operationalizes the service mesh to bring deep observability, convenient management, and policy-based security to modern container-based applications.

Proxying on Kubernetes

The proxy configuration for client applications usually uses environment variables, such as HTTP_PROXY , HTTPS_PROXY, and NO_PROXY. Let's assume we have the following proxy addresses:

• http://proxy.example.com:3128/ for HTTP proxy
• http://proxy.example.com:3129/ for HTTPS proxy

If that were the case, the following environment variables would need to be set in the YAML spec of a pod running on the Kubernetes cluster:

containers:
  - env:
      - name: "HTTP_PROXY"
        value: "http://proxy.example.com:3128"
      - name: "HTTPS_PROXY"
        value: "http://proxy.example.com:3129"
      - name: "NO_PROXY"
        value: "svc,local"

Manually inserting those variables into each and every pod would be far too tedious and error-prone a process, and would only work in a very small cluster anyway.

Admission webhooks to the rescue

The better option to inject these environment variables to a pod's containers is to use an admission webhook. Since Backyards (now Cisco Service Mesh Manager) already has a webhook to inject Istio sidecar proxies into pods, we can leverage and extend the existing webhook functionality to support proxy environment variable injection as well. Backyards (now Cisco Service Mesh Manager) uses the open source Banzai Cloud Istio operator under the hood, which provides support for exactly the kind of feature we were just talking about; we can specify the settings of the external proxy in the Istio custom resource and Backyards (now Cisco Service Mesh Manager) will take care of the rest. The corresponding Istio custom resource configuration is as follows:

spec:
  httpProxyEnvs:
    httpProxy: "http://proxy.example.com:3128"
    httpsProxy: "http://proxy.example.com:3129"
    noProxy: "svc,local"

Showtime

Setup

1. Create a Kubernetes cluster.

   You can create a Kubernetes cluster with the Banzai Cloud Pipeline platform on-prem, or over any one of five different cloud providers.

2. Point KUBECONFIG to your cluster.
3. Install Istio with Backyards (now Cisco Service Mesh Manager):This step usually takes a few minutes, and installs Istio and Backyards on your cluster.

Deploy an HTTPS proxy

Let's deploy a simple proxy inside the cluster to showcase external proxy usage for this example. Also, we're going to use the pod IP address to make the scenario more realistic. This example uses Tinyproxy, but any proxy that supports  HTTP CONNECT will do.

1. <create an="" external="" namespace

Create a new namespace without the Istio sidecar injection enabled to simulate the proxy being outside of the cluster.

$ kubectl create namespace external namespace/external
created

2. Deploy Tinyproxy

   $ kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tinyproxy
     namespace: external
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: tinyproxy
     template:
       metadata:
         labels:
           app: tinyproxy
       spec:
         containers:
         - image: vimagick/tinyproxy
           imagePullPolicy: IfNotPresent
           name: tinyproxy
   EOF

3. Obtain the IP address of the proxy pod

Store the proxy pod IP in the PROXY_IP environment variable for later use.

$ export PROXY_IP=$(kubectl get pod -n external -l
app=tinyproxy -o jsonpath='{@.items..podIP}')

Configure the proxy settings

1. Set the proxy settings in the Istio custom resource

The default Tinyproxy container will listen on port 8888 , so that's the port that we'll use.

$ kubectl -n istio-system patch istio mesh --type=merge --patch "
spec:
  httpProxyEnvs:
    httpProxy: "http://$PROXY_IP:8888"
    httpsProxy: "http://$PROXY_IP:8888"
    noProxy: "svc,local"

Deploy a simple pod to check outgoing connections

1. Label the default namespace for automatic sidecar injection

$ backyards sp ai on default INFO[0002] auto sidecar
injection successfully set to namespace default

2. Use the following snippet to create a test pod.

   $ kubectl create -f - <<EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: curl-test
   spec:
     containers:
     - name: curl
       image: curlimages/curl:7.70.0
       command: [ "/bin/sh", "-c", "--" ]
       args: [ "while true; do sleep 3000; done;" ]
   EOF

3. Check the environment variables of the created pod to see if the automatic injection worked.

   $ kubectl -n default get pods curl-test -o yaml |grep "env:" -A 12|head -13
       env:
       - name: http_proxy
         value: http://10.20.1.127:8888
       - name: HTTP_PROXY
         value: http://10.20.1.127:8888
       - name: https_proxy
         value: http://10.20.1.127:8888
       - name: HTTPS_PROXY
         value: http://10.20.1.127:8888
       - name: no_proxy
         value: svc,local
       - name: NO_PROXY
         value: svc,local

   The HTTP(S) proxy environment variables were automatically injected to the pod, and the variables are set in both uppercase and lowercase.

4. Send a request from the curl-test pod in the default namespace to an external service.

   $ kubectl exec -it curl-test -c curl -- sh -c "curl http://httpbin.org/headers"
   {
     "headers": {
       "Accept": "*/*",
       "Host": "httpbin.org",
       "User-Agent": "curl/7.70.0-DEV",
       "X-Amzn-Trace-Id": "Root=1-5ef23da7-4c7406e8ef392f32b61d7ddb"
     }
   }

5. Check the access logs of the proxy for your request

   $ kubectl -n external logs -l app=tinyproxy |grep CONNECT
   CONNECT   Jun 23 17:36:27 [10]: Established connection to host "httpbin.org" using file descriptor 10.
   CONNECT   Jun 23 17:36:38 [6]: Connect (file descriptor 9): ip-10-20-1-173.eu-west-1.compute.internal [10.20.1.173]
   CONNECT   Jun 23 17:36:38 [6]: Request (file descriptor 9): GET http://httpbin.org/headers HTTP/1.1
   CONNECT   Jun 23 17:36:39 [6]: Established connection to host "httpbin.org" using file descriptor 10.

   

The future: transparent external proxy with Envoy

An even better solution for the use case we've discussed would be to make it so the injected Envoy sidecar transparently supported routing through an external proxy. That way it would be totally unnecessary to configure via environment variables or provide support in the client application. There is ongoing work to support such a feature in Envoy, and hopefully it will land in the near future. The CONNECT support is already merged , and there is an open issue about CONNECT support for HTTP/1.1 upstream as well. Once these features are merged and released upstream, Backyards (now Cisco Service Mesh Manager) will switch to its Envoy-based method of configuring external proxies. 'Til then, happy proxying.