How to write WASM filters for Envoy and deploy it with Istio

Envoy is a high performance, programmable L3/L4 and L7 proxy that many service mesh implementations, such as Istio, are based on. At the core of Envoy's connection and traffic handling are network filters, which, once mixed into filter chains, allow the implementation of higher-order functionalities for access control, transformation, data enrichment, auditing, and so on. You can add new filters to extend Envoy's current feature set with new functionalities. There are two ways to go about doing this:

• Integrate the additional filters into Envoy's source code and compile a new Envoy version. The drawback of this approach is that you need to maintain your own version of Envoy, and continuously keep it in sync with the official distribution. Moreover, since Envoy is implemented in C++, the filter has to be implemented in C++ as well.
• Dynamically load new filters into the Envoy Proxy at runtime.

The second option is extremely interesting to us, as it greatly simplifies the process of extending Envoy with new capabilities. This solution relies on WebAssembly (WASM), which is an efficient portable binary instruction format providing an embeddable and isolated execution environment. If you are a frequent reader of this blog, you might be familiar with Backyards (now Cisco Service Mesh Manager), the Banzai Cloud Istio distribution. We see the service mesh as a key component of every modern Cloud Native stack, and are on a mission to make Istio easy to use and manage for everyone. We have also integrated several of our products with Istio, including Supertubes, which provides Apache Kafka as a Service on Kubernetes. While we see many benefits to running Apache Kafka inside an Istio service mesh, the ability to easily extend Envoy with new filters has pushed it to new levels.

Read more about running Kafka over Istio on our blog:

• The benefits of integrating Apache Kafka with Istio
• 20% performance improvement by relying on Istio’s mTLS
• Envoy protocol filter for Kafka

Our Kafka ACL WASM filter for Envoy reads the client certificate information that comes with mTLS traffic, and extracts the subject field required by Kafka to identify the client. The filter enriches the stream that targets Kafka with the extracted client identity, which Kafka will map to Kafka users. This enables Kafka to automatically authenticate and authorize client applications without configuring SSL on each broker, while maintaining mTLS communication between all interacting services. Moreover, this solution also allows client applications to identify themselves as their Kubernetes service accounts, without the need for the user to create and configure certificates for the client application while running inside the same Istio mesh as the Kafka cluster. Now let's return to the topic of WASM filters, but in greater detail.

Why WASM filters

With WASM filter implementations we get:

• Agility - filters can be dynamically loaded into the running Envoy process without the need to stop or re-compile.
• Maintainability - we don't have to change the Envoy's codebase to extend its functionality.
• Diversity - popular programming languages such as C/C++ and Rust can be compiled into WASM, thus developers can implement filters using their programming language of choice.
• Reliability and isolation - filters are deployed into a VM (sandbox), therefore are isolated from the hosting Envoy process itself (e.g. when the WASM filter crashes it will not impact the Envoy process).
• Security - since filters communicate with the host (Envoy Proxy) through a well-defined API, they have access to and can modify only a limited number of connection or request properties.

It also has a few drawbacks that need to be taken into consideration:

• Performance is ~70% as fast as native C++.
• Higher memory usage due to the need to start one or more WASM virtual machines.

Envoy Proxy WASM SDK

Envoy Proxy runs WASM filters inside a stack-based virtual machine, thus the filter's memory is isolated from the host environment. All interactions between the embedding host (Envoy Proxy) and the WASM filter are realized through functions and callbacks provided by the Envoy Proxy WASM SDK. The Envoy Proxy WASM SDK has implementations in various programming languages like:

• C++
• Rust
• AssemblyScript
• Go - still experimental

In this post, we'll discuss how to write WASM filters for Envoy using the C++ Envoy Proxy WASM SDK. We are not going to discuss the API of the Envoy Proxy WASM SDK in detail, as it falls outside the scope of the post. However, we will touch on a few of the things that are necessary to grasp the basics of writing WASM filters for Envoy. Our filter implementation must be derived from the following two classes:

class RootContext;
class Context;

When the WASM plugin (the WASM binary that contains the filter) is loaded, a root context is created. The root context has the same lifetime as the VM instance, which executes our filter and is used for:

• interactions at initial setup between your code and the Envoy Proxy
• interactions that outlive a request

onConfigure(size_t) is only ever invoked in RootContext by the Envoy Proxy to pass in VM and plugin configurations. If the plugin containing one or more of your filters is expecting a configuration to be passed in by Envoy Proxy, you can override this function and obtain the configuration using the getBufferBytes helper function via WasmBufferType::VmConfiguration and WasmBufferType::PluginConfiguration respectively. The network traffic handled by Envoy Proxy will flow through the filter chain associated with the listener that receives the traffic. For each new stream that flows through a filter chain, Envoy Proxy creates a new context which lasts until the stream ends. The Context base class provides hooks (callbacks) in the form of onXXXX(...) virtual functions for HTTP and TCP traffic which are invoked as the Envoy Proxy iterates through the filter chain. Note that which callbacks are invoked on Context depends on the level of the filter chain your filter is inserted to. For example, the FilterHeadersStatus onRequestHeaders(uint32_t) is invoked only on WASM filters that are part of an HTTP-level filter chain and won't be on TCP-level filters. Your implementation of Context base class is used by Envoy Proxy for interacting with your code throughout the lifespan of the stream. You can manipulate/mutate the traffic from within these callback functions. The SDK provides specific functions for manipulating HTTP request/response header (e.g. getRequestHeader, addRequestHeader, etc), HTTP body, TCP streams (e.g. getBufferBytes, setBufferBytes), etc. Each callback function returns a status through which you can tell Envoy Proxy whether or not to pass the processing of the stream to the next filter in the chain. The next piece in the puzzle is to register the factory instances for creating our RootContext and Context implementations by declaring a static variable of type

class RegisterContextFactory;

the variable will expect the root context factory and the context factory in the form of constructor arguments.

Example filter

Below is a very simple example that shows the skeleton for a WASM filter using the CPP Envoy Proxy WASM SDK: example-filter.cc:

#include "proxy_wasm_intrinsics.h"

class ExampleRootContext: public RootContext {
public:
  explicit ExampleRootContext(uint32_t id, StringView root_id): RootContext(id, root_id) {}

  bool onStart(size_t) override;
};

class ExampleContext: public Context {
public:
  explicit ExampleContext(uint32_t id, RootContext* root) : Context(id, root) {}

  FilterHeadersStatus onResponseHeaders(uint32_t) override;

  FilterStatus onDownstreamData(size_t, bool) override;
};

// register factories for ExampleContext and ExampleRootContext
static RegisterContextFactory register_FilterContext(CONTEXT_FACTORY(ExampleContext),
                                                      ROOT_FACTORY(ExampleRootContext),
                                                      "my_root_id");

// invoked when the plugin initialised and is ready to process streams
bool ExampleRootContext::onStart(size_t n) {
  LOG_DEBUG("ready to process streams");

  return true;
}

// invoked when HTTP response header is decoded
FilterHeadersStatus ExampleContext::onResponseHeaders(uint32_t) {
  addResponseHeader("resp-header-demo", "added by our filter");

  return FilterHeadersStatus::Continue;
}

// invoked when downstream TCP data chunk is received
FilterStatus ExampleContext::onDownstreamData(size_t, bool) {
  auto res = setBuffer(WasmBufferType::NetworkDownstreamData, 0, 0, "prepend payload to downstream data");

   if (res != WasmResult::Ok) {
     LOG_ERROR("Modifying downstream data failed: " + toString(res));
      return FilterStatus::StopIteration;
   }

   return FilterStatus::Continue;
}

Build the filter

The easiest way to build a filter is using Docker as it won't require you to keep various libraries on your local machine.

1. First, create a docker image with the C++ Envoy Proxy WASM SDK as described, here

2. Create Makefile for the WASM filter. Makefile:

   .PHONY = all clean
   
   PROXY_WASM_CPP_SDK=/sdk
   
   all: example-filter.wasm
   
   include ${PROXY_WASM_CPP_SDK}/Makefile.base_lite

3. Build the WASM filter:

   docker run -v $PWD:/work -w /work  wasmsdk:v2 /build_wasm.sh

Deploy the WASM filter with Istio

Let's see how you can deploy our Envoy WASM filter for an application running inside an Istio service mesh on Kubernetes. You can quickly spin up an Istio mesh, including a demo application on Kubernetes with Backyards (now Cisco Service Mesh Manager), the Banzai Cloud Istio distribution.

backyards install -a --run-demo

With this single command, you get a production-ready and fully operational Istio service mesh and a demo application that consists of multiple microservices running inside the mesh.

Create a config map to hold the wasm binary

Create a config map to hold the WASM binary of your filter in the backyards-demo namespace where the demo application is running.

kubectl create cm -n backyards-demo example-filter --from-file=example-filter.wasm

Inject the wasm binary into the demo application using Istio

1. Inject the wasm binary into the frontpage service of our demo application using the following two annotations:

   sidecar.istio.io/userVolume:
     '[{"name":"wasmfilters-dir","configMap": {"name":
     "example-filter"}}]'
   
   sidecar.istio.io/userVolumeMount: '[{"mountPath":"/var/local/lib/wasm-filters","name":"wasmfilters-dir"}]'

2. Execute the following:

   kubectl scale deployment -n backyards-demo frontpage-v1 --replicas=1
   
   kubectl patch deployment -n backyards-demo frontpage-v1 -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/userVolume":"[{\"name\":\"wasmfilters-dir\",\"configMap\": {\"name\": \"example-filter\"}}]","sidecar.istio.io/userVolumeMount":"[{\"mountPath\":\"/var/local/lib/wasm-filters\",\"name\":\"wasmfilters-dir\"}]"}}}}}'

   Your WASM filter binary should now be available at /var/local/lib/wasm-filters in the istio-proxy container:

   kubectl exec -n backyards-demo -it deployment/frontpage-v1 -c istio-proxy -- ls /var/local/lib/wasm-filters/
   
   example-filter.wasm

3. To enable WASM filters to log at DEBUG log-level when processing traffic which targets the frontpage service:

   kubectl port-forward -n backyards-demo deployment/frontpage-v1 15000
   
   curl -XPOST "localhost:15000/logging?wasm=debug"

4. Insert our WASM filter into the HTTP-level filter chain hooked to the HTTP port 8080:

   kubectl apply -f-<<EOF
   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: frontpage-v1-examplefilter
     namespace: backyards-demo
   spec:
     configPatches:
     - applyTo: HTTP_FILTER
       match:
         context: SIDECAR_INBOUND
         proxy:
           proxyVersion: '^1\.5.*'
         listener:
           portNumber: 8080
           filterChain:
             filter:
               name: envoy.http_connection_manager
               subFilter:
                 name: envoy.router
       patch:
         operation: INSERT_BEFORE
         value:
           config:
             config:
               name: example-filter
               rootId: my_root_id
               vmConfig:
                 code:
                   local:
                     filename: /var/local/lib/wasm-filters/example-filter.wasm
                 runtime: envoy.wasm.runtime.v8
                 vmId: example-filter
                 allow_precompiled: true
           name: envoy.filters.http.wasm
     workloadSelector:
       labels:
         app: frontpage
         version: v1
   EOF

   Note: in our testing we found that the portNumber filter specified for the listener match in the EnvoyFilter custom resource wasn't handled properly by upstream Istio, resulting in hooks not being invoked on our filter. This issue has been remediated in our Istio distribution, Backyards (now Cisco Service Mesh Manager).

5. Send some traffic to HTTP port 8080 on  the frontpage service:

   kubectl run curl --image=yauritux/busybox-curl --restart=Never -it --rm sh
   
   /home # curl -L -v http://frontpage.backyards-demo:8080

   In the response, we expect to see the header of our filter added to the response header:

   - About to connect() to frontpage.backyards-demo port
     8080 (#0)
   - Trying 10.10.178.38...
   - Adding handle: conn: 0x10eadbd8
   - Adding handle: send: 0
   - Adding handle: recv: 0
   - Curl_addHandleToPipeline: length: 1
   - - Conn 0 (0x10eadbd8) send_pipe: 1, recv_pipe: 0
   - Connected to frontpage.backyards-demo (10.10.178.38)
     port 8080 (#0)
     > GET / HTTP/1.1 User-Agent: curl/7.30.0 Host:
     > frontpage.backyards-demo:8080 Accept: _/_
     < HTTP/1.1 200 OK
   < content-type: text/plain
   < date: Thu, 16 Apr 2020 16:32:20 GMT
   < content-length: 9
   < x-envoy-upstream-service-time: 10
   < resp-header-demo: added by our filter
   < x-envoy-peer-metadata: CjYKDElOU1RBTkNFX0lQUxImGiQxMC4yMC4xLjU3LGZlODA6OmQwNDM6NDdmZjpmZWYwOmVkMjkK2QEKBkxBQkVMUxLOASrLAQoSCgNhcHASCxoJZnJvbnRwYWdlCiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjU3OGM2NTU0ZDQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9k
   ZRIHGgVpc3RpbwouCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgsaCWZyb250cGFnZQorCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIEGgJ2MQoPCgd2ZXJzaW9uEgQaAnYxChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWZyb250cGFnZS12MS01N
   zhjNjU1NGQ0LWxidnFrCh0KCU5BTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCTpXCgVPV05FUhJOGkxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYmFja3lhcmRzLWRlbW8vZGVwbG95bWVudHMvZnJvbnRwYWdlLXYxCi8KEVBMQVRGT1JNX01FVEFEQVRBEhoqGAoWCgpjbHVzdGVyX2lkEg
   gaBm1hc3RlcgocCg9TRVJWSUNFX0FDQ09VTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCT9OQU1FEg4aDGZyb250cGFnZS12MQ==
   < x-envoy-peer-metadata-id: sidecar~10.20.1.57~frontpage-v1-578c6554d4-lbvqk.backyards-demo~backyards-demo.svc.cluster.local
   < x-by-metadata: CjYKDElOU1RBTkNFX0lQUxImGiQxMC4yMC4xLjU3LGZlODA6OmQwNDM6NDdmZjpmZWYwOmVkMjkK2QEKBkxBQkVMUxLOASrLAQoSCgNhcHASCxoJZnJvbnRwYWdlCiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjU3OGM2NTU0ZDQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVp
   c3RpbwouCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgsaCWZyb250cGFnZQorCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIEGgJ2MQoPCgd2ZXJzaW9uEgQaAnYxChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWZyb250cGFnZS12MS01NzhjNjU1N
   GQ0LWxidnFrCh0KCU5BTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCTpXCgVPV05FUhJOGkxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYmFja3lhcmRzLWRlbW8vZGVwbG95bWVudHMvZnJvbnRwYWdlLXYxCi8KEVBMQVRGT1JNX01FVEFEQVRBEhoqGAoWCgpjbHVzdGVyX2lkEggaBm1hc3
   RlcgocCg9TRVJWSUNFX0FDQ09VTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCT9OQU1FEg4aDGZyb250cGFnZS12MQ==
   - Server istio-envoy is not blacklisted
     < server: istio-envoy
   < x-envoy-decorator-operation: frontpage.backyards-demo.svc.cluster.local:8080/*
   <
   - Connection #0 to host frontpage.backyards-demo left
     intact frontpage
    ```

6. If you want to register your WASM filter into a TCP filter chain for the frontpage service, which accepts TCP connections on port 8083, then the EnvoyFilter custom resource would look like this:

   kubectl apply -f-<<EOF
   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: frontpage-v1-examplefilter
     namespace: backyards-demo
   spec:
     configPatches:
     - applyTo: NETWORK_FILTER
       match:
         context: SIDECAR_INBOUND
         proxy:
           proxyVersion: '^1\.5.*'
         listener:
           portNumber: 8083
           filterChain:
             filter:
               name: "envoy.tcp_proxy"
       patch:
         operation: INSERT_BEFORE
         value:
           config:
             config:
               name: example-filter
               rootId: my_root_id
               vmConfig:
                 code:
                   local:
                     filename: /var/local/lib/wasm-filters/example-filter.wasm
                 runtime: envoy.wasm.runtime.v8
                 vmId: example-filter
                 allow_precompiled: true
           name: envoy.filters.network.wasm
     workloadSelector:
       labels:
         app: frontpage
         version: v1
   EOF

   When the filter is added to a TCP-level filter chain, only hooks specific to TCP traffic will be honoured by the SKD and invoked on the filter.

The following diagram illustrates at high level the filter deployment flow with Istio:

Write WASM filters for Envoy with WASME

solo.io has provided a solution for developing WASM filters for Envoy which is a WebAssembly hub where people can upload/download their WASM filter binaries. They provide a tool called WASME that helps you to scaffold WASM filters, building and pushing the filters to WebAssembly Hub. When a WASM filter is deployed, wasme pulls the image that contains the WASM filter plugin from WebAssembly Hub, launches a daemonset to extract the WASM plugin binary from the pulled image and make it available to Envoy Proxies on each node through hostPath volumes.

Note: the images pulled from WebAssembly Hub would not normally show up as standard Docker images

Since this solution involves publishing and storing WASM filters to an external central location (WebAssembly Hub) it may not be an option for those enterprises that, due to stringent security policies, (or for any other reason) are unwilling to publish proprietary business logic, even in binary format outside the boundaries of the company network.

Wrap-up

With WASM filters for Envoy, developers can write their custom code, compile it to WASM plugins, and configure Envoy to execute it. These plugins can hold arbitrary logic, so they're useful for all kinds of message integrations and mutations, which makes WASM filters for Envoy Proxy the perfect way for us to integrate Kafka on Kubernetes with Istio. In the next blog post, we'll talk about integrating Kafka's ACL mechanism with Istio mTLS in more detail. If you need new Envoy filters and need help in writing, building, self hosting and delivering them in an automated way, contact us. We are happy to help.