Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
INSIGHTS
12 min read
Share
Envoy is a high performance, programmable L3/L4 and L7 proxy that many service mesh implementations, such as Istio, are based on. At the core of Envoy's connection and traffic handling are network filters, which, once mixed into filter chains, allow the implementation of higher-order functionalities for access control, transformation, data enrichment, auditing, and so on. You can add new filters to extend Envoy's current feature set with new functionalities. There are two ways to go about doing this:
The second option is extremely interesting to us, as it greatly simplifies the process of extending Envoy with new capabilities. This solution relies on WebAssembly (WASM), which is an efficient portable binary instruction format providing an embeddable and isolated execution environment. If you are a frequent reader of this blog, you might be familiar with Backyards (now Cisco Service Mesh Manager), the Banzai Cloud Istio distribution. We see the service mesh as a key component of every modern Cloud Native stack, and are on a mission to make Istio easy to use and manage for everyone. We have also integrated several of our products with Istio, including Supertubes, which provides Apache Kafka as a Service on Kubernetes. While we see many benefits to running Apache Kafka inside an Istio service mesh, the ability to easily extend Envoy with new filters has pushed it to new levels.
Read more about running Kafka over Istio on our blog:
Our Kafka ACL WASM filter for Envoy reads the client certificate information that comes with mTLS traffic, and extracts the subject field required by Kafka to identify the client. The filter enriches the stream that targets Kafka with the extracted client identity, which Kafka will map to Kafka users. This enables Kafka to automatically authenticate and authorize client applications without configuring SSL on each broker, while maintaining mTLS communication between all interacting services. Moreover, this solution also allows client applications to identify themselves as their Kubernetes service accounts, without the need for the user to create and configure certificates for the client application while running inside the same Istio mesh as the Kafka cluster. Now let's return to the topic of WASM filters, but in greater detail.
With WASM filter implementations we get:
It also has a few drawbacks that need to be taken into consideration:
Envoy Proxy runs WASM filters inside a stack-based virtual machine, thus the filter's memory is isolated from the host environment. All interactions between the embedding host (Envoy Proxy) and the WASM filter are realized through functions and callbacks provided by the Envoy Proxy WASM SDK. The Envoy Proxy WASM SDK has implementations in various programming languages like:
In this post, we'll discuss how to write WASM filters for Envoy using the C++ Envoy Proxy WASM SDK. We are not going to discuss the API of the Envoy Proxy WASM SDK in detail, as it falls outside the scope of the post. However, we will touch on a few of the things that are necessary to grasp the basics of writing WASM filters for Envoy. Our filter implementation must be derived from the following two classes:
class RootContext;
class Context;
When the WASM plugin (the WASM binary that contains the filter) is loaded, a root context is created. The root context has the same lifetime as the VM instance, which executes our filter and is used for:
onConfigure(size_t) is only ever invoked in RootContext by the Envoy Proxy to pass in VM and plugin configurations. If the plugin containing one or more of your filters is expecting a configuration to be passed in by Envoy Proxy, you can override this function and obtain the configuration using the getBufferBytes helper function via WasmBufferType::VmConfiguration and WasmBufferType::PluginConfiguration respectively. The network traffic handled by Envoy Proxy will flow through the filter chain associated with the listener that receives the traffic. For each new stream that flows through a filter chain, Envoy Proxy creates a new context which lasts until the stream ends. The Context base class provides hooks (callbacks) in the form of onXXXX(...) virtual functions for HTTP and TCP traffic which are invoked as the Envoy Proxy iterates through the filter chain. Note that which callbacks are invoked on Context depends on the level of the filter chain your filter is inserted to. For example, the FilterHeadersStatus onRequestHeaders(uint32_t) is invoked only on WASM filters that are part of an HTTP-level filter chain and won't be on TCP-level filters. Your implementation of Context base class is used by Envoy Proxy for interacting with your code throughout the lifespan of the stream. You can manipulate/mutate the traffic from within these callback functions. The SDK provides specific functions for manipulating HTTP request/response header (e.g. getRequestHeader, addRequestHeader, etc), HTTP body, TCP streams (e.g. getBufferBytes, setBufferBytes), etc. Each callback function returns a status through which you can tell Envoy Proxy whether or not to pass the processing of the stream to the next filter in the chain. The next piece in the puzzle is to register the factory instances for creating our RootContext and Context implementations by declaring a static variable of type
class RegisterContextFactory;
the variable will expect the root context factory and the context factory in the form of constructor arguments.
Below is a very simple example that shows the skeleton for a WASM filter using the CPP Envoy Proxy WASM SDK: example-filter.cc:
#include "proxy_wasm_intrinsics.h"
class ExampleRootContext: public RootContext {
public:
explicit ExampleRootContext(uint32_t id, StringView root_id): RootContext(id, root_id) {}
bool onStart(size_t) override;
};
class ExampleContext: public Context {
public:
explicit ExampleContext(uint32_t id, RootContext* root) : Context(id, root) {}
FilterHeadersStatus onResponseHeaders(uint32_t) override;
FilterStatus onDownstreamData(size_t, bool) override;
};
// register factories for ExampleContext and ExampleRootContext
static RegisterContextFactory register_FilterContext(CONTEXT_FACTORY(ExampleContext),
ROOT_FACTORY(ExampleRootContext),
"my_root_id");
// invoked when the plugin initialised and is ready to process streams
bool ExampleRootContext::onStart(size_t n) {
LOG_DEBUG("ready to process streams");
return true;
}
// invoked when HTTP response header is decoded
FilterHeadersStatus ExampleContext::onResponseHeaders(uint32_t) {
addResponseHeader("resp-header-demo", "added by our filter");
return FilterHeadersStatus::Continue;
}
// invoked when downstream TCP data chunk is received
FilterStatus ExampleContext::onDownstreamData(size_t, bool) {
auto res = setBuffer(WasmBufferType::NetworkDownstreamData, 0, 0, "prepend payload to downstream data");
if (res != WasmResult::Ok) {
LOG_ERROR("Modifying downstream data failed: " + toString(res));
return FilterStatus::StopIteration;
}
return FilterStatus::Continue;
}
The easiest way to build a filter is using Docker as it won't require you to keep various libraries on your local machine.
Create Makefile for the WASM filter. Makefile:
.PHONY = all clean
PROXY_WASM_CPP_SDK=/sdk
all: example-filter.wasm
include ${PROXY_WASM_CPP_SDK}/Makefile.base_lite
Build the WASM filter:
docker run -v $PWD:/work -w /work wasmsdk:v2 /build_wasm.sh
Let's see how you can deploy our Envoy WASM filter for an application running inside an Istio service mesh on Kubernetes. You can quickly spin up an Istio mesh, including a demo application on Kubernetes with Backyards (now Cisco Service Mesh Manager), the Banzai Cloud Istio distribution.
backyards install -a --run-demo
With this single command, you get a production-ready and fully operational Istio service mesh and a demo application that consists of multiple microservices running inside the mesh.
wasm
binaryCreate a config map to hold the WASM
binary of your filter in the backyards-demo
namespace where the demo application is running.
kubectl create cm -n backyards-demo example-filter --from-file=example-filter.wasm
wasm
binary into the demo application using IstioInject the wasm
binary into the frontpage service of our demo application using the following two annotations:
sidecar.istio.io/userVolume:
'[{"name":"wasmfilters-dir","configMap": {"name":
"example-filter"}}]'
sidecar.istio.io/userVolumeMount: '[{"mountPath":"/var/local/lib/wasm-filters","name":"wasmfilters-dir"}]'
Execute the following:
kubectl scale deployment -n backyards-demo frontpage-v1 --replicas=1
kubectl patch deployment -n backyards-demo frontpage-v1 -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/userVolume":"[{\"name\":\"wasmfilters-dir\",\"configMap\": {\"name\": \"example-filter\"}}]","sidecar.istio.io/userVolumeMount":"[{\"mountPath\":\"/var/local/lib/wasm-filters\",\"name\":\"wasmfilters-dir\"}]"}}}}}'
Your WASM filter binary should now be available at /var/local/lib/wasm-filters
in the istio-proxy container:
kubectl exec -n backyards-demo -it deployment/frontpage-v1 -c istio-proxy -- ls /var/local/lib/wasm-filters/
example-filter.wasm
To enable WASM filters to log at DEBUG log-level when processing traffic which targets the frontpage
service:
kubectl port-forward -n backyards-demo deployment/frontpage-v1 15000
curl -XPOST "localhost:15000/logging?wasm=debug"
Insert our WASM filter into the HTTP-level filter chain hooked to the HTTP port 8080:
kubectl apply -f-<<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: frontpage-v1-examplefilter
namespace: backyards-demo
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
proxy:
proxyVersion: '^1\.5.*'
listener:
portNumber: 8080
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
config:
config:
name: example-filter
rootId: my_root_id
vmConfig:
code:
local:
filename: /var/local/lib/wasm-filters/example-filter.wasm
runtime: envoy.wasm.runtime.v8
vmId: example-filter
allow_precompiled: true
name: envoy.filters.http.wasm
workloadSelector:
labels:
app: frontpage
version: v1
EOF
Note: in our testing we found that the
portNumber
filter specified for the listener match in theEnvoyFilter
custom resource wasn't handled properly by upstream Istio, resulting in hooks not being invoked on our filter. This issue has been remediated in our Istio distribution, Backyards (now Cisco Service Mesh Manager).
Send some traffic to HTTP port 8080 on the frontpage
service:
kubectl run curl --image=yauritux/busybox-curl --restart=Never -it --rm sh
/home # curl -L -v http://frontpage.backyards-demo:8080
In the response, we expect to see the header of our filter added to the response header:
- About to connect() to frontpage.backyards-demo port
8080 (#0)
- Trying 10.10.178.38...
- Adding handle: conn: 0x10eadbd8
- Adding handle: send: 0
- Adding handle: recv: 0
- Curl_addHandleToPipeline: length: 1
- - Conn 0 (0x10eadbd8) send_pipe: 1, recv_pipe: 0
- Connected to frontpage.backyards-demo (10.10.178.38)
port 8080 (#0)
> GET / HTTP/1.1 User-Agent: curl/7.30.0 Host:
> frontpage.backyards-demo:8080 Accept: _/_
< HTTP/1.1 200 OK
< content-type: text/plain
< date: Thu, 16 Apr 2020 16:32:20 GMT
< content-length: 9
< x-envoy-upstream-service-time: 10
< resp-header-demo: added by our filter
< x-envoy-peer-metadata: CjYKDElOU1RBTkNFX0lQUxImGiQxMC4yMC4xLjU3LGZlODA6OmQwNDM6NDdmZjpmZWYwOmVkMjkK2QEKBkxBQkVMUxLOASrLAQoSCgNhcHASCxoJZnJvbnRwYWdlCiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjU3OGM2NTU0ZDQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9k
ZRIHGgVpc3RpbwouCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgsaCWZyb250cGFnZQorCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIEGgJ2MQoPCgd2ZXJzaW9uEgQaAnYxChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWZyb250cGFnZS12MS01N
zhjNjU1NGQ0LWxidnFrCh0KCU5BTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCTpXCgVPV05FUhJOGkxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYmFja3lhcmRzLWRlbW8vZGVwbG95bWVudHMvZnJvbnRwYWdlLXYxCi8KEVBMQVRGT1JNX01FVEFEQVRBEhoqGAoWCgpjbHVzdGVyX2lkEg
gaBm1hc3RlcgocCg9TRVJWSUNFX0FDQ09VTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCT9OQU1FEg4aDGZyb250cGFnZS12MQ==
< x-envoy-peer-metadata-id: sidecar~10.20.1.57~frontpage-v1-578c6554d4-lbvqk.backyards-demo~backyards-demo.svc.cluster.local
< x-by-metadata: CjYKDElOU1RBTkNFX0lQUxImGiQxMC4yMC4xLjU3LGZlODA6OmQwNDM6NDdmZjpmZWYwOmVkMjkK2QEKBkxBQkVMUxLOASrLAQoSCgNhcHASCxoJZnJvbnRwYWdlCiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjU3OGM2NTU0ZDQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVp
c3RpbwouCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgsaCWZyb250cGFnZQorCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIEGgJ2MQoPCgd2ZXJzaW9uEgQaAnYxChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWZyb250cGFnZS12MS01NzhjNjU1N
GQ0LWxidnFrCh0KCU5BTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCTpXCgVPV05FUhJOGkxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYmFja3lhcmRzLWRlbW8vZGVwbG95bWVudHMvZnJvbnRwYWdlLXYxCi8KEVBMQVRGT1JNX01FVEFEQVRBEhoqGAoWCgpjbHVzdGVyX2lkEggaBm1hc3
RlcgocCg9TRVJWSUNFX0FDQ09VTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCT9OQU1FEg4aDGZyb250cGFnZS12MQ==
- Server istio-envoy is not blacklisted
< server: istio-envoy
< x-envoy-decorator-operation: frontpage.backyards-demo.svc.cluster.local:8080/*
<
- Connection #0 to host frontpage.backyards-demo left
intact frontpage
```
If you want to register your WASM filter into a TCP filter chain for the frontpage
service, which accepts TCP connections on port 8083, then the EnvoyFilter
custom resource would look like this:
kubectl apply -f-<<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: frontpage-v1-examplefilter
namespace: backyards-demo
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: SIDECAR_INBOUND
proxy:
proxyVersion: '^1\.5.*'
listener:
portNumber: 8083
filterChain:
filter:
name: "envoy.tcp_proxy"
patch:
operation: INSERT_BEFORE
value:
config:
config:
name: example-filter
rootId: my_root_id
vmConfig:
code:
local:
filename: /var/local/lib/wasm-filters/example-filter.wasm
runtime: envoy.wasm.runtime.v8
vmId: example-filter
allow_precompiled: true
name: envoy.filters.network.wasm
workloadSelector:
labels:
app: frontpage
version: v1
EOF
When the filter is added to a TCP-level filter chain, only hooks specific to TCP traffic will be honoured by the SKD and invoked on the filter.
The following diagram illustrates at high level the filter deployment flow with Istio:
solo.io has provided a solution for developing WASM filters for Envoy which is a WebAssembly hub where people can upload/download their WASM filter binaries. They provide a tool called WASME that helps you to scaffold WASM filters, building and pushing the filters to WebAssembly Hub. When a WASM filter is deployed, wasme
pulls the image that contains the WASM filter plugin from WebAssembly Hub, launches a daemonset to extract the WASM plugin binary from the pulled image and make it available to Envoy Proxies on each node through hostPath
volumes.
Note: the images pulled from WebAssembly Hub would not normally show up as standard Docker images
Since this solution involves publishing and storing WASM filters to an external central location (WebAssembly Hub) it may not be an option for those enterprises that, due to stringent security policies, (or for any other reason) are unwilling to publish proprietary business logic, even in binary format outside the boundaries of the company network.
With WASM filters for Envoy, developers can write their custom code, compile it to WASM plugins, and configure Envoy to execute it. These plugins can hold arbitrary logic, so they're useful for all kinds of message integrations and mutations, which makes WASM filters for Envoy Proxy the perfect way for us to integrate Kafka on Kubernetes with Istio. In the next blog post, we'll talk about integrating Kafka's ACL mechanism with Istio mTLS in more detail. If you need new Envoy filters and need help in writing, building, self hosting and delivering them in an automated way, contact us. We are happy to help.
Get emerging insights on innovative technology straight to your inbox.
Discover how AI assistants can revolutionize your business, from automating routine tasks and improving employee productivity to delivering personalized customer experiences and bridging the AI skills gap.
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.