Boosting observability: Extending OpenTelemetry through gateway configuration

As cloud computing and microservices grow, distributed traces are becoming essential. They help monitor performance and improve application security by spotting potential threats. OpenTelemetry is the main framework for generating, collecting, and exporting these traces. However, OpenTelemetry alone may not gather all the needed information. To get more detailed insights, many companies extend OpenTelemetry to collect extra data like request headers and bodies. 

Why extra data in OpenTelemetry traces matters 

Extra data in traces is important for several reasons: 

1. Better monitoring: It allows for more detailed tracking and analysis of how the system behaves. 
2. Easier debugging: Helps in finding and fixing complex issues. 
3. Performance optimization: Assists in making the system run more efficiently. 
4. API security: Helps in examining API parameters to detect attacks. 

Collecting this extra data can be challenging. Creating and installing custom versions of OpenTelemetry across the system is a lot of work for engineering teams. A simpler solution is to integrate OpenTelemetry extensions directly into API gateways. This approach has several benefits: 

1. Simplifies data collection: No need to gather data from each individual service. 
2. Reduces overhead: Less custom development work is required. 
3. Enhances observability: Makes it easier to get detailed trace information. 

By using the built-in features of API gateways, you can collect comprehensive trace data more efficiently. 

For more on OpenTelemetry, read the OpenTelemetry: Getting Started Series. Or read our quick-start guide for deploying the basic components of OpenTelemetry (OTel) and a way to interact with the tracing output via Jaeger.

Extending OpenTelemetry on Kong gateway

As an example, we will show how you can extend the OpenTelemetry instrumentation on Kong gateway to collect headers and body using Kong’s built-in plugins. We used Kong gateway and Jaeger on Docker images. 

Install Kong 

Follow Kong documentation to install Kong on Docker. 

Follow Kong getting started documentation to create a sample route and service. We used OpenTelemetry.io as the service’s upstream application. 

https://docs.konghq.com/gateway/latest/get-started/services-and-routes/ 

You can also configure the service and route using Kong Manager web interface. 

Test your Kong setup by sending a request to your route using a browser or Postman. You should see that the response is from OpenTelemetry.io, or the upstream application you configured in example service.  

Install Jaeger and enable traces 

Run Jaeger on the same network you installed Kong on: 

docker run --name jaeger --network kong-net ^ 
    -e COLLECTOR_OTLP_ENABLED=true ^ 
    -p 16686:16686 ^ 
    -p 4317:4317 ^ 
    -p 4318:4318 ^ 
    jaegertracing/all-in-one:latest 

Follow Kong documentation and change the configuration to include traces, and restart Kong: 

tracing_instrumentation = all 

More configuration options can be found here. 

Enable OpenTelemetry on the example service: 

curl -X POST http://localhost:8001/services/example_service/plugins \ --data "name=opentelemetry" \ --data "config.endpoint=http://jaeger:4318/v1/traces"

Or via plugins section on the manager web interface. 

Now test your route again. You should be able to see the traces on your Jaeger. 

It should look like this: 

Extend OpenTelemetry  

Now, let's collect the request’s headers and body! We will do it by using Kong Functions plugin, that lets you dynamically run Lua code from within Kong.  

This is the Lua code that we will use:

local span = kong.tracing.start_span("payload-test") 

local headers = kong.request.get_headers() 

for k, v in pairs(headers) do 
  span:set_attribute("request.headers." .. k, tostring(v)) 
end 

local body = kong.request.get_raw_body() 
span:set_attribute("request.body",body) 

You can enable the post-functions plugin and add it to the config.access hook via the web interface, or write it to access.lua file and run the following: 

curl -X POST http://localhost:8001/services/example_service/plugins \ 

    -F "name=post-function"  \ 

    -F config.access[1]=@access.lua 

Now, use Postman to send a request and add a raw body of your choice. 

Let’s check how it looks now on Jaeger! 

We can now see the request body and headers in the trace. 

Notice that these can contain sensitive information and should be treated accordingly. 

Extending OpenTelemetry to collect request payloads 

You have now successfully extended OpenTelemetry by collecting requests headers and body. This enhancement allows for more detailed monitoring and analysis, making it easier to debug issues, optimize performance, and secure your application. By integrating these extensions directly into API gateways, you simplify the data collection process, reduce the need for custom development, and enhance overall observability, providing deeper insights into system interactions.  

Read more about use cases for OpenTelemetry on the Outshift Blog!