Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
INSIGHTS
4 min read
Published on 08/06/2024
Last updated on 08/06/2024
Boosting observability: Extending OpenTelemetry through gateway configuration
Share
As cloud computing and microservices grow, distributed traces are becoming essential. They help monitor performance and improve application security by spotting potential threats. OpenTelemetry is the main framework for generating, collecting, and exporting these traces. However, OpenTelemetry alone may not gather all the needed information. To get more detailed insights, many companies extend OpenTelemetry to collect extra data like request headers and bodies.
Why extra data in OpenTelemetry traces matters
Extra data in traces is important for several reasons:
- Better monitoring: It allows for more detailed tracking and analysis of how the system behaves.
- Easier debugging: Helps in finding and fixing complex issues.
- Performance optimization: Assists in making the system run more efficiently.
- API security: Helps in examining API parameters to detect attacks.
Collecting this extra data can be challenging. Creating and installing custom versions of OpenTelemetry across the system is a lot of work for engineering teams. A simpler solution is to integrate OpenTelemetry extensions directly into API gateways. This approach has several benefits:
- Simplifies data collection: No need to gather data from each individual service.
- Reduces overhead: Less custom development work is required.
- Enhances observability: Makes it easier to get detailed trace information.
By using the built-in features of API gateways, you can collect comprehensive trace data more efficiently.
For more on OpenTelemetry, read the OpenTelemetry: Getting Started Series. Or read our quick-start guide for deploying the basic components of OpenTelemetry (OTel) and a way to interact with the tracing output via Jaeger.
Extending OpenTelemetry on Kong gateway
As an example, we will show how you can extend the OpenTelemetry instrumentation on Kong gateway to collect headers and body using Kong’s built-in plugins. We used Kong gateway and Jaeger on Docker images.
Install Kong
Follow Kong documentation to install Kong on Docker.
Follow Kong getting started documentation to create a sample route and service. We used OpenTelemetry.io as the service’s upstream application.
https://docs.konghq.com/gateway/latest/get-started/services-and-routes/
You can also configure the service and route using Kong Manager web interface.
Test your Kong setup by sending a request to your route using a browser or Postman. You should see that the response is from OpenTelemetry.io, or the upstream application you configured in example service.
Install Jaeger and enable traces
Run Jaeger on the same network you installed Kong on:
docker run --name jaeger --network kong-net ^
-e COLLECTOR_OTLP_ENABLED=true ^
-p 16686:16686 ^
-p 4317:4317 ^
-p 4318:4318 ^
jaegertracing/all-in-one:latest Follow Kong documentation and change the configuration to include traces, and restart Kong:
tracing_instrumentation = all
More configuration options can be found here.
Enable OpenTelemetry on the example service:
curl -X POST http://localhost:8001/services/example_service/plugins \ --data "name=opentelemetry" \ --data "config.endpoint=http://jaeger:4318/v1/traces"Or via plugins section on the manager web interface.
Now test your route again. You should be able to see the traces on your Jaeger.
It should look like this:
Extend OpenTelemetry
Now, let's collect the request’s headers and body! We will do it by using Kong Functions plugin, that lets you dynamically run Lua code from within Kong.
This is the Lua code that we will use:
local span = kong.tracing.start_span("payload-test")
local headers = kong.request.get_headers()
for k, v in pairs(headers) do
span:set_attribute("request.headers." .. k, tostring(v))
end
local body = kong.request.get_raw_body()
span:set_attribute("request.body",body) You can enable the post-functions plugin and add it to the config.access hook via the web interface, or write it to access.lua file and run the following:
curl -X POST http://localhost:8001/services/example_service/plugins \
-F "name=post-function" \
-F config.access[1]=@access.lua Now, use Postman to send a request and add a raw body of your choice.
Let’s check how it looks now on Jaeger!
We can now see the request body and headers in the trace.
Notice that these can contain sensitive information and should be treated accordingly.
Extending OpenTelemetry to collect request payloads
You have now successfully extended OpenTelemetry by collecting requests headers and body. This enhancement allows for more detailed monitoring and analysis, making it easier to debug issues, optimize performance, and secure your application. By integrating these extensions directly into API gateways, you simplify the data collection process, reduce the need for custom development, and enhance overall observability, providing deeper insights into system interactions.
Read more about use cases for OpenTelemetry on the Outshift Blog!
Subscribe to
the Shift!
Get emerging insights on innovative technology straight to your inbox.
