Observability with eBPF and OpenTelemetry

In today's complex and ever-changing IT environments, observability is essential for ensuring the health and performance of your systems. By understanding what is happening in your system, you can identify and troubleshoot problems quickly, improve performance, prevent problems, and make better decisions.

Here are some of the key components of observability:

• Metrics: Metrics are measurements of the performance of your system.
• Logs: Logs are a record of events that happen in your system.
• Traces: Traces are a way of tracking the flow of requests through your system.

By collecting and analyzing these data, users can gain a deep understanding of their system and its performance. This helps in identifying and troubleshooting problems, improving performance, and preventing problems.

How to achieve observability?

This requires a significant amount of work, from choosing the right tools, instrumenting the code, collecting and storing the data for analysis, and generating insights from it. Having a technology that simplifies most of these tasks and also provides integration options with well-known monitoring providers will simplify implementing observability for your infrastructure and application performance monitoring. In this article we demonstrate using eBPF and OpenTelemetry to achieve observability without any code instrumentation.

What is eBPF?

eBPF is a technology that can run sandboxed programs in a privileged context such as the operating system kernel. It extends the capabilities of the kernel safely and efficiently without requiring changing kernel source code or load kernel modules.

eBPF observability is a new technology that allows for greater visibility and control over systems and networks. It does this by providing a way to collect and analyze data from the kernel without having to modify the kernel itself also without any instrumentation of the application code.  This makes it a powerful tool for troubleshooting performance problems, identifying security vulnerabilities, and optimizing system resources.

What is OpenTelemetry?

OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Crucially, OpenTelemetry is vendor- and tool-agnostic, meaning that it can be used with a broad variety of Observability backends, including open source tools like Jaeger and Prometheus, as well as commercial offerings. OpenTelemetry is a Cloud Native Computing Foundation (CNCF) project.

eBPF and OpenTelemetry integration

eBPF provides a powerful mechanism for dynamic tracing and analysis within the Linux kernel, while OpenTelemetry is a set of open standards and tools for collecting, exporting, and visualizing telemetry data. By combining eBPF and OpenTelemetry, you can gain deep visibility into your application's internals, as well as the underlying system, with minimal overhead. This combination enables you to collect fine-grained telemetry data from the kernel and various application layers, providing a comprehensive understanding of your system's behavior and performance.

Case study

For the eBPF capabilities demonstration, we picked the Cisco Nexus Dashboard (a platform that transforms Data Center, Cloud Network operations with simplicity, automation and analytics) and deployed eBPF/Pixie in k8s environment to demonstrate its capabilities with zero code instrumentation. We were able to collect the metrics from the infrastructure, application, protocol, cpu/memory/network and JVM stats.

On-demand Traceability

eBPF allows dynamic trace injection to effectively troubleshoot system level issues such as TCP drops, log level changes etc. The following diagram shows the map of TCP drops and retransmits across the Nexus Dashboard cluster. To see the number of drops between the pod pairs, you can hover over an edge in the map view. The color and thickness of the edges indicate an increase in the number of TCP drops.

The following diagram shows the traffic flow view of the micro-services along with incoming, outgoing traffic with request/bytes throughput and error rate metrics.

End To End Observability

eBPF along with Pixie can help navigate end to end metrics from node level to all the way to the function level. Lets take a look at how to navigate this flow.

Here, we are looking at the cluster view, that shows the list of nodes in the cluster and its CPU/memory usage along with no.of K8S pods running in each node.

We can select a node from the cluster view and drill down for the detailed view of the node. This view provides individual node's CPU/Memory/Network resources consumption.

The namespace view, which shows all the K8S namespaces, along with different metrics like, pod/service counts, disk throughput etc.

The pod lifetime resources view shows the total resource usage of a pod over its lifetime.

We can also look at the JVM stats for JVM memory management metrics for java applications.

Conclusion

In this demo, we demonstrated the observability of infrastructure and applications with eBPF by collecting metrics from the Cisco Nexus Dashboard platform without any code instrumentation. We then used the OpenTelemetry plugins available from Pixie to export these metrics to the Prometheus monitoring system to analyze the data using Grafana dashboards.

We have demonstrated how eBPF would help achieve observability for the Cisco Nexus Dashboard platform. eBPF is a powerful tool that can be used to collect data from a wide variety of systems. It is a promising technology for achieving observability in modern cloud-native systems.

Acknowledgements

References

• eBPF
• Pixie
• Grafana
• Prometheus
• OpenTelemetry
• Cisco Nexus Dashboard