In the Getting Started Series opener, I referenced several links to help you get up to speed on the architecture of OpenTelemetry. If you didn't get to check that blog out, here it is: https://outshift.cisco.com/blog/opentelemetry-getting-started-series 

Goal

This post provides a quick-start guide for deploying the basic components of OpenTelemetry (OTel), a way to interact with the tracing output (via Jaeger) and generate traces using a sample Spring PetClinic service that has OTel auto-instrumentation enabled.

This document does not go into a granular explanation of the OTel, Jaeger, and Spring Framework components. We will break the OTel components down into detail in future blogs.

Purpose

This document provides how-to steps to:

• Deploy a KinD Kubernetes cluster
• Deploy an OpenTelemetry (OTel) Collector
• Deploy a Jaeger All-in-One environment
• Deploy a sample application using the Spring PetClinic

Deployment Overview

Figure 1 illustrates the basic Kubernetes services, deployments, and pods used in this setup. Note: Not all components are shown, including the Jaeger and OTel operators.

Figure 1. Kubernetes Resource Overview

As shown in Figure 1, the Jaeger all-in-one deployment includes multiple services that represent Jaeger components such as the query, collector, agent (not shown), and headless collector (not shown). These components are managed via the Jaeger operator. In addition, the OTel Operator manages the various OTel collector components, which include the collector service and pod. Finally, the Spring PetClinic sample service leverages the OTel auto-instrumentation library for Java (discussed later).

Deployment Steps

1. Create a KinD Cluster: (https://kind.sigs.k8s.io/docs/user/quick-start/)
2. Deploy Cert Manager: (https://cert-manager.io/docs/installation/)
3. Deploy the Jaeger Operator: (https://github.com/jaegertracing/jaeger-operator)
4. Deploy the Jaeger all-in-one model
5. Deploy the OTel Operator: (https://github.com/open-telemetry/opentelemetry-operator)
6. Deploy the OTel Collector and aim the exporter towards the Jaeger all-in-one collector service.
7. Deploy the OTel auto-instrumentation CRD and set the endpoint as the OTel collector service.
8. Deploy the Spring PetClinic sample service and enable sidecar injection of the OTel Java auto-instrumentation library.
9. Check for service tracing of the sample service.

Looks easy, right? Let's jump into the details.

Deployment Walk-thru

Create a KinD cluster (see the link above to install KinD on your machine):

kind create cluster

Deploy Cert Manager (always check for the latest release):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

Deploy the Jaeger Operator (always check for the latest release):

kubectl create namespace observability
kubectl apply -f  https://github.com/jaegertracing/jaeger-operator/releases/download/v1.42.0/jaeger-operator.yaml -n observability

Deploy the Jaeger All-in-One Strategy:

kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
EOF

In another terminal session, port forward to the Jaeger simplest-query service on port 16686:

kubectl port-forward svc/simplest-query 16686:16686

Deploy the OTel Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Deploy the OTel Collector:

In this setup, the specific exporter configuration uses the previously deployed Jaeger collector and the endpoint connection uses the service name ("simplest-collector") and port of 14250. Ensure that the "insecure" flag is true. Additionally, the pipeline definition uses OTLP (OpenTelemetry Protocol) as the receiver protocol and Jaeger as the exporter.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s
    exporters:
      logging:
      jaeger:
          endpoint: "simplest-collector:14250"
          tls:
              insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [jaeger]
EOF

Deploy the OTel Java Auto-instrumentation CRD:

The auto-instrumentation configuration points the library towards the previously deployed OTel collector service (otel-collector) on port 4317 and references the image locations for each language type.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
EOF

Deploy the Spring PetClinic Sample Service:

In this example, the Spring PetClinic sample application deployment uses the auto-instrumentation configuration to perform a sidecar injection of the OTel Java library. In future blog posts, we will walk through other methods of instrumenting an application.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-petclinic
spec:
  selector:
    matchLabels:
      app: spring-petclinic
  replicas: 1
  template:
    metadata:
      labels:
        app: spring-petclinic
      annotations:
        sidecar.opentelemetry.io/inject: "true"
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: app
        image: ghcr.io/pavolloffay/spring-petclinic:latest
EOF

We will get into how manual and auto-instrumentation works in a future blog post, but let's take a quick look at how auto-instrumentation works in this example.

In the annotation shown above, we have indicated that we are using the sidecar method to spin up an init container when the spring-petclinic pod(s) are deployed and we want to inject the Java auto-instrumentation code into the spring-petclinic container. So, let's look at the pod with the two containers and what the injection is doing to the spring-petclinic container.

Find the name of your spring-petclinic pod and run a 'kubectl describe' on it. I have removed a bunch of extra output to focus on the two containers we want to discuss.

The first thing to note is that the annotations from our config above appear in the pod annotation section, and then we see two containers listed along with some environment data. The first container, the auto-instrumentation init container, is used to copy the '/otel-auto-instrumentation/javaagent.jar' file into the 'main body' container (in our case, the spring-petclinic container) and then run as a 'javaagent' (more on that in a second).

In the Environment section, there is the JAVA_TOOL_OPTIONS line that refers to the jar file that is copied and then executed in the spring-petclinic container. The other line to note is the OTEL_EXPORTER_OTLP_ENDPOINT line. That line indicates where the spring-petclinic container and the javaagent library will send the trace data (over OTLP). That endpoint is the OTel collector we deployed in a previous step.

# kubectl describe pod spring-petclinic-79f4794dd9-mjbfn

Name:             spring-petclinic-79f4794dd9-mjbfn
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-control-plane/172.18.0.2
Start Time:       Tue, 28 Feb 2023 10:28:11 -0500
Labels:           app=spring-petclinic
                  pod-template-hash=79f4794dd9
Annotations:      instrumentation.opentelemetry.io/inject-java: true
                  sidecar.opentelemetry.io/inject: true

Init Containers:
  opentelemetry-auto-instrumentation:
    Container ID:  containerd://310fbcf762dcb592f84330a7a9583174852a6abcdad505a9e4baa3cc78963ede
    Image:         ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
    
Containers:
  app:
    Container ID:   containerd://32e0056ea02bb749654aeb310fb59bf46055fdc94470f5a54592e5a59b0bf55a
    Image:          ghcr.io/pavolloffay/spring-petclinic:latest
    Environment:
      JAVA_TOOL_OPTIONS:                    -javaagent:/otel-auto-instrumentation/javaagent.jar
      OTEL_SERVICE_NAME:                   spring-petclinic
      OTEL_EXPORTER_OTLP_ENDPOINT:         http://otel-collector:4317
      OTEL_RESOURCE_ATTRIBUTES_POD_NAME:   spring-petclinic-79f4794dd9-mjbfn (v1:metadata.name)
      OTEL_RESOURCE_ATTRIBUTES_NODE_NAME:   (v1:spec.nodeName)
      OTEL_PROPAGATORS:                    tracecontext,baggage,b3
      OTEL_TRACES_SAMPLER:                 parentbased_traceidratio
      OTEL_TRACES_SAMPLER_ARG:             0.25
      OTEL_RESOURCE_ATTRIBUTES:            k8s.container.name=app,k8s.deployment.name=spring-petclinic,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=spring-petclinic-79f4794dd9

Now, let's look at the log for the spring-petclinic container and see what the init container did. Use the 'kubectl logs' command to view the logs for the spring-petclinic container.

The first line shows the javaagent running the auto-instrumentation code in the spring-petclinic container. The process copies the jar file and executes it according to the information in the Environment section above. After that, the usual business logic of the petclinic runs.

# kubectl logs spring-petclinic-79f4794dd9-mjbfn

Picked up JAVA_TOOL_OPTIONS:  -javaagent:/otel-auto-instrumentation/javaagent.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2023-02-28 15:28:34:943 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.23.0


              |\      _,,,--,,_
             /,`.-'`'   ._  \-;;,_
  _______ __|,4-  ) )_   .;.(__`'-'__     ___ __    _ ___ _______
 |       | '---''(_/._)-'(_\_)   |   |   |   |  |  | |   |       |
 |    _  |    ___|_     _|       |   |   |   |   |_| |   |       | __ _ _
 |   |_| |   |___  |   | |       |   |   |   |       |   |       | \ \ \ \
 |    ___|    ___| |   | |      _|   |___|   |  _    |   |      _|  \ \ \ \
 |   |   |   |___  |   | |     |_|       |   | | |   |   |     |_    ) ) ) )
 |___|   |_______| |___| |_______|_______|___|_|  |__|___|_______|  / / / /
 ==================================================================/_/_/_/

:: Built with Spring Boot :: 2.5.4

In an new terminal session, port forward to the Spring PetClinic deployment on port 8080:

kubectl port-forward deployment.apps/spring-petclinic 8080:8080

Verification:

Drive this section on your own. You can't break anything, so have a blast!

Note: Before starting this section, make sure both of your port forwarding sessions are still running (8080 and 16686).

Verify that the Spring PetClinic service works by opening a browser to http://localhost:8080. Click through various links in the sample service to generate traffic and traces.

Verify that the Jaeger all-in-one deployment works by opening a browser to http://localhost:16686. After a few seconds (you may need to refresh the browser page), the Jaeger UI will show two services in the "Search" panel on the left. Select the "spring-petclinic" service and then click "Find Traces" at the bottom of the Search panel. If you have yet to use the PetClinic UI, there may only be basic single span results in the search. Click around the PetClinic UI to generate more spans.

Click through the spans and expand the "Tags" and "Process" sections to see the details of each span.

Walkthru of Jaeger and Spring PetClinic UIs:

In this section, we will look at the PetClinic UI and then dig into a couple of areas of the Jaeger UI to visualize some trace elements - the trace ID and span ID.

In the Spring PetClinic UI (http://localhost:8080/), click on the "Find Owners" tab at the top as shown in Figure 2.

Figure 2. Spring PetClinic - Find Owners

In the Jaeger UI (http://localhost:16686/), select "spring-petclinic" in the "service" field and click "Find Traces". A trace result should show something similar to "spring-petclinic: GET /owners" as shown in Figure 3. Click on the trace result.

Figure 3. Jaeger - Get Owners

After clicking on the trace result of GET /owners, you will see the trace detail view as shown in Figure 4. This view shows you the trace and span summary for GET /owners. This is an excellent view for quickly visualizing the call flow of the trace and each span that makes up the trace. You can also see the per-span timeline, which is great for quickly determining if a specific span has an issue that is causing high latency.

Figure 4. Jaeger - Trace Detail View

Click on one of the spans to get detailed information about the process in that span. For example, as shown in Figure 5, the span detail view shows all kinds of information about OpenTelemetry, the code function, and even the infrastructure environmental information such as the Kubernetes node info (node, pod, operating system).

Figure 5. Jaeger - Span Detail

Now, check out the logs of the OTel collector to verify its connection to Jaeger.

Verify that the OTLP receiver is started and the connection from OTel to the Jaeger is ready:

kubectl logs deployment.apps/otel-collector

The output should show "Receiver started":

builder/receivers_builder.go:73	Receiver started.<strong>	</strong>{"kind": "receiver", "name": "otlp"}

And the backend is "READY":

jaegerexporter@v0.41.0/exporter.go:186	State of the connection with the Jaeger Collector backend	{"kind": "exporter", "name": "jaeger", "state": "READY"}

Figure 6 illustrates the connection flow for each of the major components (Jaeger, OTel Collector and Spring Pet Clinic service).

Figure 6. Connection Flow

On the Jaeger All-in-One pod, check the connections to the query service (16686) and the OTel Collector -to- Jaeger connection (14250). Note: Output of netstat command has been summarized. There is an incoming connection from the local browser to the port forwarded service on 16686. There is also and incoming connection from the OTel Collector on port 14250.

# kubectl exec -it simplest-797dd8fc67-6l9q4 -- netstat -at

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:47280         localhost:16686         ESTABLISHED
tcp        0      0 simplest-797dd8fc67-6l9q4:14250 10-244-0-11.otel-collector.default.svc.cluster.local:49768 ESTABLISHED

I am a day-in-the-life of a connection type of guy, so, lets verify the connection between the Spring PetClinic pod and the OTel Collector. Note: The default image does not include any network connection tools (e.g., netstat, lsof, etc.). Connect to a shell on the Spring PetClinic pod and install net-tools:

# kubectl exec -it spring-petclinic-79f4794dd9-mjbfn -- /bin/bash
apt update
apt install net-tools -y

Check the connections. The two primary connections to look for are the connections from the browser to the PetClinic UI (8080) and the connection from the Java OTel library process to the OTel Collector (4317):

root@spring-petclinic-79f4794dd9-mjbfn:/# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:55028         localhost:8080          ESTABLISHED
tcp        0      0 10.244.0.12:34208       otel-collector.def:4317 ESTABLISHED

That's it! You did it! You are now on your way to being an OpenTelemetry guru.

In future blogs, we will tear apart some of what we glossed over and explain how some of these components work, how to modify their configurations for specific use cases, and how to mix in different OTel SDKs (node.js, python, etc.). We will also talk about exporting traces, metrics, and logs to different back-ends and much more!

Shannon McFarland is a Distinguished Engineer and open-source advocate at Outshift, formerly Cisco’s Emerging Technology & Incubation organization. You can follow him on Twitter @eyepv6.