Jaeger Distributed Tracing

The invisible pathways of a microservice ecosystem are as intricate as a honey‑comb. By illuminating each corridor, Jaeger lets engineers, AI agents, and even conservationists see where the nectar of a request flows—and where it stalls.

In today’s cloud‑native world, a single user action can trigger a cascade of calls across dozens of services, containers, and serverless functions. The latency you experience in a mobile app is rarely the result of a single bottleneck; it is the sum of many tiny delays that compound as the request hops from one node to the next. Without a systematic way to capture, visualize, and analyze those hops, teams spend hours digging through logs, guessing where the problem lives, and often deploying costly “over‑engineered” fixes that merely mask the symptom.

Jaeger, the open‑source distributed tracing system originally built at Uber and now a CNCF graduated project, was created to solve exactly that problem. It records spans—the start and end of an operation—links them into traces, and stores the data in a backend that can be queried in real time. The result is a live, end‑to‑end map of request latency that can be inspected down to the millisecond. For API‑centric platforms like Apiary, where we monitor both bee‑conservation APIs and autonomous AI agents that coordinate field sensors, Jaeger becomes a shared language for performance, reliability, and accountability.

In this pillar article we will explore Jaeger from the ground up: its architecture, data model, sampling strategies, storage options, and integration patterns. We will walk through a concrete end‑to‑end example of tracing a purchase flow, compare Jaeger to alternative observability tools, and discuss how the same principles that keep a microservice mesh humming can inspire better monitoring of bee colonies and AI‑driven conservation actions. By the end, you should be able to design, deploy, and operate a production‑grade Jaeger installation that reveals hidden latency, reduces MTTR, and supports the broader mission of Apiary’s self‑governing agents.

1. What Distributed Tracing Actually Is

Distributed tracing is the practice of assigning a trace identifier to a logical request (e.g., “user clicks ‘Donate’”) and then propagating that identifier across every service that participates in fulfilling the request. Each service records a span—a named interval with a start timestamp, duration, and optional key‑value tags (e.g., http.status_code=200). Spans can be nested (parent/child) or follow a causal relationship, forming a directed acyclic graph (DAG) that represents the full execution path.

The value of tracing lies not just in measuring latency but in exposing causal relationships. If a downstream database experiences a 200 ms lock, every upstream service that called it will show the same spike, allowing you to pinpoint the true source of degradation. In the context of Apiary’s AI agents, a trace can reveal whether a sensor‑data aggregation pipeline is the bottleneck behind a delayed pollinator‑risk alert.

Concrete Numbers

• Scale at Uber – Jaeger processes > 10 million spans per day on a fleet of 4 000 services.
• Latency overhead – Benchmarks from the Jaeger team (2023) show a typical 1–2 % increase in request latency when using the default probabilistic sampler at 0.1 % rate.
• Storage cost – A Cassandra cluster storing 30 days of full‑resolution spans for a 5 k RPS service consumes ≈ 500 GB; compression and down‑sampling can reduce this to < 150 GB while preserving 99 % of latency‑critical data.

2. Jaeger Architecture – From Agent to UI

Jaeger follows a collector‑agent‑query model that decouples data ingestion from storage and retrieval. The diagram below (textual) shows the main components:

[Application] → (Jaeger client SDK) → [Jaeger Agent] → [Jaeger Collector] → [Storage Backend] → [Query Service] → [UI / API]

2.1 Jaeger Client SDK

Every language that Jaeger supports (Go, Java, Python, Node.js, Rust, etc.) provides a client library that implements the OpenTelemetry API and the Jaeger Thrift/Proto format. The SDK is responsible for:

• Generating trace and span IDs using a cryptographically safe random generator (RFC 4122‑compatible).
• Injecting the IDs into outbound carrier (HTTP headers, gRPC metadata, Kafka message headers) using the traceparent and tracestate fields defined by the W3C Trace Context spec.
• Recording tags, logs, and baggage (user‑defined key‑value pairs that propagate downstream).
• Applying sampling rules before sending the span to the local Jaeger Agent.

Typical usage in Go:

import "go.opentelemetry.io/otel"
import "go.opentelemetry.io/otel/trace"

tracer := otel.Tracer("apiary.payment")
ctx, span := tracer.Start(context.Background(), "ChargeCard")
defer span.End()

span.SetAttributes(
    attribute.String("payment.method", "credit_card"),
    attribute.Int("amount_cents", 2500),
)

2.2 Jaeger Agent

A lightweight UDP daemon that runs on the same host as the instrumented process. It buffers spans, performs batching, and forwards them to the Collector over gRPC or Thrift. Because it uses UDP by default, the Agent can absorb brief network hiccups without dropping spans—provided the buffer is sized appropriately (default 10 k spans). Production teams often run the Agent as a sidecar in Kubernetes pods, ensuring one‑to‑one proximity to the application.

2.3 Jaeger Collector

The Collector receives spans from many agents, validates them, and writes them to the configured storage backend. It can be horizontally scaled behind a load balancer (e.g., Envoy) and supports dynamic configuration via a YAML file or environment variables. Important knobs:

2.4 Storage Backends

Jaeger is storage‑agnostic. The most common options are:

Choosing a backend is a trade‑off between write durability, query latency, and cost. For Apiary’s production environment—where we need to keep a 30‑day trace history for compliance with AI‑agent audit logs—a Cassandra + Elasticsearch hybrid works well: raw spans land in Cassandra for retention, while a nightly ETL populates Elasticsearch for fast UI queries.

2.5 Query Service & UI

The Query Service reads spans from storage, assembles them into traces, and serves them via a gRPC or HTTP API. The official Jaeger UI (a React single‑page app) consumes this API to render:

• Trace timelines (Gantt chart view).
• Service dependency graphs (DOT‑format).
• Search by operation name, tags, or duration.

The UI can be embedded into other dashboards (e.g., Grafana) using the {{traceId}} variable, enabling a “drill‑through” from a latency alert to the exact trace that triggered it.

3. Instrumentation – Getting Real Data Into Jaeger

Instrumentation is the bridge between business logic and observability. A well‑instrumented system yields high‑fidelity traces without overwhelming the network or storage.

3.1 Automatic vs. Manual Instrumentation

• Automatic – Many frameworks (Spring Boot, Express.js, gRPC) have auto‑instrumentation modules that wrap incoming/outgoing calls and generate spans automatically. For example, the opentelemetry-instrumentation-aws-sdk library creates a span for each S3 GetObject call.
• Manual – Critical sections that cross language boundaries or involve complex business logic (e.g., a machine‑learning inference pipeline) often need explicit spans. This also lets you attach domain‑specific tags such as bee_species=apis_mellifera or agent_id=weather‑collector‑42.

3.2 Propagation Standards

Jaeger supports both its native Thrift format and the W3C Trace Context. Modern ecosystems gravitate toward the latter because it is language‑agnostic and works seamlessly with OpenTelemetry. A typical HTTP request will include:

traceparent: 00-0f4e5c2a9b1d4f3a8c7e9d2b6a1c3e4f-9b1d4f3a-01
tracestate: jaeger=0f4e5c2a9b1d4f3a8c7e9d2b6a1c3e4f:9b1d4f3a;rojo=1

The tracestate field can carry baggage (e.g., region=midwest) that downstream services can read for routing or policy decisions.

3.3 Example: End‑to‑End Trace of a Donation Flow

Consider a user donating to a bee‑conservation campaign through the Apiary mobile app. The request traverses the following services:

1. API Gateway (/donate) – receives HTTP request, extracts traceparent.
2. Auth Service – validates JWT, creates a child span Auth.CheckToken.
3. Donation Service – writes transaction to PostgreSQL (DB.Write).
4. Payment Processor (external) – calls Stripe via gRPC (Stripe.Charge).
5. Notification Service – publishes a message to Kafka (Kafka.Publish).
6. Analytics Service – records the event for later cohort analysis.

A simplified trace (JSON) looks like:

{
  "traceID": "0f4e5c2a9b1d4f3a8c7e9d2b6a1c3e4f",
  "spans": [
    {"spanID":"1a2b3c","operationName":"HTTP GET /donate","start":1696070400123,"duration":2150,"tags":[{"key":"http.method","value":"POST"}]},
    {"spanID":"4d5e6f","parentID":"1a2b3c","operationName":"Auth.CheckToken","start":1696070400150,"duration":78},
    {"spanID":"7g8h9i","parentID":"1a2b3c","operationName":"Donation.WriteDB","start":1696070400230,"duration":120,"tags":[{"key":"db.type","value":"postgresql"}]},
    {"spanID":"j0k1l2","parentID":"1a2b3c","operationName":"Stripe.Charge","start":1696070400350,"duration":340,"tags":[{"key":"http.status_code","value":200}]},
    {"spanID":"m3n4o5","parentID":"1a2b3c","operationName":"Kafka.Publish","start":1696070400700,"duration":55},
    {"spanID":"p6q7r8","parentID":"1a2b3c","operationName":"Analytics.Record","start":1696070400760,"duration":30}
  ]
}

When rendered in Jaeger UI, the critical path (the longest chain of spans) is API Gateway → Stripe.Charge → Kafka.Publish, indicating that the external payment provider adds ~340 ms latency. If the business SLA is 500 ms, this trace tells us we have only ~160 ms left for all other processing—a clear signal to either cache the Stripe token or negotiate a faster endpoint.

3.4 Tagging for Bee‑Conservation Context

Adding domain tags enriches trace searchability:

These tags are searchable via the Jaeger UI and can be used to generate dynamic alerts (e.g., “If latency > 200 ms for habitat=urban traces, trigger a scaling event”).

4. Sampling – Balancing Fidelity and Cost

Collecting every span in a high‑traffic system can quickly overwhelm storage and increase network load. Jaeger’s sampling mechanisms let you capture a representative subset while preserving the ability to drill down when needed.

4.1 Probabilistic Sampling

The simplest strategy: each trace is kept with probability p. The default p = 0.001 (0.1 %) yields 1 trace per 1 000 requests. The overhead is linear with p. In practice:

• Low‑traffic services (≤ 10 RPS) – set p = 1.0 to capture all traces.
• High‑traffic front‑ends (≥ 1 k RPS) – use p = 0.001 to keep storage under control.

The client SDK can be configured via environment variable:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.001

4.2 Rate‑Limiting Sampling

A token bucket approach caps the number of traces per second, regardless of request volume. This is useful when traffic spikes (e.g., a sudden surge of donations after a news article) would otherwise flood the collector. Example configuration in Jaeger Agent:

sampling:
  strategy: rate_limiting
  param: 100   # max 100 traces per second

4.3 Adaptive (Dynamic) Sampling

Jaeger can switch sampling rates based on error rate or latency thresholds. The collector evaluates incoming spans and, if a trace contains a span with error=true or duration > 500ms, it up‑samples that trace to 100 % retention. This ensures that problematic requests are fully captured for root‑cause analysis while keeping the baseline volume low.

Implementation tip: set the sampling.strategies-file to a JSON file that defines per‑service rules, e.g.:

{
  "service": "payment",
  "type": "probabilistic",
  "param": 0.005
},
{
  "service": "analytics",
  "type": "rate_limiting",
  "param": 200
}

4.4 Sampling in the Context of AI Agents

Self‑governing AI agents often generate high‑frequency telemetry (e.g., sensor readings every 100 ms). Sampling at the trace level (rather than metric level) prevents the trace store from filling up with repetitive “heartbeat” spans. Instead, agents can be configured to sample only on state changes (e.g., when a pollinator‑risk threshold crosses a boundary). This aligns with the principle of “trace what matters,” mirroring how beekeepers focus on abnormal hive behavior rather than routine activity.

5. Storage Backends – Choosing the Right Engine

The storage layer determines how long you can keep traces, how fast you can query them, and how much you spend on infrastructure. Below we dive deeper into the four most common backends, with performance numbers drawn from Jaeger’s own benchmark suite (2024).

5.1 Cassandra

• Write throughput – 150 k spans/s on a 5‑node cluster (RF=3).
• Read latency – 12 ms median for a 5‑span trace, 45 ms for a 200‑span trace.
• Retention – TTL per table; typical 30‑day policy.

Why use it? High write scalability and built‑in data replication make Cassandra ideal for high‑velocity services (e.g., API Gateway). Its column‑family model stores spans as rows keyed by trace_id and span_id, enabling efficient range scans for a given trace.

Gotchas: Requires careful tuning of compaction_strategy (TimeWindowCompactionStrategy recommended) and memtable_flush_writers. Misconfiguration can lead to write amplification and increased GC pressure.

5.2 Elasticsearch

• Query latency – 2 ms for a trace ID lookup (single shard), 30 ms for a tag‑based search across 30 M spans.
• Storage cost – ~1.2 GB per 10 M spans (with default source.enabled: false).

Why use it? Full‑text search on tags (operationName, error, custom fields) and powerful aggregation pipelines (e.g., “average latency per service per hour”). Integration with Kibana lets you create dashboards that combine traces with logs and metrics.

Gotchas: Memory‑heavy; each node needs at least 8 GB heap + 50 % of RAM for OS page cache. Shard count must be balanced—over‑sharding leads to “search throttling” under load.

5.3 Badger (Embedded KV)

• Write latency – 0.6 ms per span (single‑process).
• Maximum throughput – ~30 k spans/s on a 4‑core VM.

Why use it? Perfect for development, CI pipelines, or edge devices (e.g., a Raspberry‑Pi weather sensor that runs a tiny Jaeger Agent). No external service required; data lives on disk (LSM‑tree) and is automatically compacted.

Gotchas: No HA; a node crash loses all data unless you replicate the Badger directory via a sidecar volume.

5.4 Kafka + ClickHouse

• Ingestion – Kafka can absorb > 500 k spans/s with 3‑replica topics.
• Analytical latency – ClickHouse can run “average latency per service per day” queries in < 100 ms on 100 M rows.

Why use it? When you need real‑time streaming analytics (e.g., a dashboard that shows latency spikes within seconds) and cold‑storage for compliance (ClickHouse columns are highly compressible). The pipeline is: Agent → Collector → Kafka → ClickHouse (via clickhouse-sink connector).

Gotchas: Requires a separate stream processing layer (Kafka Streams or Flink) to assemble spans into traces before loading into ClickHouse. More operational complexity.

5.5 Decision Matrix

6. Visualizing Latency – From Gantt Charts to Dependency Graphs

Jaeger’s UI provides two primary visualizations that help you understand latency distribution.

6.1 Trace Timeline (Gantt)

Each span appears as a horizontal bar, positioned according to its start time. Overlapping bars illustrate parallelism (e.g., two microservices called concurrently). Hovering reveals:

• **Duration