Observability Pyramid

By Apiary’s Observability Team

Introduction

In the modern cloud‑native era, a single user request can travel across dozens of microservices, jump through load balancers, and be transformed by serverless functions before it finally returns a response. When any one of those hops fails, the symptom that reaches the end‑user is often a vague “timeout” or a generic “500 error.” Without a systematic way to see what happened, where it happened, and why it happened, engineers spend hours—sometimes days—reproducing the problem in a sandbox instead of fixing it in production.

Observability solves that dilemma by turning raw telemetry into a coherent, searchable narrative. The most widely accepted mental model is the Observability Pyramid, where metrics form the base, logs sit in the middle, and traces crown the structure. Each layer adds resolution and context, and together they enable you to correlate signals across a distributed system, pinpoint root causes, and build automated remediation loops. For Apiary, whose mission is to protect pollinator habitats while supervising fleets of self‑governing AI agents, the stakes are concrete: a mis‑behaving data‑collector can misreport hive health, a latency spike can delay a pesticide‑avoidance alert, and a cascade failure can jeopardize an entire conservation campaign.

This article is a deep dive into the three pillars of observability, the mechanics that bind them, and the practical steps you can take to build a reliable pyramid on top of your own services. We’ll sprinkle in real numbers, concrete tooling choices, and a few analogies to bee colonies and swarm intelligence—because the same principles that keep a hive thriving also keep a distributed system healthy.

1. Foundations of Observability

1.1 From Monitoring to Observability

Traditional monitoring is a reactive discipline: you define thresholds (e.g., CPU > 80 %) and receive alerts when they are breached. Observability, by contrast, is proactive and exploratory. It asks whether the system’s internal state is sufficiently exposed to answer any arbitrary question an operator might have. In practice this means:

The difference is comparable to a bee scout versus a forager. A scout monitors the environment for any sign of danger (monitoring), while the whole colony maintains a shared “honey‑comb map” of nectar sources, temperature, and pheromone levels that lets any bee instantly locate resources (observability).

1.2 The Pyramid Analogy

The pyramid’s three tiers reflect both granularity and cost:

1. Metrics – Low‑volume, high‑level numbers (e.g., request latency percentiles). They are cheap to store, fast to query, and ideal for dashboards.
2. Logs – Structured text streams that capture events with timestamps and context (e.g., “user‑id=1234 ; action=login_failed”). They are richer than metrics but larger in size.
3. Traces – End‑to‑end request graphs that stitch together spans across services. They provide the highest resolution, showing the exact path a request took.

Each level can be independently useful, but the true power emerges when you correlate them. A latency spike (metric) can be drilled down into the specific request IDs (trace) that contributed, and then the logs for those spans can reveal the exact exception message.

1.3 Core Principles

These pillars are the same rules that a bee colony follows: every bee carries the colony’s scent (context), the hive maintains a clean comb (data hygiene), and the queen ensures each brood cell is provisioned (development discipline).

2. The Metrics Layer

2.1 What Metrics Are

Metrics are numerical time‑series that describe a system’s health. They can be counters (monotonically increasing, e.g., http_requests_total), gauges (instantaneous values, e.g., memory_usage_bytes), or histograms (distribution buckets, e.g., request_latency_seconds). A well‑instrumented service typically emits 10–30 distinct metrics per endpoint; a high‑traffic API can generate millions of data points per minute.

Concrete example: Apiary’s “Hive‑Status” endpoint receives an average of 2,500 requests/second during peak pollination season. Using a histogram with exponential buckets (0.005, 0.01, 0.02, …, 5.0 seconds), we can compute the 99th‑percentile latency in real time. If latency exceeds 300 ms (the SLA), an alert fires.

2.2 Instrumentation Mechanics

The most common open‑source stack today is Prometheus (scrape‑based) + OpenTelemetry (instrumentation library). The workflow is:

1. Add a SDK (@opentelemetry/api-metrics for Node.js, opentelemetry-java for Java, etc.).
2. Create meters and define instruments (counter, gauge, histogram).
3. Record values at the point of interest (e.g., before and after DB query).
4. Expose an endpoint (/metrics) that Prometheus scrapes every 15 seconds.

Code snippet (Go):

var (
    requestLatency = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "apiary_hive_status_latency_seconds",
            Help:    "Latency of Hive‑Status endpoint",
            Buckets: prometheus.ExponentialBuckets(0.005, 2, 12), // 0.5 ms → 10 s
        },
        []string{"method", "status"},
    )
)

func handler(w http.ResponseWriter, r *http.Request) {
    timer := prometheus.NewTimer(requestLatency.WithLabelValues(r.Method, "200"))
    defer timer.ObserveDuration()
    // …handle request…
}

2.3 Aggregation and Alerting

Metrics are cheap enough to store at high resolution for 30 days, then down‑sample for longer retention. A typical Prometheus cluster can ingest 10 k samples/second per node with < 2 % CPU overhead. For large fleets, you’ll likely need Thanos or Cortex to provide horizontal scalability and long‑term storage on object stores (e.g., Amazon S3).

Alert rule example (PromQL):

# Alert when 99th‑percentile latency > 300 ms for 5 minutes
histogram_quantile(0.99, sum(rate(apiary_hive_status_latency_seconds_bucket[5m])) by (le)) > 0.3

When this condition fires, the alert manager routes the notification to Slack, PagerDuty, and the incident‑response playbook, ensuring the right bee‑keeper (on‑call engineer) is awoken.

2.4 Metrics in the Context of Bees

A beehive’s temperature is monitored with a thermometer (metric). The hive maintains a target range of 34‑35 °C; if the temperature drifts beyond ±0.5 °C, the colony activates ventilation or heating. Similarly, your service monitors latency, error rate, and CPU, and automatically triggers mitigation (e.g., autoscaling) to keep the system in its “optimal temperature.”

3. The Logs Layer

3.1 Structured Logging Fundamentals

Logs capture event‑level detail: timestamps, severity, component, and a payload of key‑value pairs. Moving from free‑form text to structured logs (JSON or protobuf) yields two immediate benefits:

1. Machine‑readability – enables powerful queries (json.payload.user_id = 1234).
2. Consistent schema – simplifies downstream indexing and retention policies.

A good rule of thumb: log no more than 1 KB per request. At Apiary’s traffic level, that translates to about 2 GB of log data per hour, a manageable volume for a modern log aggregation system.

3.2 Choosing a Log Collector

For a Kubernetes‑native deployment, Fluent Bit deployed as a DaemonSet is the most cost‑effective choice. It tails container stdout, adds the trace‑id from the request context, and ships to a Loki cluster.

3.3 Correlating Logs with Metrics

When a metric alarm fires, you can auto‑populate a log query using the same label set. For example, the latency alert above includes a method label. The incident response tool can automatically run:

{app="apiary-hive-status", method="GET"} |~ "error"
| json | line_format "{{.timestamp}} {{.level}} {{.msg}}"
| limit 100

This returns the most recent error‑level logs for the same HTTP method, instantly narrowing the search space.

3.4 Log Retention and Cost

Logs are typically compressed with gzip (average 70 % reduction) and stored for 30 days hot and 90 days warm. With Loki’s chunk‑based storage, a 2 TB raw log volume can be reduced to ≈ 600 GB after compression, costing roughly $0.02/GB/month on AWS S3. This is still an order of magnitude more expensive than metrics, reinforcing the pyramid’s cost hierarchy.

3.5 Bees and Log Hygiene

Think of logs as the pollen grains collected by foragers. Each grain carries a tiny piece of information about a flower (event). If you only count the grains (metrics), you miss the exact species of flower (log details). By cataloguing each grain with its plant species, you can later analyze which flowers contributed most to the hive’s honey, just as you can trace which request contributed to a latency spike.

4. The Traces Layer

4.1 Distributed Tracing Basics

A trace represents the lifecycle of a single request as it traverses multiple services. It is composed of spans, each describing an operation (e.g., “SQL query”, “HTTP GET”). Spans carry:

• Start timestamp and duration
• Attributes (e.g., db.system=postgresql, http.status_code=200)
• Events (e.g., “error”, “retry”)
• Parent‑child relationships (forming a DAG)

The OpenTelemetry specification defines a trace ID (128‑bit) and a span ID (64‑bit). Propagation is done via HTTP headers (traceparent, tracestate) or messaging metadata.

4.2 Sampling Strategies

Tracing every request can be prohibitive. Three common sampling techniques:

Apiary’s production environment uses a hybrid sampler: 0.05 % of all requests plus 100 % of any request that exceeds the latency SLA or returns a 5xx error. With ~2,500 RPS, this yields ≈ 1.25 traces/sec on average, well within the ingestion capacity of Jaeger.

4.3 Tracing Backends

For a cloud‑native stack, Tempo paired with Grafana offers the best long‑term storage cost (object store pricing) while keeping query latency under 2 seconds for traces up to 24 hours old.

4.4 End‑to‑End Example

1. Client sends an HTTP GET to apiary.com/hive/42/status. The client library injects traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01.
2. API Gateway extracts the trace ID, creates a root span, and forwards the request downstream with the same header.
3. Hive Service creates a child span GET /hive/42/status, records a DB query span (SELECT * FROM hives WHERE id=42), and logs an event cache_miss.
4. DB Proxy creates a span for the actual PostgreSQL call, includes db.statement attribute, and returns the result.
5. All spans are exported via the OpenTelemetry collector to Tempo. The trace appears in Grafana’s Explore > Traces view, where you can see the exact latency contributed by the DB call (12 ms) versus the network hop (3 ms).

4.5 Bees Analogy

A trace is like a waggle dance: a bee communicates the exact direction, distance, and quality of a nectar source to the rest of the colony. The dance includes timing (duration), direction (span hierarchy), and intensity (attributes). By watching the dance, the hive can instantly locate the best flowers—just as you can locate the bottleneck in a request by following its trace.

5. Correlating Signals – The Pyramid in Action

5.1 The “Three‑Signal Join” Query

When a latency alert fires, the goal is to answer three questions:

1. Which request(s) contributed? – Use traces to retrieve the trace IDs.
2. What did those requests do? – Inspect spans to see slow components.
3. What did the code log? – Pull logs for the same trace ID.

A practical implementation uses Grafana’s built‑in variables. Example dashboard:

• Panel A – Metric: 99th‑percentile latency (histogram_quantile).
• Panel B – Table: trace_id list from Tempo (via the traceql query select trace_id where duration > 0.3s).
• Panel C – LogQL query: {trace_id="$trace_id"} feeding Loki.

When the user clicks a spike in Panel A, the dashboard automatically populates $trace_id and shows the exact spans and logs for that request. This “click‑through” experience reduces MTTR (Mean Time To Recovery) from hours to minutes.

5.2 Real‑World Incident Walkthrough

Scenario: At 02:15 UTC, the API’s latency SLA is breached for the “Hive‑Status” endpoint.

Step 1 – Metric Alert: Prometheus fires LatencyHigh alert. PagerDuty notifies the on‑call engineer.

Step 2 – Trace Retrieval: The engineer opens the Grafana dashboard, which auto‑populates the top 5 trace IDs that contributed to the 99th‑percentile latency.

Step 3 – Span Analysis: The traces reveal that Service B (hive‑analytics) spent 120 ms in a redis GET call, whereas historically it averages 15 ms.

Step 4 – Log Inspection: Using the trace ID, the engineer queries Loki and finds a log line:

2026-06-18T02:15:12Z level=error msg="Redis connection timeout" host=redis-01.region1.aws.com retry=3

Step 5 – Root Cause: The Redis node had a network partition after a recent security patch. The patch unintentionally disabled the EC2 instance’s ENI (Elastic Network Interface) secondary IP.

Step 6 – Remediation: The engineer triggers an automated rollback of the patch, and the system automatically recovers. The latency metric returns to baseline within 5 minutes.

Metrics after fix: 99th‑percentile latency drops from 350 ms back to 80 ms; error rate goes from 2 % to < 0.1 %.

This incident demonstrates the speed and precision that the pyramid provides: a single alert leads to a trace‑driven drill‑down, a log‑driven confirmation, and a rapid fix.

5.3 Automation with the Pyramid

Beyond manual debugging, the pyramid can power self‑healing loops:

• Rule: If redis_get_latency_seconds_bucket{le="0.05"} < 0.8 (i.e., > 20 % of calls exceed 50 ms) AND trace_error_count > 5 for the same trace ID, then trigger a circuit_breaker on the dependent service.
• Implementation: Use OpenTelemetry’s Metrics Bridge to expose trace‑derived metrics, then feed them into Prometheus rules. The alert manager can invoke a Kubernetes Job that runs a remediation script (e.g., restart the Redis pod).

Such observability‑as‑code patterns enable the same feedback loop that a bee colony uses: pheromone signals (metrics) trigger forager behavior changes (autoscaling), while scout reports (traces) inform the colony’s strategy.

6. Instrumentation Strategies for Distributed Systems

6.1 Language‑Specific SDKs

For each language, enable auto‑instrumentation where possible to avoid missing spans. Manual instrumentation should be reserved for business‑critical code paths (e.g., AI model inference).

6.2 Context Propagation Best Practices

• Never generate a new trace ID in downstream services; always extract from incoming request headers.
• Pass the context explicitly (e.g., via function arguments) rather than relying on thread‑local storage, especially in async frameworks.
• Include the trace ID in log fields (trace_id, span_id) using a logging hook. In Go, use logrus’s AddHook to inject the IDs automatically.

6.3 Sampling Configuration

A practical configuration for a high‑traffic service:

sampler:
  type: parentbased_always_on
  remote_sampling: true
  max_attributes: 128
  max_events: 32
  max_links: 16
  head_sampling:
    probability: 0.0005   # 0.05 % of all requests
  tail_sampling:
    error_rate: true
    latency_threshold_seconds: 0.3

This YAML can be loaded by the OpenTelemetry Collector, enabling dynamic adjustment without redeployments.

6.4 Observability in AI Agents

Self‑governing AI agents (e.g., Apiary’s “Pollination Planner”) often run batch jobs that process sensor data and produce recommendations. For these workloads:

• Metrics: Track job queue depth, processing time, model inference latency, and success rate.
• Logs: Emit structured events for each major decision (decision=apply_pesticide_avoidance, confidence=0.92).
• Traces: Wrap the entire job execution in a root span, with child spans for each model stage (pre‑processing, inference, post‑processing).

Because AI agents can be non‑deterministic, capturing the random seed and model version as trace attributes is crucial for reproducibility and auditability.

7. Data Retention, Sampling, and Cost Management

7.1 Tiered Storage Architecture

Implement this with Prometheus + Thanos for metrics, Loki for logs, and Tempo for traces. Each component can point to a separate S3 bucket with lifecycle rules that automatically transition objects from hot to warm to cold storage.

7.2 Cost Estimation Example

Assume the following daily ingestion rates:

• Metrics: 15 M samples/day → 0.5 GB raw → $0.01/day (hot), $0.001/day (cold)
• Logs: 2 GB/day → $0.04/day (hot), $0.01/day (cold)
• Traces: 0.7 GB/day (compressed) → $0.02/day (hot), $0.005/day (cold)

Total monthly cost ≈ $45 for hot storage + $20 for long‑term storage. This is a fraction of the cost of a typical incident (average $13k per outage per hour).

7.3 Sampling Policies to Control Costs

• Metrics: Down‑sample after 30 days using Prometheus’s record rules (e.g., store only 5‑minute averages).
• Logs: Apply log throttling (max_age=1h, max_size=200MB) on edge collectors to avoid flooding the backend with duplicate debug logs.
• Traces: Use tail sampling to retain only error‑related traces beyond a certain latency.

By aligning sampling thresholds with business SLAs, you keep the pyramid affordable while retaining the ability to investigate rare, high‑impact events.

8. Real‑World Case Study: Apiary’s Honey‑Comb Service

8.1 System Overview

The Honey‑Comb service aggregates sensor data from over 12,000 beehives worldwide, normalizes it, and serves it to the Hive‑Dashboard UI. Its architecture consists of:

1. Ingress Gateway (Envoy) – terminates TLS and forwards to services.
2. API Service (Go) – handles REST endpoints.
3. Processor Service (Python) – runs a TensorFlow model for hive health prediction.
4. Cache Layer (Redis) – stores recent predictions.
5. PostgreSQL – persists raw sensor readings.

During the 2025 pollination season, the service processed 1.2 B sensor records per day, equivalent to ≈ 14 k RPS.

8.2 Observability Implementation

All components forward telemetry to a central OpenTelemetry Collector that fans out to Prometheus, Loki, and Tempo.

8.3 Incident Narrative

Date: 2026‑04‑12, 03:42 UTC Problem: API latency spiked to 1.2 seconds (4× SLA) for the /hive/summary endpoint.

Investigation Steps:

1. Metric Alert: api_latency_seconds 99th‑percentile > 1 s for 2 minutes.
2. Trace Extraction: Top 3 trace IDs showed a processor span lasting 950 ms.
3. Span Details: The processor span had an attribute model_version=2026.03. The span’s child model_inference lasted 920 ms, far beyond the normal 120 ms.
4. Log Query: Using the trace ID, Loki returned:

   2026-04-12T03:42:13Z level=error msg="TensorFlow out‑of‑memory" model_version=2026.03 batch_size=256

1. Root Cause: A recent firmware update added two new sensors per hive, increasing input size from 128 KB to 256 KB per inference. The model’s memory allocation was not increased accordingly, causing frequent OOM errors and GPU thrashing.
2. Remediation: Deployed a hot‑fix that reduced batch_size to 128 for the affected version and scheduled a model retraining with the larger input size.

Outcome: Latency returned to 80 ms within 7 minutes. The incident cost was limited to a brief SLA breach (no financial penalties) thanks to rapid detection.

8.4 Lessons Learned

• Cross‑signal correlation (metric → trace → log) reduced MTTR from an estimated 2 hours (if only metric alerting were used) to 7 minutes.
• Uniform trace IDs across Go and Python services were essential; a prior bug where the Python service generated a new trace ID broke the chain.
• Sampling: The hybrid sampler ensured the problematic trace was captured despite the low head‑sampling rate.
• Bee‑analogy: The incident mirrors a hive where a sudden influx of pollen (new sensors) overloads the foragers (model inference), forcing the colony to temporarily reduce foraging intensity until the hive adapts.

9. Tools and Open‑Source Ecosystem

All of these tools are cloud‑agnostic and can be deployed on‑premises or on public clouds. The choice often boils down to operational maturity: a small team may start with a single‑node Prometheus + Loki + Tempo stack, then scale out with Thanos and Cortex as traffic grows.

10. Future Directions – AI‑Augmented Observability

Observability is entering a new phase where AI agents assist in signal correlation, anomaly detection, and even automated remediation. Some emerging trends:

1. Anomaly Detection with ML – Services like Grok, Anodot, or open‑source Prometheus‑ML train models on metric time‑series to detect subtle drifts that static thresholds miss.
2. Root‑Cause Recommendation Engines – By ingesting historic incident data, a model can suggest the most likely cause (e.g., “Redis latency” vs. “DB connection pool”) given a new alert signature.
3. Self‑Healing Controllers – Kubernetes operators can be driven by observability signals. For example, a custom controller watches for a high error rate metric and automatically rolls back a deployment.
4. Explainable Traces – Projects like OpenTelemetry AI aim to summarize a trace into a natural‑language paragraph (“The request spent 850 ms in model inference due to GPU memory pressure”).

For Apiary, integrating an AI‑augmented observability layer could enable the platform to automatically adjust sensor sampling rates based on environmental conditions, ensuring that data collection never overwhelms downstream pipelines while preserving scientific fidelity.

Why It Matters

A robust observability pyramid is not a luxury; it is the nervous system of any distributed application. It gives you the ability to:

• Detect problems before they cascade into outages.
• Diagnose with pinpoint accuracy, reducing mean time to resolution.
• Learn from each incident, feeding insights back into system design and AI models.
• Protect the mission‑critical work of bee conservation, where a delayed alert could mean a lost pollination window or an unnecessary pesticide application.

By treating metrics, logs, and traces as complementary lenses rather than isolated silos, you empower your team—and your autonomous agents—to act with the same coordinated precision that a honeybee colony displays every day. The result is a resilient, transparent platform that can scale to the global challenge of preserving the planet’s pollinators while embracing the next generation of self‑governing AI.