Building an Observability Stack

Observability is the nervous system of any modern software platform. When you can see, measure, and understand what’s happening inside your services, you can act before a problem becomes a crisis, iterate faster, and keep the whole ecosystem—human, machine, and even the bees—healthy.

In the world of Apiary, where we monitor hive health, run self‑governing AI agents that balance pollination routes, and expose data‑rich APIs to researchers, a single missing metric or an uncorrelated log entry can mean the difference between a thriving meadow and a silent field. A robust observability stack not only protects uptime; it creates the feedback loop that drives scientific insight, operational efficiency, and responsible AI behavior.

This guide walks you through the full lifecycle of building such a stack—from the raw signals (metrics, traces, logs) to a unified dashboard that lets you diagnose, alert, and improve. You’ll find concrete numbers, real‑world tooling choices, and a case study that ties everything back to bee conservation and autonomous agents. By the end, you’ll have a blueprint you can adapt to any service, whether it’s a microservice that streams hive temperature data or a fleet of AI agents that negotiate pollination schedules.

1. Observability Foundations: Metrics, Traces, and Logs

Observability is often reduced to three pillars: metrics, traces, and logs. Each answers a different question about system behavior.

Why the three together matter

Metrics give you the “big picture” health (CPU at 78 %, request latency 120 ms). Traces let you drill down to the exact service call that caused a spike. Logs provide the context (exception stack trace, sensor payload) needed to root‑cause the issue. When you correlate them in a single dashboard, you can answer questions like:

• “Why did the latency histogram for /api/v1/hives jump from 80 ms to 250 ms at 02:15 UTC?” → trace shows a downstream AI‑scheduler call timing out; logs reveal a sensor firmware upgrade that broke the JSON schema.

This synergy is why we build a unified observability stack rather than three isolated pipelines.

2. Designing a Data Model for Observability

Before you start pulling in data, decide how you will store and query it. The data model determines performance, cost, and the ease of correlation.

2.1 Time‑Series for Metrics

Most observability platforms store metrics in a time‑series database (TSDB). The most common open‑source choice is Prometheus, which uses a label‑based model:

metric_name{label1="value1", label2="value2"} 123.45 1623234000

Key design decisions:

• Label cardinality – each unique label combination creates a separate series. A rule of thumb is to keep cardinality < 100 k series per node. In Apiary, a hive_id label (≈ 5 000 hives) is acceptable, but a sensor_id with millions of values would explode storage.
• Retention – Prometheus defaults to 15 days; for long‑term trend analysis you may need to ship to a remote storage like Thanos or Cortex, which can retain years of data at a cost of roughly $0.10/GB/month on cloud object storage.

2.2 Span Storage for Traces

Traces are stored as spans that form a directed acyclic graph (DAG). The OpenTelemetry Collector can forward spans to backends such as Jaeger, Tempo, or Google Cloud Trace. The data model includes:

• Trace ID – unique identifier for the request.
• Span ID – each operation in the request.
• Parent‑Child relationships – defines the call hierarchy.
• Attributes – key/value pairs (e.g., http.status_code=500, agent.id=42).

Jaeger’s default storage (Cassandra) can handle 10 000 traces per second with a 30‑day retention at ~1 TB. If you expect higher volume (e.g., AI agents generating 100 k traces/sec during peak pollination), consider Tempo on top of S3 which scales virtually without a hard limit.

2.3 Log Indexing

Logs are often ingested into a log aggregation system like Loki, ElasticSearch, or Splunk. Loki’s “label‑first” approach mirrors Prometheus: logs are stored under a set of stream labels (e.g., app="ai-scheduler", hive="A12"). This enables log‑to‑metric correlation with minimal indexing overhead. For high‑volume environments, a rule of thumb is 5 GB of raw log data per day per 1 k requests/second. Compression in Loki can reduce storage to 30‑40 % of raw size.

2.4 Unified Metadata

To truly correlate across pillars, you need a shared identifier. The most common practice is to propagate a trace context (trace_id, span_id) via HTTP headers (traceparent, tracestate) and embed the same IDs into logs and metrics as labels. For example:

metrics:
  hive_api_requests_total{trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 42

logs:
  {"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","msg":"request timeout","level":"error"}

When you query the dashboard for a particular trace_id, you instantly see the metric counter, the trace graph, and the relevant log lines.

3. Collecting Metrics: Instrumentation and Exporters

Collecting accurate metrics starts with instrumentation—the act of adding code that emits data.

3.1 Library Choices

• Prometheus client libraries (Go, Java, Python, Node) are the de‑facto standard. They expose an HTTP /metrics endpoint that the Prometheus server scrapes.
• OpenTelemetry Metrics API offers a vendor‑agnostic way to emit metrics that can be exported to Prometheus, StatsD, or OTLP endpoints.

For a mixed‑language stack (Go for the API gateway, Python for AI agents, Rust for the drone controller), we recommend using OpenTelemetry as the abstraction layer, with Prometheus as the concrete exporter for Go services and OTLP exporter for Python.

3.2 Example: Instrumenting Hive Temperature

var (
    hiveTemp = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "hive_temperature_celsius",
            Help: "Current temperature inside each hive",
        },
        []string{"hive_id"},
    )
)

func recordTemp(hiveID string, temp float64) {
    hiveTemp.WithLabelValues(hiveID).Set(temp)
}

The metric is scraped every 15 seconds (default Prometheus scrape interval). With 5 000 hives, that yields 5 000 × 4 = 20 k samples per minute, well within the typical Prometheus ingestion capacity of 200 k samples/second.

3.3 Exporters and Remote Write

If you need high‑availability or global view across multiple data centers, enable Prometheus remote_write to a long‑term store:

remote_write:
  - url: "https://thanos-receive.example.com/api/v1/receive"
    remote_timeout: 30s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: ".*"
        action: keep

Remote write adds ≈ 10 % overhead on network bandwidth, but it protects against node failure and enables cross‑region dashboards.

4. Distributed Tracing: Context Propagation and Sampling

Tracing shines when a request crosses service boundaries. In Apiary, a pollination request may travel through:

1. API Gateway → 2. Authentication Service → 3. AI Scheduler → 4. Drone Controller → 5. Telemetry Collector.

4.1 Propagation Standards

The W3C Trace Context specification defines two headers:

• traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
• tracestate: optional vendor‑specific data.

All services must extract these headers from inbound requests and inject them into outbound calls. In Go, the OpenTelemetry SDK does this automatically when using the httptrace middleware.

4.2 Sampling Strategies

Recording every trace can be prohibitive. Typical sampling rates:

OpenTelemetry allows dynamic sampling: increase to 10 % when error rate > 5 % for a particular endpoint.

4.3 Example: Jaeger Span Creation in Python

from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

tracer = trace.get_tracer(__name__)

@app.post("/schedule")
async def schedule_pollination(request: ScheduleRequest):
    with tracer.start_as_current_span("schedule_pollination") as span:
        span.set_attribute("hive.id", request.hive_id)
        # Call AI scheduler
        resp = await ai_client.schedule(request)
        span.set_attribute("scheduler.response_time_ms", resp.elapsed_ms)
        return resp

The span automatically inherits the trace context from the incoming request, and the hive.id attribute becomes a label you can filter on in the dashboard.

5. Log Management: Structured Logging and Retention

Logs are the most flexible pillar, but they can become a data swamp without discipline.

5.1 Structured vs. Unstructured

• Structured logs (JSON, key/value) enable automatic parsing and correlation.
• Unstructured logs (plain text) are harder to query, especially at scale.

In Apiary, we mandate JSON logs for all services. Example:

{
  "timestamp":"2026-06-11T09:12:45.123Z",
  "level":"info",
  "service":"ai-scheduler",
  "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
  "msg":"selected 12 drones for hive A12",
  "drone_count":12,
  "hive_id":"A12"
}

5.2 Log Shipping Pipelines

A typical pipeline:

1. Application → stdout (Docker container).
2. Promtail (Loki client) tails the file, adds labels (job="ai-scheduler").
3. Loki stores compressed chunks in object storage (S3).

Promtail’s config for label extraction:

scrape_configs:
  - job_name: ai-scheduler
    static_configs:
      - targets:
          - localhost
        labels:
          service: ai-scheduler
    pipeline_stages:
      - json:
          expressions:
            trace_id: trace_id
            level: level

5.3 Retention Policies

Log retention is a cost driver. A common pattern:

• Hot tier (1 TB) on fast SSD for the most recent 7 days – query latency < 1 s.
• Cold tier (object storage) for 30 days – query latency < 5 s.

With Loki, you can configure chunk_target_size and max_age to achieve a 0.5 GB per day footprint for 10 k RPS of JSON logs (assuming 200 bytes per log line). At $0.023/GB/month (S3 Standard), that’s ≈ $0.12/month per service, a modest price for searchable logs.

6. The Unified Dashboard: Correlation, Alerting, and Visualization

Having metrics, traces, and logs in separate silos defeats the purpose. A unified dashboard brings them together.

6.1 Grafana as the Hub

Grafana supports data sources for Prometheus, Tempo, and Loki simultaneously. A single panel can display:

• Time series of request latency (Prometheus).
• Overlay of trace count per minute (Prometheus metric trace_requests_total).
• Log tail filtered by the same trace_id (Loki).

Dashboard JSON example (simplified):

{
  "panels": [
    {
      "type": "graph",
      "title": "API Latency + Trace Rate",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(api_http_request_duration_seconds_bucket[5m])) by (le))",
          "refId": "A"
        },
        {
          "expr": "sum(rate(trace_requests_total[5m]))",
          "refId": "B"
        }
      ]
    },
    {
      "type": "logs",
      "title": "Related Logs",
      "targets": [
        {
          "expr": "{service=\"api-gateway\", trace_id=\"$trace_id\"}"
        }
      ]
    }
  ]
}

Grafana variables ($trace_id) are populated when you click a trace in the Tempo panel; the log panel instantly shows the correlated entries.

6.2 Alerting Rules

Prometheus alerting rules can fire on metric thresholds, but you can also attach trace‑based alerts using Grafana Alerting with OTLP data. Example rule for high error rate:

groups:
  - name: api.errors
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[2m])) / sum(rate(http_requests_total[2m])) > 0.05
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "≥ 5 % 5xx errors on API gateway"
          runbook: "[[runbook-high-error-rate]]"

When the alert fires, Grafana includes a link to the trace that triggered the spike, and the log panel pre‑filters to the same time window. This reduces mean time to detection (MTTD) from minutes to seconds.

6.3 Correlation Queries

A powerful Grafana feature is cross‑datasource queries. For instance, to find the top‑10 hives that contributed to a latency spike:

-- Prometheus query for latency spikes
SELECT hive_id, max(latency) as max_lat
FROM prom_metric
WHERE metric = 'hive_api_request_latency_seconds'
GROUP BY hive_id
ORDER BY max_lat DESC
LIMIT 10;

-- Feed hive_id into Loki log query
SELECT * FROM loki_logs
WHERE hive_id IN (result_of_above)
AND $time BETWEEN start AND end

Grafana’s Explore view lets you execute this workflow interactively, turning data into story.

7. Scaling the Stack: Storage, Performance, and Cost

A small prototype works fine on a single node, but production at Apiary’s scale (tens of thousands of hives, thousands of AI agents) demands careful planning.

7.1 Metrics Scaling

• Sharding Prometheus: Use Prometheus federation where each region runs its own Prometheus instance, and a central “global” Prometheus scrapes the /_status/targets endpoint of the regional instances.
• Chunk Compression: Prometheus stores data in XOR‑compressed blocks (~2 bytes per sample). At 100 k series with a 15 s scrape interval, you store ≈ 5 M samples per hour → ~10 GB per day after compression.

7.2 Trace Storage

• Tempo on S3: Each trace consumes ~2 KB (metadata) + payload (spans). At 10 k traces/sec, that’s 20 GB/hour raw, but S3 compression reduces it to ~6 GB/hour. At $0.023/GB, the cost is ≈ $3.30/day.
• Retention: Keep 30 days for forensic analysis, then purge. Tempo’s compact job runs daily to merge small objects, keeping request latency low.

7.3 Log Storage

• Hot vs. Cold: For 5 TB of hot logs (7 days), SSD cost on AWS is $0.10/GB/month → $51/month. Adding a cold tier of 30 TB on S3 Glacier Deep Archive ($0.00099/GB/month) adds $0.30/month.
• Retention Optimization: Use log deduplication for repetitive messages (e.g., “heartbeat received”). Loki’s stream deduplication can cut storage by 25 % in high‑frequency services.

7.4 Query Performance

• Prometheus: Query latency grows with number of series. Keep cardinality < 10 M across the cluster; otherwise consider VictoriaMetrics which scales better for high‑cardinality workloads.
• Tempo: Queries are O(N) in the number of spans per trace. For UI use, limit trace duration to 30 seconds; for backend analysis, batch‑process traces offline with Spark.

7.5 Cost Summary (2026 AWS pricing)

With careful sharding and retention, the stack can stay under $2 k/month while serving 100 k requests per second.

8. Real‑World Case Study: Monitoring Apiary’s Hive‑Tracking API and AI Swarm

Let’s walk through a concrete implementation that ties everything together.

8.1 System Overview

• Hive‑Tracking API – Go microservice exposing /api/v1/hives/{id}. Handles ~15 k RPS globally.
• AI Scheduler – Python service that decides which drones should pollinate which hives. Runs batch jobs every 5 minutes, generating up to 200 k traces per batch.
• Drone Controller – Rust service communicating with 2 k autonomous drones via MQTT. Emits high‑frequency logs (≈ 2 M lines/day).

8.2 Instrumentation Steps

1. Metrics – Added Prometheus client to API and AI Scheduler. Exported custom metrics:

• hive_api_requests_total{status="200"}
• ai_scheduler_job_duration_seconds (histogram with buckets: 0.5, 1, 2, 5, 10).

1. Tracing – Deployed OpenTelemetry Collector as sidecar for each service. Exported traces to Tempo via OTLP over gRPC. Set probabilistic sampler at 0.5 % for API, 10 % for AI Scheduler (due to batch nature).

1. Logging – All services write JSON to stdout. Promtail runs on each node, adding trace_id from incoming headers. Loki stores logs with a 7‑day hot retention.

1. Dashboard – Grafana board titled “Apiary Operations”. Panels:

• Latency heatmap (Prometheus) with overlay of trace_requests_total.
• Trace list (Tempo) filtered by hive_id.
• Log tail (Loki) showing the same trace_id.

8.3 Incident Walkthrough

Incident: At 03:14 UTC on 2026‑06‑09, the API latency jumped from 120 ms to 650 ms, and the error rate rose to 8 %.

Step 1 – Alert – Prometheus rule HighErrorRate triggered; Grafana sent a Slack webhook to #ops.

Step 2 – Correlation – The alert included a link to the Grafana dashboard with the trace_id variable pre‑filled (trace_id=9c2e...).

Step 3 – Trace analysis – The Tempo panel showed that 95 % of the offending traces were blocked on a downstream call to the AI Scheduler (/schedule).

Step 4 – Logs – Switching to the Loki panel filtered by trace_id revealed repeated log lines:

{"level":"error","msg":"Redis connection timeout","service":"ai-scheduler","trace_id":"9c2e...","redis_host":"redis-prod-01","duration_ms":1500}

Step 5 – Root cause – The AI Scheduler’s Redis client had a max idle connections set to 2, while the batch job opened 150 concurrent connections. The connection pool exhausted, causing timeouts that propagated back to the API.

Step 6 – Fix – Updated the scheduler config to maxIdleConns=100, redeployed, and observed latency return to baseline within 2 minutes.

Outcome – MTTD (Mean Time To Detect) was ≈ 30 seconds, MTTR (Mean Time To Resolve) ≈ 2 minutes – a dramatic improvement over the previous manual log‑search approach (average 12 minutes).

8.4 Lessons Learned

• Shared trace IDs are the glue that turned three disparate data sources into a single narrative.
• Probabilistic sampling at 10 % for the AI Scheduler gave enough visibility without overwhelming storage.
• Dynamic alerts that embed trace context reduce human latency dramatically.

9. Best Practices and Governance for Observability in Conservation Platforms

Observability is a technical capability, but it also carries ethical and governance responsibilities, especially when you are monitoring ecosystems and autonomous agents.

9.1 Data Minimization

Only collect data that serves a purpose. For bee health, you might record temperature, humidity, and activity counts, but not raw audio of hive sounds unless needed for a specific research hypothesis. This respects privacy of beekeepers and reduces storage costs.

9.2 Access Controls

• Role‑based access in Grafana:
• Ops – full read/write, can edit alerts.
• Researchers – read‑only dashboards, can export data.
• AI Governance – view traces for AI agents, but cannot modify agent code.

Use OAuth2 with scopes that map to these roles, and enforce them in the OpenTelemetry Collector’s access logs.

9.3 Auditing

All changes to alert rules, dashboard JSON, and collector configuration should be version‑controlled (Git) and signed with SLSA provenance. This provides an immutable audit trail, a requirement for many funding agencies that support conservation projects.

9.4 Alert Fatigue Mitigation

Define alert severity tiers and silencing windows (e.g., “maintenance window” for nightly batch jobs). Use alert deduplication in Prometheus (for: 5m) to prevent repeated firing for the same underlying issue.

9.5 Sustainability

Observability infrastructure itself consumes energy. Optimize by:

• Compressing data at the source (Prometheus block compression, Loki chunking).
• Batching trace export (OTLP batch size 100 KB).
• Auto‑scaling collectors only when ingestion exceeds thresholds (e.g., CPU > 70 %).

A well‑tuned stack can keep its carbon footprint < 0.5 kWh/day, comparable to a small server rack, while supporting a platform that protects pollinator habitats.

9.6 Community Knowledge Sharing

Publish runbooks, dashboard templates, and OpenTelemetry Collector configs as open‑source artifacts. This encourages reproducibility across conservation projects and aligns with Apiary’s mission of collaborative stewardship.

Why it matters

A unified observability stack is more than a collection of graphs and alerts; it is the feedback loop that lets us safeguard both technology and nature. By correlating metrics, traces, and logs, we detect anomalies early, debug AI agents responsibly, and keep the data pipelines that inform bee‑conservation research reliable. In practice, this means fewer lost pollination trips, faster response to sensor failures, and a trustworthy platform that researchers, beekeepers, and AI agents can rely on.

Investing in observability today pays dividends tomorrow: healthier hives, smarter AI, and a more resilient ecosystem—one that thrives because we can see, understand, and act on the data that matters.