Prometheus Metrics Collection

Introduction

In the modern cloud‑native ecosystem, observability is no longer a luxury—it’s a prerequisite for reliable, scalable services. Prometheus has become the de‑facto standard for time‑series monitoring, offering a powerful data model, a flexible query language (PromQL), and a robust alerting pipeline. Yet the true power of Prometheus is unlocked only when metrics are named, labeled, and alerted on with discipline. A sloppy naming convention can explode label cardinality, degrade query performance, and drown teams in false alarms. Conversely, a well‑engineered metric strategy turns raw numbers into actionable insight, enabling rapid response to incidents and long‑term capacity planning.

For Apiary, where we track everything from hive temperature to the health of autonomous pollination drones, the stakes are tangible. A sudden spike in failed API calls could indicate a network outage that leaves a beehive without its monitoring feed, while an unnoticed rise in latency might cause an AI agent to misjudge the optimal time to deploy a pollination swarm. By mastering Prometheus labeling and alerting best practices, we safeguard both our digital infrastructure and the living ecosystems it supports.

This guide dives deep into the mechanics of Prometheus metrics collection, offering concrete, production‑grade advice. We’ll explore the data model, label hygiene, metric types, instrumentation patterns, exporter strategies, and the art of crafting meaningful alerts. Wherever possible, we’ll draw parallels to bee conservation and self‑governing AI agents—showcasing how rigorous observability fuels responsible stewardship of nature and technology alike.

1. The Prometheus Data Model: Foundations for Clean Metrics

Prometheus stores data as time‑series, each uniquely identified by a metric name and an ordered set of labels. A series is essentially a tuple:

<metric_name>{label_name="label_value", …}

Every data point is a (timestamp, value) pair attached to this tuple. Understanding this model is critical because labels are the only way to add dimensions (e.g., instance, job, region). However, each distinct label combination creates a separate series in the on‑disk database, consuming memory and CPU during scrapes and queries.

Concrete numbers

• A single Prometheus server with default --storage.tsdb.retention.time=15d and a scrape interval of 15 seconds can comfortably store ~1 million series on a 8 CPU, 32 GB RAM node.
• Adding 10 extra labels with high cardinality (e.g., user_id) can push series count beyond 5 million, leading to >30 % increase in memory usage and >2× query latency.

Example: Hive temperature

hive_temperature_celsius{apiary="north", hive_id="42"}

Here hive_temperature_celsius is the metric name, while apiary and hive_id are labels that let us slice data per location. The series count grows linearly with the number of hives—manageable and intentional.

Cross‑link

For a deeper dive into the data model, see prometheus-data-model.

2. Designing Metric Names and Labels: The Art of Clarity

A well‑named metric reads like a sentence: what is measured, how, and in which unit. Prometheus recommends the Metric Naming Conventions, which we’ll translate into concrete rules.

Label hygiene

1. Static vs. dynamic labels

• Static: job, instance, region – rarely change, safe for high cardinality.
• Dynamic: user_id, session_id – can explode series count. Reserve for low‑cardinality contexts (e.g., status="200").

1. Label value normalization

• Use lower‑case alphanumerics and underscores.
• Replace spaces with underscores ("North America" → "north_america").
• Trim leading/trailing whitespace.

1. Avoid duplicate information

• Don’t encode the metric name into a label (metric_name="cpu_usage").

Real‑world illustration

Our AI pollination agents expose a metric for decision latency:

agent_decision_latency_seconds{agent_id="alpha-3", model_version="v2.1", region="southwest"}

• agent_id is static per deployed robot.
• model_version changes only when a new model is rolled out (low cardinality).
• region groups agents geographically.

If we added task_id (unique per mission) as a label, a single agent could generate thousands of series, quickly saturating storage.

3. Managing Label Cardinality: Performance‑First Practices

Cardinality = the number of unique label combinations for a metric. High cardinality is a silent killer of Prometheus performance.

Empirical thresholds

A widely cited rule of thumb is no more than 100 labels per metric and no label value set larger than 10 k distinct entries. Exceeding these numbers often signals a design problem.

Strategies to tame cardinality

1. Prometheus relabeling – Filter or drop high‑cardinality labels at scrape time. Example: drop pod_name for a metric that already includes pod_id.

   relabel_configs:
     - source_labels: [__meta_kubernetes_pod_name]
       target_label: pod_name
       action: replace
     - source_labels: [pod_name]
       regex: .*
       action: labeldrop

1. Aggregating at source – Instead of exposing per‑request metrics, aggregate within the application (e.g., bucketed histograms).

1. Use summary sparingly – Summaries compute quantiles client‑side, generating many series if labeled heavily. Prefer histograms with a fixed bucket set.

1. Leverage instance and job – These built‑in labels already provide host‑level dimensions; avoid duplicating them.

Bee‑centric case study

A hive monitoring system initially exposed a metric hive_sensor_readings_total{hive_id, sensor_type, timestamp} where timestamp was a Unix epoch second. This resulted in ~86 k series per day per hive (one series per second). By removing timestamp from the label set and using a counter that increments per reading, we collapsed the cardinality to a single series per (hive_id, sensor_type), cutting storage by >99 % while preserving the ability to compute rates.

4. Choosing the Right Metric Type: Counters, Gauges, Histograms, Summaries

Prometheus defines four core metric types, each suited to specific patterns.

Concrete example: API latency

# Histogram definition (Go client)
api_request_duration_seconds{method="GET", endpoint="/hives"} // implicit buckets

Prometheus automatically creates:

• api_request_duration_seconds_bucket{le="0.05"}
• api_request_duration_seconds_bucket{le="0.1"}
• …
• api_request_duration_seconds_sum
• api_request_duration_seconds_count

We can query the 95th percentile across all hives:

histogram_quantile(0.95,
  sum by (le) (rate(api_request_duration_seconds_bucket[5m]))
)

Why not use a Summary for this?

If each hive emitted its own summary, the resulting series count would be #hives × #quantiles. With 500 hives and 5 quantiles, that’s 2,500 series, which is manageable but prevents us from aggregating across hives without losing precision. Histograms retain the ability to compute quantiles after aggregation, making them the preferred choice for distributed systems.

AI agent monitoring

Our autonomous pollination drones emit a gauge for battery voltage:

drone_battery_volts{agent_id="beta-7", model_version="v3.0"}

When a drone’s battery falls below 3.5 V, an alert triggers (see Section 6). The gauge’s simplicity ensures low overhead even when thousands of drones report every 10 seconds.

5. Instrumentation Patterns: Language‑Specific Recipes

Below are concise snippets for the three most common languages in the Apiary stack: Go, Python, and Java. Each example follows the naming conventions discussed earlier and demonstrates how to avoid common pitfalls.

Go (using prometheus/client_golang)

var (
    apiRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "api_requests_total",
            Help: "Total number of API requests, labeled by method and status code.",
        },
        []string{"method", "code"},
    )
    apiRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "api_request_duration_seconds",
            Help:    "Latency of API requests.",
            Buckets: prometheus.ExponentialBuckets(0.01, 2, 10), // 10 ms → ~10 s
        },
        []string{"method", "endpoint"},
    )
)

func init() {
    prometheus.MustRegister(apiRequestsTotal, apiRequestDuration)
}

Pitfall avoided: No timestamp or user_id labels; only static dimensions.

Python (using prometheus_client)

from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter(
    "api_requests_total",
    "Total API requests",
    ["method", "code"]
)

LATENCY = Histogram(
    "api_request_duration_seconds",
    "API request latency",
    ["method", "endpoint"],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
)

def handle_request(method, endpoint):
    with LATENCY.labels(method=method, endpoint=endpoint).time():
        # process request …
        REQUESTS.labels(method=method, code="200").inc()

Java (using simpleclient)

static final Counter API_REQUESTS = Counter.build()
    .name("api_requests_total")
    .help("Total API requests")
    .labelNames("method", "code")
    .register();

static final Histogram API_LATENCY = Histogram.build()
    .name("api_request_duration_seconds")
    .help("API request latency")
    .labelNames("method", "endpoint")
    .buckets(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10)
    .register();

Exporter tip

If your service cannot be instrumented directly (e.g., a legacy C++ binary), consider using an exporter such as the node_exporter for system metrics or building a custom textfile collector that reads metrics from a file written by the application. Exporters should also respect label cardinality limits—avoid dumping raw log lines as labels.

6. Crafting Alerting Rules: From Noise to Signal

Prometheus alerts are defined in the Alertmanager ecosystem. A well‑crafted rule contains:

1. A clear expression that captures an abnormal state.
2. A severity label (severity="critical" vs. "warning").
3. A concise, human‑readable description.
4. Optional runbook_url linking to remediation steps.

Example: High error rate

groups:
- name: api-errors
  rules:
  - alert: High5xxErrorRate
    expr: |
      sum by (job, instance) (
        rate(http_requests_total{code=~"5.."}[2m])
      )
      /
      sum by (job, instance) (
        rate(http_requests_total[2m])
      ) > 0.05
    for: 3m
    labels:
      severity: critical
    annotations:
      summary: "5xx error rate > 5% on {{ $labels.instance }}"
      description: |
        The instance {{ $labels.instance }} has returned more than 5 % HTTP 5xx responses
        over the last 2 minutes (current rate: {{ printf \"%.2f\" $value }}).
        Check upstream services and database connectivity.
      runbook_url: https://docs.apiary.org/runbooks/http-5xx

Key points:

• for: 3m prevents flapping by requiring the condition to persist.
• severity: critical enables routing to on‑call pagers.
• runbook_url bridges to a concrete remediation guide (e.g., restart the API pod).

Alert on label dimensions

When a metric has high cardinality, alerting on every label combination can overwhelm the team. Instead, aggregate first, then filter:

# Alert if any hive’s temperature exceeds 35 °C for >5 min
expr: max by (hive_id) (hive_temperature_celsius) > 35
for: 5m

This reduces the number of alerts to one per hive rather than per metric instance.

Silencing and inhibition

Use Alertmanager’s inhibition rules to suppress lower‑severity alerts when a higher‑severity one is firing:

inhibit_rules:
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal: [alertname, instance]

If High5xxErrorRate (critical) fires, a concurrent SlowResponseTime (warning) for the same instance will be silenced, focusing attention on the root cause.

AI agent alert example

- alert: DroneBatteryLow
  expr: drone_battery_volts < 3.5
  for: 2m
  labels:
    severity: warning
    team: robotics
  annotations:
    summary: "Battery voltage low on {{ $labels.agent_id }}"
    description: |
      Drone {{ $labels.agent_id }} (model {{ $labels.model_version }}) has reported a voltage
      of {{ printf \"%.2f\" $value }} V for the past 2 minutes.
      Schedule a landing and battery swap.
    runbook_url: https://docs.apiary.org/runbooks/drone-battery

7. Exporters and Service Discovery: Scaling Observability

A single Prometheus server can scrape hundreds of thousands of targets, but only if it knows where to find them. Two mechanisms dominate:

1. Static configuration

Simple YAML lists of targets. Suitable for small, static environments.

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['10.0.0.1:9100', '10.0.0.2:9100']

2. Dynamic service discovery

Kubernetes, Consul, EC2, and file‑based SD automatically adjust the target list as services scale up or down.

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

Exporter selection guidelines

Tip: When deploying a large number of identical exporters (e.g., one per hive), use relabel_configs to drop the instance label if you already encode the hive ID elsewhere. This prevents duplicate dimensions.

Real‑world scaling story

Apiary’s monitoring cluster grew from 5 to 120 hive gateways in six months. Initially each gateway exposed a per‑sensor metric sensor_reading_total{hive_id, sensor_type, timestamp}. After applying the cardinality reduction from Section 3 (dropping timestamp) and consolidating the exporters via a single hive_exporter per region, the total series count fell from ≈4 M to ≈200 k, allowing the Prometheus server to maintain sub‑second query latency.

8. Alerting on High‑Cardinality Dimensions: Best‑Practice Patterns

When a metric naturally carries many dimensions (e.g., per‑user request latency), you must decide what to alert on without exploding alert volume.

Pattern A: Top‑N aggregation

Identify the worst offenders and alert only on them.

# Alert if any user’s 99th‑percentile latency exceeds 2 s
expr: |
  topk(5, max by (user_id) (
    histogram_quantile(0.99,
      sum by (le, user_id) (rate(http_request_duration_seconds_bucket[5m]))
    )
  )) > 2
for: 5m

This limits alerts to the five slowest users, which is often sufficient for capacity planning.

Pattern B: Rate‑based throttling

Combine a rate function with a label filter to focus on a subset.

# Alert on API error spikes for premium customers only
expr: |
  sum by (customer_tier) (
    rate(http_requests_total{code=~"5..", customer_tier="premium"}[1m])
  ) > 0.2
for: 2m

Pattern C: Use alertmanager grouping

Group alerts by a high‑cardinality label (e.g., hive_id) so they appear as a single notification per hive.

route:
  group_by: ['alertname', 'hive_id']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

9. Scaling Prometheus: Federation, Remote Write, and Thanos

When a single Prometheus instance cannot handle the ingest rate or retention requirements, the ecosystem provides horizontal scaling options.

Federation

• Parent‑child model: Each child scrapes a subset (e.g., a region) and the parent queries the children via /federate.
• Pros: Simple, works with existing PromQL.
• Cons: Limited to down‑sampling; the parent cannot rewrite child rules.

Example: A national Apiary monitoring system has a child Prometheus per state. The central server federates only the hive_temperature_celsius metric, reducing cross‑region traffic.

Remote Write

• Pushes samples to an external storage (e.g., Cortex, Mimir, Thanos).
• Enables global deduplication, long‑term retention, and high availability.

remote_write:
  - url: "https://thanos-receiver.apiary.org/api/v1/receive"
    remote_timeout: 30s
    write_relabel_configs:
      - source_labels: [__name