Deploying Applications On Kubernetes

Deploying Applications On Kubernetes is the cornerstone of modern cloud‑native engineering. Whether you’re running a tiny hobby project that tracks hive temperature or a multi‑regional data platform that powers AI agents for bee‑conservation analytics, Kubernetes (often abbreviated K8s) gives you the tools to automate deployment, scale with precision, and keep your services healthy with minimal manual effort. In the past five years, the Cloud Native Computing Foundation (CNCF) reported that over 78 % of enterprises now run at least one production workload on Kubernetes, and the number of active clusters grew from 1.2 million in 2020 to 3.4 million in 2024. Those numbers aren’t just statistics—they reflect a shift toward a declarative, infrastructure‑agnostic mindset that lets engineers focus on what to run rather than how to run it.

For a platform like Apiary, which blends bee‑conservation data with self‑governing AI agents, the stakes are especially high. The data pipelines that ingest hive sensor streams, the model‑training jobs that predict colony health, and the public APIs that surface insights must be reliable, elastic, and secure. Kubernetes provides a single, consistent control plane that can orchestrate containers across on‑premise edge nodes (e.g., beehive‑mounted gateways) and public clouds alike, while exposing the same API to developers and to AI agents that might request resources autonomously. In this guide we’ll walk through every essential step—from containerizing your code to operating a production‑grade cluster—so you can launch, scale, and evolve applications with confidence.

1. Understanding the Kubernetes Architecture

Before you start writing manifests, it’s crucial to grasp the control plane vs. data plane separation that underpins Kubernetes. The control plane consists of the API server, scheduler, controller manager, and etcd—collectively responsible for maintaining the desired state of the cluster. The data plane is made up of worker nodes that run the kubelet, kube-proxy, and the container runtime (Docker, containerd, or CRI‑O).

In practice, a managed Kubernetes service (EKS, GKE, AKS, or the open‑source kubernetes-architecture) supplies a highly‑available control plane, letting you focus on the worker nodes and the workloads you run. For Apiary, you might run a hybrid cluster: edge nodes attached to beehives that need only a few CPU cores, plus a cloud‑based pool for heavy model training. Kubernetes’s taints and tolerations let you keep those two worlds separate while still using a single API.

The Declarative Model

Kubernetes works on a desired‑state model: you submit a manifest that declares what you want (e.g., “run three replicas of my API server”), and the control plane continually reconciles the actual state to match. This eliminates the “drift” problem common in imperative scripts, where a manual change can silently break a system. The reconciliation loop runs every 5 seconds by default, making the system responsive to failures: if a pod crashes, the controller spawns a replacement automatically.

Namespaces, Labels, and Selectors

Namespaces provide logical isolation (e.g., dev, staging, prod). Labels are key‑value pairs attached to objects; selectors let controllers target subsets of resources. For a bee‑monitoring system you could label all edge‑gateway pods with environment=field and role=gateway, then create a NetworkPolicy that only allows traffic from role=processor pods in the prod namespace. This fine‑grained taxonomy is the backbone of both security and observability.

2. Preparing Your Application for Containerization

Kubernetes does not magically make any binary container‑ready; you must first containerize the application. The most common format is the Docker image, but any OCI‑compatible image works. Below are the steps you should follow, illustrated with a Python microservice that ingests hive sensor data.

2.1 Choose a Base Image

Start with a minimal, security‑hardened base. Alpine Linux is popular (python:3.11-alpine) but can cause glibc compatibility issues; Distroless images (gcr.io/distroless/python3) remove shells entirely, reducing attack surface. For a production‑grade API, the Distroless variant is recommended.

# Dockerfile
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt

FROM gcr.io/distroless/python3
COPY --from=builder /root/.local /usr/local
COPY . /app
WORKDIR /app
CMD ["main.py"]

The multi‑stage build keeps the final image under 80 MiB, a stark contrast to the 300 MiB typical of naïve builds.

2.2 Externalize Configuration

Hard‑coding values (e.g., DB credentials) defeats the purpose of declarative deployment. Use environment variables, ConfigMaps, and Secrets. For example, the sensor ingestion service reads SENSOR_API_URL from the environment, while the DB password lives in a Secret.

apiVersion: v1
kind: ConfigMap
metadata:
  name: sensor-config
data:
  SENSOR_API_URL: "https://api.bee-data.org/v1/sensors"
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YXBpX3VzZXI=   # base64('api_user')
  password: c2VjcmV0cGFzcw==   # base64('secretpass')

2.3 Health Checks

Kubernetes uses liveness and readiness probes to know when a container is healthy or ready to receive traffic. Add a simple HTTP endpoint (/healthz) that returns 200 OK when the service can connect to its downstream dependencies.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 15

If the liveness probe fails three consecutive times, the kubelet restarts the container; readiness failures simply remove the pod from Service load‑balancing.

2.4 Build and Push

Push the image to a registry with proper access controls. In a production environment, use a private registry (e.g., Amazon ECR, Google Artifact Registry) and enable image scanning. As of 2024, ECR’s built‑in scanner flags ≈ 2 % of images for known CVEs, providing an early warning before the image ever lands in a cluster.

docker build -t 123456789012.dkr.ecr.us-east-1.amazonaws.com/apiary-sensor:1.3.0 .
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/apiary-sensor:1.3.0

3. Writing Kubernetes Manifests: Pods, Deployments, Services

Now that the image exists, you can describe the desired state in YAML. The three most common objects are Pod, Deployment, and Service. Below we walk through each, adding concrete fields that matter in production.

3.1 Pods: The Atomic Unit

A Pod is a single or multiple containers that share a network namespace. Directly creating Pods is rarely recommended because they lack self‑healing. However, understanding the Pod spec is essential for troubleshooting.

apiVersion: v1
kind: Pod
metadata:
  name: sensor-ingest-pod
  labels:
    app: sensor-ingest
spec:
  containers:
  - name: sensor-ingest
    image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/apiary-sensor:1.3.0
    resources:
      requests:
        cpu: "250m"
        memory: "128Mi"
      limits:
        cpu: "500m"
        memory: "256Mi"
    envFrom:
    - configMapRef:
        name: sensor-config
    - secretRef:
        name: db-credentials
    ports:
    - containerPort: 8080

Notice the resource requests and limits. Requests are used by the scheduler to place the pod; limits enforce a hard cap via cgroups. In a cluster with 100 nodes, a typical policy is to keep CPU request utilization under 70 % to avoid overcommit.

3.2 Deployments: Desired‑State Controllers

A Deployment abstracts Pods and provides rolling updates, rollbacks, and replica management. Here’s a production‑grade Deployment for the sensor service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sensor-ingest
  labels:
    app: sensor-ingest
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # at most 1 extra pod during update
      maxUnavailable: 0 # keep all existing pods available
  selector:
    matchLabels:
      app: sensor-ingest
  template:
    metadata:
      labels:
        app: sensor-ingest
    spec:
      containers:
      - name: sensor-ingest
        image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/apiary-sensor:1.3.0
        resources:
          requests:
            cpu: "250m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
        envFrom:
        - configMapRef:
            name: sensor-config
        - secretRef:
            name: db-credentials
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 15

A few key points:

• Replica count (replicas: 3) ensures high availability; if one node fails, two pods remain.
• The RollingUpdate strategy with maxSurge: 1 and maxUnavailable: 0 guarantees zero downtime during upgrades.
• Pod anti‑affinity can be added to spread replicas across nodes, reducing the risk of a single point of failure.

3.3 Services: Stable Networking

Kubernetes Services give pods a stable DNS name and load‑balance traffic. For the sensor API we expose a ClusterIP Service internally and an Ingress for external traffic.

apiVersion: v1
kind: Service
metadata:
  name: sensor-ingest-svc
  labels:
    app: sensor-ingest
spec:
  selector:
    app: sensor-ingest
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP

And an Ingress (using the NGINX Ingress Controller):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sensor-ingest-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - apiary.example.com
    secretName: apiary-tls
  rules:
  - host: apiary.example.com
    http:
      paths:
      - path: /sensors
        pathType: Prefix
        backend:
          service:
            name: sensor-ingest-svc
            port:
              number: 80

Notice the TLS secret (apiary-tls). For compliance with GDPR and bee‑conservation data policies, all external traffic must be encrypted; Kubernetes makes it trivial to rotate certificates via cert‑manager.

4. Managing Secrets and ConfigMaps

Sensitive data should never be baked into container images or plain‑text manifests. Kubernetes offers two built‑in objects: Secrets (base64‑encoded, optionally encrypted at rest) and ConfigMaps (non‑sensitive configuration). Let’s explore best practices.

4.1 Encryption at Rest

By default, etcd stores Secrets in plain base64, which is trivially reversible. Enable EncryptionConfiguration on the API server:

apiVersion: apiserver.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64‑encoded‑32‑byte‑key>
      - identity: {}

When using a managed service, the provider usually enables this automatically; for self‑hosted clusters, you must add the flag --encryption-provider-config=/etc/kubernetes/encryption.yaml.

4.2 Dynamic Secrets with External Vaults

For rotating credentials (e.g., AWS IAM keys for AI agents that fetch remote datasets), integrate with a secret store like HashiCorp Vault or AWS Secrets Manager. The Secrets Store CSI driver mounts secrets as volumes, keeping them out of etcd entirely.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-credentials
spec:
  provider: vault
  parameters:
    roleName: "k8s-db-reader"
    vaultAddress: "https://vault.example.com"
    secretPath: "kv/data/db"
    objects: |
      - objectName: "username"
        secretKey: "username"
      - objectName: "password"
        secretKey: "password"

The pod then consumes the secret as a file, avoiding environment variable leakage.

4.3 ConfigMaps for Feature Flags

Feature flags let you toggle behavior without redeploying. Store them in a ConfigMap and have the application watch for changes using the Kubernetes API or a sidecar like kube‑watcher. For example, a flag ENABLE_HONEYCOMB_ANALYTICS could be toggled during a trial period, and the change propagates instantly to all pods.

apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-flags
data:
  ENABLE_HONEYCOMB_ANALYTICS: "false"

When you need to flip the flag, a simple kubectl apply -f updates the ConfigMap, and the running pods pick up the new value within seconds.

5. Scaling and Autoscaling

One of Kubernetes’ core promises is elastic scaling—the ability to adjust resources automatically based on demand. Three mechanisms cover most scenarios.

5.1 Horizontal Pod Autoscaler (HPA)

The HPA adjusts the replica count of a Deployment based on observed metrics (CPU, memory, custom metrics). As of Kubernetes 1.27, the HPA supports multiple metrics and external metrics (e.g., Kafka lag).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sensor-ingest-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sensor-ingest
  minReplicas: 3
  maxReplicas: 12
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Pods
    pods:
      metric:
        name: request_rate
      target:
        type: AverageValue
        averageValue: "1000"

During a bloom season, sensor traffic can surge to 2 k requests per second per pod; the HPA will spin up additional replicas to keep CPU utilization under 60 %, preserving latency under 150 ms.

5.2 Vertical Pod Autoscaler (VPA)

VPA adjusts resource requests/limits for individual pods. It’s useful for batch jobs like AI model training where the workload is CPU‑intensive but not easily expressed as a simple metric.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: model-trainer-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       model-trainer
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: trainer
      minAllowed:
        cpu: 500m
        memory: 1Gi
      maxAllowed:
        cpu: 4
        memory: 8Gi

The VPA observes the pod’s actual consumption and recommends (or applies) larger requests, reducing OOMKill incidents—an issue that historically affected ≈ 12 % of AI workloads on Kubernetes.

5.3 Cluster Autoscaler

When the HPA or VPA demands more pods than the current node pool can host, the Cluster Autoscaler adds new nodes (or removes idle nodes). In a cloud environment, it integrates with the provider’s API to provision EC2 Spot Instances or Preemptible VMs, cutting compute costs by up to 70 %.

Key parameters:

• --scale-down-utilization-threshold=0.5 – nodes are considered for removal when utilization drops below 50 %.
• --expander=least-waste – prefers nodes that leave the least unused capacity, ideal for heterogeneous workloads (e.g., mixing edge‑gateway nodes with GPU nodes for model training).

6. Rolling Updates and Rollbacks

Downtime is unacceptable for a public API that feeds AI agents monitoring bee colonies. Kubernetes’ native rolling update mechanism, combined with readiness probes, provides zero‑downtime deployments.

6.1 The Rollout Process

When you apply a new Deployment manifest (kubectl apply -f deployment.yaml), the controller creates a ReplicaSet for the new version while retaining the old one. It then gradually scales up the new ReplicaSet while scaling down the old, respecting the maxSurge and maxUnavailable fields. If any pod fails its readiness probe, the rollout pauses, preventing traffic from being sent to a broken pod.

You can monitor rollout status:

kubectl rollout status deployment/sensor-ingest

6.2 Manual Pausing and Resuming

Sometimes you need to intervene—for example, a new model version introduces an unexpected latency spike. You can pause the rollout:

kubectl rollout pause deployment/sensor-ingest

After fixing the issue, resume:

kubectl rollout resume deployment/sensor-ingest

6.3 Rollback

If the new version is catastrophically broken, Kubernetes lets you revert:

kubectl rollout undo deployment/sensor-ingest

The system automatically rolls back to the previous ReplicaSet, preserving the old pods for the duration of the rollback. All of this happens without needing an external CI/CD tool, though integrating with one (see ci-cd-pipelines) adds audit trails and automated testing.

7. Monitoring, Logging, and Observability

A production cluster is only as good as the insight you have into its behavior. Three pillars—metrics, logs, and traces—form a comprehensive observability stack.

7.1 Metrics: Prometheus & Grafana

Prometheus scrapes metrics from the Kubernetes API (/metrics endpoint) and from instrumented applications. A typical scrape config for the sensor service:

scrape_configs:
- job_name: 'sensor-ingest'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    action: keep
    regex: sensor-ingest

Prometheus stores time‑series data locally for 15 days by default. Grafana dashboards built on top of this data can display:

• CPU & memory utilization per pod (useful for HPA tuning)
• Request latency percentiles (p95, p99)
• Error rate (HTTP 5xx)

A real‑world metric: during a recent hive‑migration, the sensor service sustained 5,400 requests per minute with a p99 latency of 112 ms, comfortably under the 250 ms SLA.

7.2 Logs: Loki + Fluent Bit

Logging containers directly to stdout/stderr allows the Kubernetes logging driver to capture them. Deploy Fluent Bit as a DaemonSet to ship logs to Grafana Loki, a cost‑effective log aggregation system.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush 5
        Daemon Off
        Log_Level info
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
    [OUTPUT]
        Name  loki
        Match *
        Host  loki.logging.svc.cluster.local
        Port  3100

Because Loki indexes only labels, not full text, storage costs stay low—≈ 0.10 USD/GB in 2024.

7.3 Traces: OpenTelemetry & Jaeger

Distributed tracing helps identify bottlenecks across microservices. Instrument the Python service with OpenTelemetry SDK, exporting spans to a Jaeger backend.

from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
jaeger_exporter = JaegerExporter(
    agent_host_name='jaeger-agent',
    agent_port=6831,
)
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(jaeger_exporter)
)

FastAPIInstrumentor().instrument_app(app)

When a request traverses the sensor‑ingest → data‑processor → analytics chain, you can see the exact time spent in each service, helping you keep end‑to‑end latency under the 200 ms threshold critical for real‑time AI agents.

8. Security Best Practices

Running workloads that handle ecological data and AI model artifacts demands a solid security posture. Below are concrete steps you can implement today.

8.1 Role‑Based Access Control (RBAC)

Kubernetes uses RBAC to limit what users and service accounts can do. Create a service account for the sensor ingestion Deployment and bind it to a least‑privilege role.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sensor-ingest-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sensor-ingest-role
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sensor-ingest-binding
subjects:
- kind: ServiceAccount
  name: sensor-ingest-sa
roleRef:
  kind: Role
  name: sensor-ingest-role
  apiGroup: rbac.authorization.k8s.io

Now the pod can only read its own ConfigMaps and patch its own Deployment, preventing privilege escalation.

8.2 Network Policies

Kubernetes NetworkPolicies enforce zero‑trust networking. A simple policy that only allows inbound traffic from the ingress-nginx namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: sensor-ingest-policy
spec:
  podSelector:
    matchLabels:
      app: sensor-ingest
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 80
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/8
    ports:
    - protocol: TCP
      port: 5432   # DB

This isolates the service from other workloads, a crucial measure when you have AI agents that may request privileged operations.

8.3 Image Vulnerability Scanning

Enable image scanning in your container registry (ECR, GCR, or Harbor). Automate a nightly scan and block images with Critical CVEs (CVSS ≥ 9.0). In 2023, the average time to remediate a Critical vulnerability in public images was 12 days; with automated scanning, you can bring that down to under 24 hours.

8.4 Runtime Security

Tools like Falco or Kube‑Armor watch for anomalous system calls. For example, Falco can alert if a container attempts to open a raw socket, which is disallowed for most web services. Configure alerts to feed into your incident‑response system.

rules:
  - rule: Unexpected Network Socket
    desc: Detect containers trying to create raw sockets.
    condition: evt.type = socket and evt.rawargs contains "SOCK_RAW"
    output: "Container %container.id %container.name attempting raw socket"
    priority: WARNING

9. Deploying with CI/CD Pipelines

Manual kubectl apply works for experiments, but production demands repeatable, audited pipelines. A typical GitOps workflow uses Argo CD or Flux to sync manifests from a Git repository to the cluster.

9.1 GitOps Overview

1. Commit: Developer pushes code and a new image tag to a repo.
2. CI: A CI system (GitHub Actions, GitLab CI, Jenkins) builds the Docker image, runs unit tests, scans for vulnerabilities, and pushes the image to the registry.
3. Manifest Update: CI updates the deployment.yaml with the new image tag (e.g., sensor-ingest:1.4.0).
4. GitOps Operator: Argo CD detects the change, validates the manifest, and applies it to the cluster.
5. Verification: Automated smoke tests run against the new version; if they pass, the rollout proceeds.

A minimal GitHub Actions workflow for the sensor service:

name: CI/CD Pipeline

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2
    - name: Login to ECR
      uses: aws-actions/amazon-ecr-login@v1
    - name: Build & Push
      run: |
        IMAGE_TAG=${{ github.sha }}
        docker build -t ${{ env.ECR_REPO }}:$IMAGE_TAG .
        docker push ${{ env.ECR_REPO }}:$IMAGE_TAG
    - name: Update Manifest
      run: |
        sed -i "s|image: .*$|image: ${{ env.ECR_REPO }}:${{ github.sha }}|" k8s/deployment.yaml
        git config user.name "github-actions"
        git config user.email "actions@github.com"
        git commit -am "ci: bump image to ${{ github.sha }}"
        git push

9.2 Canary Deployments

For high‑impact services (e.g., the public API that AI agents query), you might want a canary rollout: route a small percentage of traffic to the new version before full promotion. Argo CD supports weight‑based canary using Istio or Linkerd.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: sensor-ingest-vs
spec:
  hosts:
  - apiary.example.com
  http:
  - route:
    - destination:
        host: sensor-ingest-svc
        subset: v1
      weight: 90
    - destination:
        host: sensor-ingest-svc
        subset: v2
      weight: 10

When the canary passes health checks, you increase the weight to 100 % automatically via a Progressive Delivery operator.

10. Real‑World Case Study: Bee Data Platform on Kubernetes

To illustrate how all these pieces fit together, let’s walk through a production deployment that powers Apiary’s Bee Data Platform. The platform ingests ≈ 1.2 M sensor readings per hour from thousands of hives, runs daily AI model training to predict colony health, and serves a public GraphQL API used by conservation researchers and autonomous AI agents.

10.1 Architecture Overview

+----------------+      +-------------------+      +-------------------+
| Edge Gateways  | ---> | Ingress (NGINX)   | ---> | Sensor Ingest     |
| (Raspberry Pi) |      | (TLS termination) |      | Deployment (3)   |
+----------------+      +-------------------+      +-------------------+
                                                   |
                                                   v
                                          +-------------------+
                                          | Kafka Cluster     |
                                          +-------------------+
                                                   |
                                                   v
                                          +-------------------+
                                          | Spark / Flink     |
                                          | (Batch & Stream) |
                                          +-------------------+
                                                   |
                                                   v
                                          +-------------------+
                                          | Model Trainer (VPA)|
                                          +-------------------+
                                                   |
                                                   v
                                          +-------------------+
                                          | Postgres + Redis  |
                                          +-------------------+
                                                   |
                                                   v
                                          +-------------------+
                                          | GraphQL API (HPA) |
                                          +-------------------+

Key points:

• Edge Gateways run a lightweight K3s cluster (single node) that forwards sensor data via a Secure MQTT broker to the central cluster.
• The Ingress terminates TLS using cert‑manager‑managed certificates.
• Kafka provides durable buffering; the sensor‑ingest service reads from Kafka, decodes payloads, and writes to Postgres.
• Horizontal Pod Autoscaler scales the GraphQL API to handle spikes during research conferences (up to 200 RPS).
• Vertical Pod Autoscaler automatically adjusts the trainer’s CPU request from 2 vCPU to 8 vCPU as model size grows.
• Cluster Autoscaler adds GPU nodes when the trainer exceeds CPU capacity, using Spot Instances to keep cost under $0.30 per GPU‑hour.

10.2 Performance Numbers

The biggest win came from autoscaling: during a sudden bloom, request volume jumped 4×; the HPA automatically added two extra API pods, keeping latency under the 150 ms SLA.

10.3 Lessons Learned

1. Start Small, Scale Fast – Deploy a single‑replica test environment, then progressively enable HPA/VPA.
2. Separate Concerns – Keep edge‑gateway workloads on a dedicated K3s cluster; this isolates flaky hardware from the core.
3. Automate Secrets – Using the Secrets Store CSI driver eliminated the need for manual secret rotation.
4. Observability Pays Off – Grafana alerts caught a memory leak before it caused an OOMKill, saving a day’s worth of data loss.
5. Security is Not Optional – NetworkPolicies prevented a compromised AI agent from reaching the database, averting a potential data breach.

Why It Matters

Deploying applications on Kubernetes is not just a technical exercise; it’s a catalyst for resilience, scalability, and stewardship. For Apiary, a platform that intertwines bee conservation data with self‑governing AI agents, Kubernetes empowers you to:

• Scale responsibly – Autoscaling lets you meet spikes in data collection without over‑provisioning, reducing carbon footprint and operational cost.
• Protect ecosystems – Fine‑grained RBAC and NetworkPolicies keep sensitive ecological data safe from accidental leaks or malicious actors.
• Enable autonomous agents – AI agents can request resources through the Kubernetes API (e.g., spin up a training job) while the platform enforces policy, creating a trustworthy feedback loop for conservation decisions.
• Accelerate innovation – Rolling updates and GitOps pipelines mean new analytics or sensor‑firmware features reach the field in minutes, not weeks.

In short, Kubernetes gives you the foundation to turn data into action—whether that action is a new predictive model that warns beekeepers of disease, a dashboard that visualizes hive health for policymakers, or an autonomous agent that reallocates resources to protect vulnerable colonies. By mastering the concepts in this guide, you’re not just deploying containers; you’re building a living, adaptable system that can help preserve the planet’s most vital pollinators.