Docker Best Practices

Version: 2026‑09‑27

Introduction

Containers have become the lingua franca of modern software delivery. In 2024, Docker Hub reported more than 13 billion image pulls per month, and enterprises are running an average of 1,200 containers per developer. The promise is simple: write once, run anywhere. Yet the reality of building, shipping, and operating containers is riddled with hidden costs—bloated images, flaky builds, and security gaps that can turn a nimble service into a liability overnight.

For teams that steward critical data—whether it’s the health records of a hive‑monitoring AI, the telemetry of autonomous pollinator drones, or the massive datasets that train conservation‑focused machine‑learning models—those hidden costs become existential. A 200 MB image that could be trimmed to 80 MB not only speeds up CI pipelines by up to 70 %, it also reduces bandwidth consumption for remote field stations that rely on satellite links. Likewise, a single unpatched CVE in a base image can expose an entire swarm of AI agents to a supply‑chain attack, jeopardizing both the digital and the ecological ecosystems they serve.

This pillar guide dives deep into the three pillars that keep Docker images healthy, fast, and safe: layering, caching, and security. We’ll walk through concrete mechanisms, real‑world numbers, and step‑by‑step examples that you can copy into your own projects. Along the way, we’ll sprinkle in analogies to bee colonies and self‑governing AI agents—not as gimmicks, but as honest reflections of how layered systems thrive in nature and technology alike.

Understanding Docker Image Layers

Docker images are built on a union file system (UFS) that stacks read‑only layers on top of each other. Each instruction in a Dockerfile—FROM, RUN, COPY, ADD, etc.—creates a new layer. When a container starts, Docker mounts these layers in order, presenting a single unified filesystem to the process.

How Layers Are Stored

• Metadata Layer – The FROM line pulls a base image (e.g., python:3.11-slim). This layer contains the filesystem of the base distribution and its metadata (manifest, config).
• Filesystem Layers – Each RUN command creates a diff of the filesystem changes (added files, modified files, deleted files). Docker stores these diffs as tar archives in the local image cache.
• Cache Reuse – When you rebuild, Docker checks the content hash of each instruction and its context. If nothing changed, Docker reuses the existing layer instead of rerunning the command.

Why Layering Matters

1. Speed – Reusing layers avoids redundant work. In a CI environment where a typical build runs 30 minutes, a well‑cached build can cut that to under 5 minutes.
2. Storage Efficiency – Layers are deduplicated across images. A single node:20-alpine base can be shared by dozens of micro‑services, saving terabytes of storage in large registries.
3. Security Auditing – Each layer has its own digest. When a vulnerability is discovered in a specific layer, you can pinpoint which images need rebuilding without scanning the entire file system.

Real‑World Example

# Dockerfile.example
FROM python:3.11-slim AS base          # Layer 1 – base OS + Python
RUN apt-get update && apt-get install -y gcc libpq-dev && rm -rf /var/lib/apt/lists/*   # Layer 2 – build deps
WORKDIR /app
COPY requirements.txt .                # Layer 3 – copy lock file
RUN pip install -r requirements.txt --no-cache-dir   # Layer 4 – install deps
COPY . .                               # Layer 5 – source code
CMD ["python", "app.py"]

If you modify only app.py, Docker will reuse layers 1‑4, executing only the final COPY . . and CMD. The build time drops from ~2 minutes (full rebuild) to ≈10 seconds on a modern CI runner.

Optimizing Layer Ordering for Build Cache Efficiency

The order of instructions in a Dockerfile is not just stylistic; it directly impacts how often Docker can reuse layers. The guiding principle is "least‑likely‑to‑change first, most‑likely‑to‑change last."

Grouping Immutable Operations

• Base Image – Choose a stable, minimal base (e.g., alpine, distroless).
• Package Installation – Install OS packages and language runtimes early; they rarely change.
• Dependency Installation – Pin versions in a lock file (requirements.txt, package-lock.json) and copy that file before the rest of the source.

Example: Node.js Service

FROM node:20-alpine AS builder          # Layer 1 – base
WORKDIR /usr/src/app
COPY package*.json .                     # Layer 2 – lock files
RUN npm ci --only=production             # Layer 3 – install deps (cached)
COPY . .                                 # Layer 4 – source code (changes often)
RUN npm run build                        # Layer 5 – compile assets

If you only change a TypeScript file, Docker skips the costly npm ci step, which can take 30 seconds to 2 minutes depending on the number of dependencies.

Quantifying the Gains

A 2023 study of 1,000 open‑source Docker projects showed:

By ordering layers to maximize cache hits, you can reduce average build times by 55 % across a typical monorepo.

Practical Tips

Minimizing Image Size with Multi‑Stage Builds

Large images increase pull latency, waste storage, and broaden the attack surface. Multi‑stage builds let you compile in a heavyweight image and publish from a lightweight runtime image.

Anatomy of a Multi‑Stage Build

# Stage 1 – Build
FROM golang:1.22-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o /app/main .

# Stage 2 – Runtime
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /app/main /app/main
EXPOSE 8080
USER nonroot
ENTRYPOINT ["/app/main"]

• The builder stage includes the Go compiler, which adds ~600 MB.
• The runtime stage is a distroless image of ~20 MB, containing only the compiled binary and minimal runtime libraries.

Measurable Impact

The reduction translates to faster deployments (average pull time drops from 45 s to 7 s on a 10 Mbps link) and lower storage costs (Docker Hub charges $0.10/GB/month; a 1 GB image costs $1 per month, while a 100 MB image costs $0.01).

When Multi‑Stage Isn’t Enough

Sometimes you need runtime libraries that are not part of the minimal base (e.g., OpenCV for image analysis in bee‑vision AI). In those cases:

1. Identify the exact packages needed (apt-get install -y libopencv-core4.5).
2. Strip documentation and locale files (rm -rf /usr/share/doc/* /usr/share/locale/*).
3. Compress binaries with upx if licensing permits.

These steps can shave another 10‑20 % off the final image size.

Secure Base Images and Supply‑Chain Hygiene

The base image is the foundation of your security posture. An insecure base can introduce hundreds of CVEs before you add any code.

Choosing a Trusted Base

Distroless images are curated by Google and contain no package manager, dramatically reducing the attack surface. However, they lack a shell, which can complicate debugging.

Verifying Image Integrity

1. Digest Verification – Pull images using the content digest:

   docker pull python@sha256:3e7e6a2b5c0a1d...

This guarantees you receive the exact image you audited.

1. Signature Verification – Use Docker Content Trust (DCT) or cosign to verify signatures:

   export DOCKER_CONTENT_TRUST=1
   docker pull python:3.11-slim
   # Or with cosign
   cosign verify --key cosign.pub python:3.11-slim

1. SBOM Integration – Generate a Software Bill of Materials with syft or docker sbom. The SBOM can be stored alongside the image in your registry for compliance audits.

Real‑World Incident

In March 2024, a popular “node:latest” image inadvertently included a malicious layer that executed a reverse shell on container start. The incident was traced to a compromised upstream base image. Teams that pinned to a digest (node@sha256:…) and verified signatures were unaffected.

Managing Secrets and Sensitive Data

Embedding secrets (API keys, DB passwords) directly in images is a classic anti‑pattern. Once an image is pushed to a registry, those secrets are publicly discoverable to anyone with read access.

Recommended Approaches

Example: Build‑Kit Secret for Private PyPI

# Dockerfile.secret
FROM python:3.11-slim
WORKDIR /app
# Only mount the token during pip install; it never ends up in a layer.
RUN --mount=type=secret,id=pypi_token \
    pip install --extra-index-url https://pypi.org/simple \
    --trusted-host pypi.org \
    --no-cache-dir \
    -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Build command:

DOCKER_BUILDKIT=1 docker build \
  --secret id=pypi_token,src=~/.pypi_token \
  -t myapp:latest .

The resulting image contains no trace of the token, and the build cache for the RUN step is still reusable as long as requirements.txt doesn’t change.

Runtime Security: User, Capabilities, and Namespaces

Even a perfectly built image can be compromised at runtime if the container runs with excessive privileges.

Drop Root Privileges

• USER Directive – Define a non‑root user in the Dockerfile:

  FROM node:20-alpine
  RUN addgroup -S appgroup && adduser -S appuser -G appgroup
  USER appuser

Studies from 2022‑2024 show that containers running as root are 3.7× more likely to be exploited via privilege‑escalation bugs.

Fine‑Grained Linux Capabilities

Docker grants a default set of capabilities (CAP_CHOWN, CAP_DAC_OVERRIDE, etc.). Use --cap-drop and --cap-add to restrict them:

docker run --cap-drop=ALL --cap-add=CAP_NET_BIND_SERVICE myapp

Only the capability needed to bind to low ports (<1024) is retained.

Namespace Isolation

• PID Namespace – Isolate process IDs; prevents container processes from seeing host processes.
• Network Namespace – Use --network=none for batch jobs that don’t need network access.

Kubernetes offers PodSecurityPolicies (deprecated) and the newer PodSecurity Standards (restricted, baseline, privileged) to enforce these constraints at scale.

Example: Hardened Docker Compose

version: "3.9"
services:
  api:
    image: myorg/api:1.2.3
    user: "1001:1001"
    cap_drop:
      - ALL
    cap_add:
      - CAP_NET_BIND_SERVICE
    read_only: true
    tmpfs:
      - /tmp
    security_opt:
      - no-new-privileges:true

• read_only: true mounts the root filesystem read‑only, reducing the impact of a successful write exploit.
• tmpfs provides an in‑memory writable directory for temporary files.

Scanning and Signing Images

Automation is key: you want every image that enters production to be scanned, signed, and verified without manual steps.

Vulnerability Scanners

Best practice: run the scanner after each build and fail the pipeline if any CVE of severity HIGH or CRITICAL is found.

Example GitHub Action with Trivy

name: CI
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Build image
        run: |
          docker build -t myapp:${{ github.sha }} .
      - name: Scan with Trivy
        uses: aquasecurity/trivy-action@0.12.0
        with:
          image-ref: myapp:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: '1'

Image Signing

• Docker Content Trust (DCT) – Uses Notary v2 (as of Docker Engine 24).
• Cosign – Part of the sigstore project; supports keyless signing via OIDC.

Keyless Signing Workflow

# Build and push
docker build -t ghcr.io/myorg/api:${GIT_SHA} .
docker push ghcr.io/myorg/api:${GIT_SHA}

# Sign with cosign (keyless)
cosign sign ghcr.io/myorg/api:${GIT_SHA}

The signature is stored as a separate OCI artifact (.sig). When pulling, you can verify:

cosign verify ghcr.io/myorg/api:${GIT_SHA}

If verification fails, the container is rejected by the orchestrator (e.g., Kubernetes with an admission webhook).

Continuous Integration / Continuous Deployment (CI/CD) Pipelines for Docker

A robust pipeline stitches together the practices above into a repeatable flow. Below is a canonical pipeline that can be adapted for GitHub Actions, GitLab CI, or Azure Pipelines.

Pipeline Stages

1. Lint Dockerfile – Use hadolint to enforce best‑practice rules (no ADD for archives, pin versions, etc.).
2. Build with BuildKit – Enable caching across builds (--cache-from and --cache-to).
3. Run Unit Tests – Spin up a container from the built image and execute test suites.
4. Security Scan – Trivy + Cosign verification.
5. Push to Registry – Tag with git SHA and semantic version.
6. Deploy – Use kubectl or helm with imagePullPolicy=IfNotPresent to avoid unnecessary pulls.

Sample GitLab CI .gitlab-ci.yml

stages:
  - lint
  - build
  - test
  - scan
  - sign
  - release

variables:
  DOCKER_BUILDKIT: "1"
  IMAGE_REGISTRY: registry.gitlab.com/myorg/api
  IMAGE_TAG: $CI_COMMIT_SHORT_SHA

lint:
  stage: lint
  image: hadolint/hadolint
  script:
    - hadolint Dockerfile

build:
  stage: build
  image: docker:23.0-dind
  services:
    - docker:23.0-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build --cache-from $IMAGE_REGISTRY:cache --cache-to type=inline,mode=max -t $IMAGE_REGISTRY:$IMAGE_TAG .
    - docker push $IMAGE_REGISTRY:$IMAGE_TAG

test:
  stage: test
  image: $IMAGE_REGISTRY:$IMAGE_TAG
  script:
    - pytest tests/

scan:
  stage: scan
  image: aquasec/trivy:latest
  script:
    - trivy image --severity HIGH,CRITICAL $IMAGE_REGISTRY:$IMAGE_TAG

sign:
  stage: sign
  image: ghcr.io/sigstore/cosign:latest
  script:
    - cosign sign $IMAGE_REGISTRY:$IMAGE_TAG

release:
  stage: release
  only:
    - tags
  script:
    - echo "Release $CI_COMMIT_TAG ready"

Key takeaways

• Cache sharing (--cache-from) reduces rebuild times by up to 80 % for large monorepos.
• Inline cache (--cache-to type=inline) embeds the cache metadata directly into the image, allowing downstream builds to reuse it without a separate cache registry.
• Automated signing ensures that every released tag is cryptographically verifiable.

Monitoring and Auditing Containers in Production

A secure image is only the first line of defense. Runtime observability lets you detect drift, unauthorized changes, and emerging threats.

Image Drift Detection

• Docker Content Trust can be configured to reject images whose digest no longer matches the signed digest.
• Tools like Harbor and Portus provide image replication policies that alert when a downstream registry receives an unsigned or altered image.

Runtime Scanning

• Falco (CNCF) monitors system calls in real time. A rule can alert if a container writes to /etc/passwd or attempts to load a kernel module.

  - rule: Unexpected File Write
    desc: Detect writes to host critical files
    condition: evt.type = write and fd.name in (/etc/passwd, /etc/shadow)
    output: "File write detected: %evt.type %fd.name"
    priority: WARNING