An agent pipeline has several things people casually call “the cache.” They have different correctness rules.
| Layer | Stores | Safe reuse condition |
|---|---|---|
| Raw artifact store | Original files, responses, tool output | Content hash and access policy match |
| Derived-artifact cache | ASTs, OCR, embeddings, summaries | Input hash plus producer version match |
| Retrieval cache | Ranked chunks or query results | Corpus revision, query, filters, and ranker match |
| Model-prefix cache | Repeated instructions and tool schemas | Provider's exact-prefix rules match |
| Decision cache | Final recommendation or action | Policy, evidence, environment, and approval state all match |
The closer a cached value is to a consequential decision, the harder it is to reuse safely. Cache evidence aggressively. Cache authorization and safety decisions conservatively.
Keys are dependency manifests
A timestamp alone is not a sufficient version. A useful derived-result key names every dependency that can change the result:
tenant
task type
input content hashes
repository or corpus revision
parser/model identifier
prompt and tool-schema version
policy/rubric version
output-schema versionHash the canonical form of that tuple. Keep the readable fields as metadata for debugging.
This design turns invalidation into dependency checking. A new policy version does not require guessing which decisions are stale; it creates a different key. Time-to-live remains useful for storage control and external data with known freshness windows, but TTL is not proof that a value is correct.
OpenAI's prompt cache is a narrower mechanism. It reuses matching prompt prefixes to reduce latency and cost. Static instructions, tools, and schemas belong at the front; request-specific content belongs later. A prompt-cache hit says nothing about whether an application-level decision is still valid.
Preserve provenance
Every cached derived artifact should answer:
- What exact inputs produced it?
- Which code, model, prompt, and schema versions ran?
- Which tenant owns it?
- When was it produced and when does it expire?
- Which source evidence supports it?
- Was the run complete, partial, or failed?
Store large immutable artifacts separately from the small metadata and evidence ledger. Do not overwrite raw evidence with a summary. The raw object is what makes reprocessing and audit possible.
Negative results need types
“No result” can mean at least four things:
- authoritative absence
- incomplete search
- temporary upstream failure
- denied access
Only the first is normally safe to negative-cache, and even then it needs a short, source-appropriate freshness rule. Never cache a timeout or permission failure as “not found.” That converts an infrastructure problem into false evidence.
Represent negative entries as structured states, not null:
{
"state": "upstream_unavailable",
"source": "manuals-index",
"retry_after": "2026-08-15T12:10:00Z"
}Match the store to the consistency requirement
Cloudflare Workers KV is optimized for read-heavy workloads and is eventually consistent. That can be appropriate for configuration snapshots or retrieval material where bounded staleness is acceptable. It is a poor source of truth for a just-revoked permission or a safety gate that must change immediately.
The Workers Cache API is data-center local and does not replicate cached objects across locations. Design for misses and regional differences. R2 lifecycle rules manage retention and storage-class transitions; they are not a substitute for application invalidation.
Redis expiry removes keys after a configured duration. It solves lifetime, not dependency tracking. Pair expiry with versioned keys or explicit invalidation events.
For high-impact decisions, read current authorization and policy from a store with the required consistency, then recompute from cached evidence if needed.
Isolate tenants in every dimension
A tenant identifier in a key is necessary but not sufficient. Also enforce tenant-scoped authorization before reads, separate encryption and retention policy where required, and prevent shared retrieval indexes from returning cross-tenant chunks. Logs and traces need the same treatment.
A robust lookup sequence is:
- Authorize the caller for the tenant and task.
- Compute the complete versioned key.
- Read the candidate entry.
- Validate provenance, status, and freshness.
- Recheck live policy for consequential actions.
- Return the value or recompute.
- Record hit, miss, rejection reason, and latency.
The best cache hit is not the one with the longest TTL. It is the one the system can prove was produced from the same relevant world.