A hybrid coding agent should not ask "local or cloud?" once. It should route each step according to latency, capability, privacy, and failure cost.
The local side is best treated as an execution boundary, not merely a small language model. Parsers, linters, search indexes, test runners, and repository reads are deterministic, cheap, and close to the code. A local model can add classification or summarization when its measured quality is sufficient. The cloud side is useful when a task needs broader reasoning, a larger context budget, or managed infrastructure.
OpenAI's local-shell design makes this boundary explicit: the model proposes commands, but a developer-supplied runtime executes them locally and returns results. Ollama exposes a local HTTP API for model inference. Codex cloud environments show the other side of the split: code can be checked out and processed in an isolated remote environment. These are building blocks, not a routing policy.
Route operations, not whole sessions
Keep a small task envelope:
intent + repository version + allowed tools + data class + budget + deadlineThen classify each operation:
| Operation | Default location | Reason |
|---|---|---|
| Parse, lint, format, symbol lookup | Local | Deterministic and latency-sensitive |
| Run a focused unit test | Local sandbox | Fast feedback against the current tree |
| Retrieve private repository context | Local index first | Minimize unnecessary transfer |
| Cross-file design or ambiguous debugging | Cloud-capable reasoner | Benefits from stronger reasoning |
| Long build or isolated experiment | Local or cloud sandbox | Choose by compute, dependencies, and policy |
| External side effect | Wherever the tool lives, behind approval | Location does not remove authorization |
Cloudflare's Agents documentation separates three concerns that are often conflated: how the model sees a tool, where the tool executes, and where the tool comes from. Preserve that separation. A local tool can be invoked by a cloud model; a cloud API can be invoked through a local orchestrator.
Send a context packet, not the repository
The handoff to cloud reasoning should be explicit and inspectable:
{
"task": "repair failing authorization test",
"repo_revision": "commit hash",
"files": ["src/auth.ts", "test/auth.test.ts"],
"diagnostics": ["structured test failure"],
"constraints": ["no schema changes"],
"redactions": ["secrets", "customer records"]
}Do not assume that "cloud" has one privacy behavior. Provider retention and training controls are product-specific and can change. Check the provider's current data-control documentation, minimize transmitted data, and record what crossed the boundary.
Make fallback asymmetric
Fallback must preserve trust:
- If local inference is unavailable, deterministic checks should still run.
- If cloud reasoning is unavailable, queue or narrow the task; do not silently downgrade a safety-critical review.
- If the repository changes during a run, invalidate the handoff and rebase the evidence.
- If a tool would mutate external state, require the same approval regardless of where reasoning happened.
The orchestrator should also cap retries. Repeating the same failed route is not resilience.
Measure the router
Evaluate routing decisions separately from answer quality. Useful measures are local completion rate, cloud escalation rate, p50 and p95 latency, transferred bytes, cost per accepted task, retry rate, and quality by route. Compare a proposed policy against a pinned baseline. A cheaper route is only better if it clears the same correctness and safety gates.
The practical architecture is modest: deterministic local tools first, local inference where it has earned trust, cloud reasoning where it adds measured value, and one policy layer that can explain every handoff.