Distributed Inference at Continental Scale
Operating inference across regions, providers, and hardware.
How we run low-latency inference across continents — routing, caching, and failover patterns for mixed CPU/GPU/NPU fleets. A practitioner case study drawn from operating Zansoc’s Distributed Cloud fabric.
Routing
Latency- and cost-aware model routing.
Caching
Semantic caches for repeat and near-repeat traffic.
Failover
Region and provider failover with SLA guarantees.
Observability
Deep telemetry for hybrid inference fleets.
Operating reality
Capacity cliffs, provider incidents, and diurnal traffic swings are normal. This case study documents the control loops we use to keep p99 latency inside error budgets while blending spot and reserved accelerators.
Routing and caches
We describe semantic caches for near-duplicate prompts, prefix caches for multi-turn agents, and placement policies that prefer warm replicas within a latency radius of the user.
Lessons for builders
The appendix includes failure stories — thundering herds after regional recovery, cache stampede mitigations, and why GPU metrics alone are insufficient without request-level quality signals.
Discuss this work with our labs.