One inference layer, five ways in
A routing service that lets five very different consumers hit the same models without each one reinventing the client.
AI Data Engineer: design and implementation
The problem
Different consumers wanted the same predictions in incompatible shapes. A dashboard wanted a synchronous call. A batch job wanted throughput. A live view wanted a stream. Each team was about to build its own client against the serving endpoints, which would have meant five different retry policies, five caches and five places for the contract to drift.
What I did
- Designed the routing layer and its request contract.
- Implemented validation, batching, caching and routing as separate, testable stages.
- Built the five integration paths and the persistence layer behind them.
- Load-tested the service and tuned it against the concurrency target.
Architecture
- Every request enters through one gateway and is normalised into a single internal shape, so downstream logic never branches on how the caller arrived.
- The pipeline is four ordered stages (validate, batch, cache, route), each independently testable. Most of the reliability comes from that ordering rather than from any one stage.
- Batching coalesces concurrent requests for the same model, which is what keeps latency flat as concurrency rises.
- Caching keys on the semantic content of a request, not the transport it arrived over, so a dashboard call and a scheduled job hit the same warm entry.
- Results persist to Unity Catalog Delta tables, which means predictions are queryable history rather than a fire-and-forget response.
- Five integration paths are supported: synchronous REST, client polling, server-sent events, WebSockets, and scheduled Databricks Jobs.
What went wrong first
Concurrency exposed the cache, not the models
Under load the bottleneck was not inference; it was duplicate work. Coalescing identical in-flight requests removed a large slice of the load before it ever reached a serving endpoint.
Streaming and batching pull in opposite directions
Server-sent events want to emit early; batching wants to wait for company. I capped the batch window so the streaming path stays responsive, and accepted slightly worse batch efficiency to keep the interactive path honest.
Five consumers means five failure modes
A dropped WebSocket and a failed scheduled job need different handling. Normalising at the edge but keeping transport-specific error surfaces stopped the abstraction from lying to its callers.
Results
- integration patterns on one contract
- 5integration patterns on one contract
- response time
- <2sresponse timeat 1,000 concurrent users in load testing
- client contract instead of five
- 1client contract instead of five
Consolidating on one inference layer meant new consumers could integrate in days rather than building a client from scratch. Every prediction became auditable, because they all land in the same governed tables.