Stopping one slow question from freezing an app
Diagnosing and fixing an event-loop stall in a Databricks Labs project I don't maintain.
External contributor
The problem
In OntoBricks' Graph Chat, one broad question could freeze the entire application until it was redeployed. Not slow: frozen, for every user at once. That is the kind of bug that looks like an infrastructure problem and is actually an architecture problem.
What I did
- Diagnosed the root cause: a blocking graph read executing directly on the single uvicorn event loop.
- Established the second half of the failure: reads were unbounded in time and result size, pinning a database session indefinitely.
- Implemented server-side bounds, off-loop execution and an auto-sized worker pool.
- Added regression tests covering both failure paths, and documented the trade-off for reviewers.
Architecture
- The application ran on a single uvicorn event loop. A deep traversal, an unfiltered lookup or a heavy resolver ran on that loop, so while it worked, nothing else could, including health checks.
- The fix has two halves. Blocking work moves off the event loop so a slow query occupies a worker instead of the whole process. Every read gains a server-side bound so it cannot run forever.
- Bound resolution follows an explicit order (request override, then environment, then default) with clamping so a caller cannot disable the protection.
- The worker pool auto-sizes to the instance rather than assuming a fixed count, because the deployment target varies.
- Statement timeouts reset on pooled connections, so a bound applied to one request can never leak to whoever borrows that connection next. That detail is easy to miss and causes very confusing bugs later.
- The result is a behaviour change, stated plainly in the PR: a genuinely oversized query now fails as one cancelled request instead of taking the application down.
What went wrong first
The reported symptom was not the bug
'Graph Chat is slow' pointed at query performance. Slowness was real but incidental. The reason it took the whole app down was that it ran on the shared event loop. Fixing only the query would have made the freeze rarer and no less total.
Bounds leak if you are careless with pools
Applying a statement timeout to a pooled connection silently affects the next borrower. Resetting it on release was a small change that prevented a class of bug that would have been very hard to attribute later.
Contributing to a codebase you don't own
This meant matching their conventions, their changelog format and their review expectations, and being explicit about what I could not verify locally: the toolchain would not fully install in my environment, so I stated exactly which subset I ran and left the rest to CI.
Results
- PR #116, labelled 'status: in progress'
- OpenPR #116, labelled 'status: in progress'
- milestone the maintainers accepted it into
- v0.7.0milestone the maintainers accepted it into
- PRs opened to the project
- 4PRs opened to the project
Most production incidents are not exotic. They are a blocking call on a shared resource with no upper bound. Recognising that shape quickly, in a codebase you have never seen, is most of the job.