The model was starting and the first job didn't know

If a lazy-start routine sets its state flag inside the worker thread rather than before launching it, the caller that triggered the start can read the old state and take the fallback path. The window is the few microseconds between thread creation and the thread's first line, which is small enough to never reproduce under a debugger and large enough to hit the very first request every time, because the first request is the one doing the starting.

The design, which was good

A transcription service loads a multi-gigabyte model on demand rather than at boot. Nothing is resident until the first job that needs it arrives, it starts in the background, jobs that show up meanwhile wait for it, and after an idle hour it stops and frees the memory.

There is also a cloud fallback for when the local model is not running, which is the thing that made this bug quiet. There was no error. Jobs just went to the slow, more expensive path.

The code

class ModelRuntime:
    def __init__(self):
        self.state = "stopped"

    def start(self):
        threading.Thread(target=self._load, daemon=True).start()

    def _load(self):
        self.state = "starting"      # <- first line of the new thread
        load_model()
        self.state = "ready"

# The caller:
if runtime.state == "stopped":
    runtime.start()

if runtime.state in ("starting", "ready"):
    wait_for_model()                 # the intended path
else:
    use_cloud_fallback()             # what actually ran

start() returns as soon as the thread is created. The thread has not necessarily executed a single instruction yet. The caller reads state on the next line, gets "stopped" because _load has not run, and goes to the fallback.

Why this is worse than an ordinary flaky race

Most races are probabilistic and hit a small fraction of requests. This one is systematic, and it targets the exact request you care about.

The only caller that observes the window is the one that called start(), and that is by definition the first job after an idle period. Every job after it reads "starting" or "ready" and behaves correctly. So the model loads, works fine, and the job that paid for loading it is the one job that does not get to use it.

The fix is one line in the right place

def start(self):
    # Published before the thread exists, so any caller that gets here
    # after us sees "starting" rather than the state we just left.
    self.state = "starting"
    threading.Thread(target=self._load, daemon=True).start()

The principle generalises past threads: publish a state transition in the code path that causes it, not in the code path that completes it. The same mistake with an await instead of a thread, or with a promise, reads the same way and is just as easy to miss.

Strictly this also wants a lock around the check-then-start, or two simultaneous first requests can both call start(). We had one already for that reason; it did not help here, because the problem was not two callers racing each other. It was one caller racing the thread it had just created.

Testing a window you cannot hit

You cannot reliably reproduce a microsecond gap by trying. You can remove the thread from the test and assert on the ordering directly.

def test_state_is_published_before_the_thread_starts(monkeypatch):
    seen = []
    # Never actually run the target: just record what the state was
    # at the moment the thread would have been created.
    monkeypatch.setattr(threading, "Thread",
                        lambda **kw: _Fake(on_start=lambda: seen.append(rt.state)))
    rt = ModelRuntime()
    rt.start()
    assert seen == ["starting"]      # fails against the original

This is the shape I reach for with most ordering bugs. Do not try to win the race. Replace the concurrent part with something that records the order, and assert the order.

Two more things the same investigation turned up

• A start that keeps failing retried forever. It now stops retrying once no job has needed the model for the idle period, and the next job that wants it tries again.
• A chunk past its timeout could not be cancelled, because cancel() cannot stop a running inference. Its slot went to the next chunk while the worker stayed busy, so the pool quietly lost capacity. A timed-out chunk now marks the pool broken, and stopping a broken pool kills its workers so a hung one does not keep its memory alongside the replacement.

All three are the same category: a state machine where some transition was not observable at the moment it mattered.

Threading mistakes are not all races. A suspending read wrapped in runBlocking and called from the UI thread is the same carelessness about which thread does the work, with an ANR at the end instead of a slow path.