Hundreds of ffmpeg processes and a load average of 800

Per-job concurrency limits do not bound total concurrency. If each job spawns five ffmpeg processes and twenty jobs run at once, you have a hundred processes competing for the same cores, each one slower than if it had run alone. The cap has to go on the resource, shared across all jobs, not on the job.

What a load average of 800 looks like

A dev server with 4 vCPUs. Load average around 800, which means roughly 800 processes wanting a core and four cores to go round. SSH took a minute to give a prompt. Nothing had crashed and nothing was going to finish.

A single lecture was being processed. One. It had 18 ffmpeg cuts running at the same time, and each cut was itself a multithreaded decode.

The arithmetic nobody had done

Every individual limit in the system was reasonable. The problem was that they multiplied.

Whisper path:  6 lectures in flight  x  5 chunks each   =  30 cuts
Gemini path:  20 lectures in flight  x 20 chunks each   = 400 cuts
                                                   total  430 ffmpeg processes

Available: 4 vCPUs on the dev box, 16 on production.

Each of those numbers was chosen sensibly by someone looking at one layer. Five chunks in flight per lecture matched the worker layout. Twenty lectures in the background was what the queue allowed. Nobody owned the product of the two, because no single file contained both.

Why this got worse after an optimisation

This appeared right after I made each cut 15x faster by moving -ss before -i. Cheap cuts meant the pipeline could start many more of them before anything blocked, and the thing that had been limiting total concurrency was, accidentally, how slow each cut was.

That is worth sitting with. The old code had no cap either. It was protected by its own inefficiency, and removing the inefficiency removed the protection. You can ship a speedup and cause an outage with it.

Cap the resource, not the job

A single semaphore shared by every lecture, sized against the machine rather than against any one job.

# One global gate for the thing that eats CPU. Model calls and network
# calls stay fully parallel: they are not what is saturating the box.
CHUNK_CUT_SLOTS = int(os.getenv("CHUNK_CUT_SLOTS", "4"))
_cut_gate = asyncio.Semaphore(CHUNK_CUT_SLOTS)

async def cut_chunk(src: str, start: float, dur: float, dst: str) -> None:
    async with _cut_gate:
        await run_ffmpeg(src, start, dur, dst)   # unchanged command

The ffmpeg invocation is byte-for-byte what it was, so the chunk audio is identical and there is nothing to re-verify about correctness. Only the number running at once changed.

Gate only what contends

The semaphore wraps the cut and nothing else. The model inference and the API calls stay as parallel as they were, because they contend for different resources: inference runs in its own worker pool with its own sizing, and a network call is mostly waiting.

Putting the gate around the whole per-chunk pipeline would have been one line simpler and would have serialised the waiting as well as the work. The point of a cap is to stop oversubscribing one resource, not to make everything take turns.

Why four, and why it is an env var

Four is one per vCPU on the smallest machine this runs on. ffmpeg is already internally multithreaded, so going wider than core count mostly buys context switching. The right number differs per deployment, which is why it reads from the environment with a conservative default rather than being computed cleverly.

The symptom is distinctive enough to recognise next time. If throughput goes down as you add parallelism, and load average is far above your core count, you are not running out of CPU. You are spending it on scheduling.