Three threads per worker on AMD, two on Intel

For CPU inference, the best split of a fixed core budget between worker processes and threads per worker is not portable across CPU vendors. On a 16-core AMD machine, 5 workers at 3 threads each beat 7x2, 8x2, 4x4 and 12x1. On a 16-core Intel machine the same code was fastest at 2 threads per worker. The vendor also decides which math backend the runtime selects, which is most of why.

The question

A transcription service runs a speech model on CPU, with a pool of worker processes, each holding its own copy of the model and each allowed some number of threads. Sixteen cores is a budget you can spend many ways, and the layouts are not equivalent.

• More workers, fewer threads each: more chunks in flight, more memory, since each worker holds a full copy of the model.
• Fewer workers, more threads each: each chunk finishes sooner, less memory, more coordination overhead inside a single inference.

Someone had benchmarked this on Intel and settled on 2 threads per worker. Then the workload moved to AMD machines and nobody re-ran it, because the number looked like a property of the software.

The measurements

Same 24 chunks, same service, same pool code. Throughput as a multiple of real time, with every worker busy.

AMD EPYC Milan, 16 cores, no SMT

layout    throughput   median chunk
 5 x 3      3.69x         67 s      <- best
 4 x 4      3.66x         71 s
 8 x 2      3.63x         88 s
 7 x 2      3.42x         94 s
12 x 1      2.82x        140 s

5x3 against the inherited 7x2: 30% faster per chunk, 30% less memory.
Intel (n4-standard-16): 4 x 2 measured best. 2 threads stays correct there.

Two things stand out. 5x3 is both faster per chunk and lighter, because fewer workers means fewer copies of the model resident, so this is not the usual speed-versus-memory trade. And 12x1 is dramatically worse, which is the clearest signal that single-threaded inference wastes something the threaded path uses well.

Why the vendor matters

The inference runtime picks its math backend from the CPU vendor: Intel MKL on Intel, oneDNN elsewhere. Those are different implementations with different threading behaviour and different cache assumptions, so you are not tuning one piece of software across two machines. You are tuning two.

On top of that, this AMD part has no SMT, so 16 cores is 16 threads and not 32. Core count, hardware thread count and the number you should configure are three different numbers, and conflating them is how you end up at 12x1.

Encoding it without hardcoding a machine

def threads_per_worker() -> int:
    # An explicit setting always wins: benchmarks age, deployments vary.
    override = os.getenv("WHISPER_CPU_THREADS")
    if override:
        return int(override)

    # Measured, not guessed. See docs/whisper-benchmarks.md for the runs.
    # The vendor is also what switches the runtime between MKL and oneDNN.
    return 3 if cpu_vendor() == "AMD" else 2

Two decisions in that. The override comes first, because a measurement has a shelf life and whoever is on call should not need a deploy to change it. And the comment names where the numbers came from, so the next person can re-run them rather than inheriting two magic constants the way I did.

How to run this yourself

1. Fix the input set and reuse it exactly. Different audio has different silence and different difficulty, and both move the numbers more than the layout does.
2. Measure with the pool saturated. A single chunk on an idle box tells you about one inference, not about throughput.
3. Record memory alongside throughput. Here the fastest layout was also the lightest, and you only learn that if you log both.
4. Report the median per chunk as well as total throughput. Averages hide a layout where most chunks are quick and a few stall.

And size the pool itself from what the machine actually has, which is a separate trap: the pod had 15 GB of memory and 3 GB of it.