The pod had 15 GB of memory and 3 GB of it

Sizing a worker pool from total system memory is correct only when something enforces that the total is yours. Inside a container with a memory limit it is. On a VM shared with other services, or a pod deployed without a limit, the total overstates what you can actually take, and a pool sized from it gets killed by the OOM killer. Read the cgroup limit when there is one and fall back to memory currently free when there is not.

Where it bit

A speech-to-text service loads a model into one worker process per slot. Each worker holds roughly 4 GB. The sizing logic read total system memory, divided, and started that many workers.

On a shared dev box: 15 GB total, about 3 GB actually free, the rest held by other services that were there first. The arithmetic said three workers. There was room for zero, and the kernel resolved the disagreement.

total          15.0 GB   <- what the sizing read
in use         12.0 GB   <- other services, already resident
free            3.0 GB   <- what was actually available
per worker      4.0 GB
workers started       3  -> OOM

Three different questions

"How much memory is there" has at least three answers and they are not interchangeable.

• Total physical memory. What the hardware has. Says nothing about your share.
• The cgroup limit. What the kernel will let this container use before it kills it. A hard ceiling, and authoritative when it exists.
• Memory currently free. What you could allocate right now. The only honest answer when nothing is enforcing a share.

Inside a properly specified container the first two are the same, which is why reading total works in production and is exactly why this bug survives review. It is correct in the environment everyone tests in.

def available_memory_bytes() -> int:
    # 1. cgroup v2, then v1. A real limit is authoritative: the kernel
    #    enforces it, so it is a promise rather than an observation.
    for path in ("/sys/fs/cgroup/memory.max",
                 "/sys/fs/cgroup/memory/memory.limit_in_bytes"):
        try:
            raw = open(path).read().strip()
        except OSError:
            continue
        if raw == "max":
            break                       # cgroup exists, no limit set
        value = int(raw)
        # v1 reports a sentinel near 2^63 to mean "unlimited".
        if value < (1 << 62):
            return value

    # 2. No limit: only what is free right now can be counted on.
    try:
        meminfo = dict(
            (k.strip(), v) for k, v in
            (l.split(":", 1) for l in open("/proc/meminfo"))
        )
        return int(meminfo["MemAvailable"].split()[0]) * 1024
    except (OSError, KeyError):
        pass

    # 3. No /proc at all (macOS). Total is the only number available;
    #    say so in the log rather than pretending it is a limit.
    log.warning("no cgroup or /proc; sizing from total memory")
    return psutil.virtual_memory().total

MemAvailable rather than MemFree is the detail worth knowing. MemFree excludes the page cache, which the kernel will happily reclaim under pressure, so it dramatically understates what you can allocate on any machine that has been up for a while. MemAvailable is the kernel's own estimate of what a new allocation could get, and it is the number you want.

Why this is a fallback and not the default

Free memory is a measurement at one instant, and another process can take it a second later. A cgroup limit is a guarantee the kernel enforces. So where a limit exists it is used as-is, and free memory is only consulted when nothing is enforcing anything.

That ordering also means production behaviour does not change at all. The fallback only engages where the previous code was already wrong.

Two things that follow

Once the pool sizes itself from the machine, a single job can no longer saturate it. The code sent a fixed five chunks in flight, matching an old benchmark, so on a 7-worker machine one lecture kept five busy and left two idle at about 940% CPU out of 1600%. It now sends as many chunks as there are workers, never fewer than five.

And the number of threads each worker gets turned out to depend on the CPU vendor, which I did not expect and had to benchmark: three threads on AMD, two on Intel.

The general rule I would write on the wall: when code reads an environment limit to size itself, it should state which limit it read, in a log line, at startup. This was a five-minute diagnosis once the log said "sizing from total memory" on a machine where that was the wrong question.

Sizing the pool is one problem. Knowing whether a worker is actually up is another, and a flag set inside the thread instead of before it meant callers read 'stopped' and took the slow path.