Scale on what's queued and you'll remove the pod mid-job

An autoscaler driven by queue depth alone scales down as soon as pods dequeue the work, because the queue is now empty while the work is still running. For long jobs that is actively destructive: each terminated pod returns its job to the queue, the job restarts from the beginning, and the queue refills. Scale on queued plus running, and give the scale-down a stabilization window about as long as one job.

The failure

Transcribing one lecture takes several minutes. The obvious scaler is queue depth: more lectures waiting, more pods.

t0   10 queued,  0 running   -> scaler sees 10, scales to 5 pods
t1    0 queued, 10 running   -> scaler sees 0,  scales to 0
t2   pods terminate mid-lecture; the work is handed back
t3   10 queued,  0 running   -> scaler sees 10, scales to 5 pods
     ...

A pipeline that burns CPU continuously and completes nothing. Worse than no autoscaling, because a fixed replica count would at least have finished.

Why the obvious metric is wrong

Queue depth measures unstarted demand. What a scaler needs is total demand, and for short jobs those are close enough that the distinction never surfaces. A job that finishes in 200ms is barely in the running state.

For a job measured in minutes, almost all the demand is in the running state almost all the time. Queue depth then reads near zero for the entire period the system is busiest, which is precisely inverted.

Count both states

triggers:
  - type: prometheus
    metadata:
      # Work that still needs a pod: waiting plus in flight.
      query: |
        sum(transcript_lectures_queued) + sum(transcript_lectures_running)
      # One pod's capacity, so desired replicas = demand / capacity.
      threshold: "6"            # WHISPER_LECTURE_WORKERS per pod

advanced:
  horizontalPodAutoscalerConfig:
    behavior:
      scaleDown:
        # About one lecture's run time. Demand has to stay low for the
        # whole window before anything is removed.
        stabilizationWindowSeconds: 600
        policies:
          - type: Pods
            value: 1            # one pod at a time, never a cliff
            periodSeconds: 120

Three parts, each doing something distinct. The query stops the metric collapsing when work is claimed. The threshold is per-pod capacity, so the division gives the replica count directly. The stabilization window means a brief lull cannot trigger a removal, because the autoscaler takes the highest recommendation over the whole window.

Instrument the running count deliberately

This only works if the running gauge is honest, and the ways it goes wrong are the usual ways gauges go wrong. A pod that crashes without decrementing leaves demand permanently inflated and the deployment never scales down. A gauge decremented in a success path but not a failure path drifts in the other direction.

@contextmanager
def running_lecture():
    LECTURES_RUNNING.inc()
    try:
        yield
    finally:
        LECTURES_RUNNING.dec()      # every exit, including exceptions

The finally is the whole point, and it still does not survive a SIGKILL. Which is an argument for deriving the gauge from a source of truth, a lease or a claim row with a TTL, rather than from an in-process counter, if you can.

The alternative worth considering

If your jobs are genuinely discrete units of work, ScaledJob rather than ScaledObject is a better fit: one Kubernetes Job per unit, which runs to completion and is not subject to scale-down at all. We stayed with long-lived pods because the model takes time to load and we did not want to pay that per lecture.

That is the actual trade, and it is worth stating plainly. Long-lived pods amortise an expensive warm-up and inherit the scale-down problem. Per-job pods have no scale-down problem and pay the warm-up every time.