Scheduling, Nice, and Why Load Average Isn't CPU

Load average is lying to you (kind of)

Your monitoring says load average 42 on a 16-core box, but CPU is 90% idle. Before you page anyone, you need to know what the scheduler actually counts — because on Linux, "load" is not CPU usage.

The kernel tracks tasks in a small set of states: R (running or runnable, sitting on a runqueue), S (interruptible sleep, waiting on an event), D (uninterruptible sleep, usually blocked in a syscall on I/O), Z (zombie), T (stopped). Load average is a decaying average of tasks in R plus D. That's the trick: 30 threads blocked on a slow NFS mount inflate load without touching a CPU cycle. Check per-task states with ps -eo pid,stat,comm,wchan | grep ' D'.

How CPU time is shared

Since kernel 2.6.23 the default scheduler is the Completely Fair Scheduler; kernel 6.6 (2023) folded in the EEVDF algorithm, which keeps the same bookkeeping idea: every runnable task accumulates vruntime, and the task that has received the least CPU so far runs next. Nice values (-20 most favored to 19 least, set via nice/renice or sched_setscheduler) scale how fast vruntime accumulates — a nice -5 process's virtual clock ticks slower, so it wins the race more often.

Two counters matter more than priorities in production:

vmstat 1 5        # cs column: context switches per second
pidstat -w 1      # cswch/s vs nvcswch/s: voluntary vs forced

Voluntary switches mean the task blocked on something (I/O, locks, futex). Non-voluntary switches mean the scheduler preempted it — you have more runnable work than cores, or a latency-sensitive thread is losing to bulk work.

Try this

# inflate load without using CPU: 20 tasks sleeping on disk I/O
for i in $(seq 20); do (dd if=/dev/sda of=/dev/null bs=1M iflag=direct &); done
sleep 5; cat /proc/loadavg; uptime
kill %1 2>/dev/null; kill $(jobs -p) 2>/dev/null

Field notes

• Steal time (st in top) means the hypervisor gave your vCPU's slot to another guest. Your service is slow and it's not even your process's fault — this is the classic noisy-neighbor symptom on shared hosts.
• SCHED_FIFO/SCHED_RR real-time threads preempt everything below them; a spinning RT thread can starve the box. Reserve them for actual latency-critical work.
• Context switches cost microseconds each (cache eviction, TLB pressure); 500k/s of cs with high sy% CPU usually means lock contention, not scheduling problems.