fix: Use cgroup-aware CPU monitoring inside Docker containers - #1099
Conversation
defaultCpuLoad used os.cpus() which returns host CPU counters inside containers, causing inflated load values that made the LiveKit server reject job dispatches with "no servers available". Replace with a cgroup-aware monitor (ported from the Python agents framework) that reads usage from /sys/fs/cgroup/ when available, falling back to os.cpus() on bare metal.
🦋 Changeset detectedLatest commit: aa4fe7a The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Resolves #1082 |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| #readCpuMax(): [string, number] { | ||
| try { | ||
| const data = readFileSync('/sys/fs/cgroup/cpu.max', 'utf-8').trim().split(/\s+/); | ||
| const quota = data[0] ?? 'max'; |
There was a problem hiding this comment.
🟡 Nullish coalescing (??) doesn't guard against empty-string quota, causing cpuCount() to return NaN
In CGroupV2CpuMonitor.#readCpuMax() at agents/src/cpu.ts:85, the expression data[0] ?? 'max' uses the nullish coalescing operator, which only falls back for null/undefined, not for empty strings. If /sys/fs/cgroup/cpu.max contains unexpected content (e.g., empty or whitespace-only), data[0] will be "", which passes the ?? check unchanged. Then at agents/src/cpu.ts:59, parseInt("") returns NaN, so cpuCount() returns NaN. This propagates through cpuPercent() where usageSeconds / (intervalSeconds * NaN) = NaN, and Math.max(Math.min(NaN, 1), 0) still evaluates to NaN. The NaN load value is then sent to the server in the updateWorker message (agents/src/worker.ts:655), and the NaN >= threshold check (agents/src/worker.ts:636) always evaluates to false, so the worker would be marked as available despite having an unknown load.
| const quota = data[0] ?? 'max'; | |
| const quota = data[0] || 'max'; |
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
defaultCpuLoadinworker.tsusedos.cpus()which returns host CPU counters inside containers, producing inflated load values that caused the LiveKit server to intermittently reject dispatches with "no servers available"Replace with a cgroup-aware CPU monitor that auto-detects cgroup v2/v1 and reads usage directly from /sys/fs/cgroup/, falling back to os.cpus() on bare metal
Supports NUM_CPUS environment variable override for cases where cgroup quotas don't reflect actual allocation
How it works
CGroupV2CpuMonitor/sys/fs/cgroup/cpu.maxquota/periodCGroupV1CpuMonitor/sys/fs/cgroup/cpu/cpu.cfs_quota_usDefaultCpuMonitoros.cpus().lengthVerified in Docker
os.cpus().length (host value): 14
Selected monitor: CGroupV2CpuMonitor
cpuCount(): 2 # from cgroup quota 200000/100000
cpuPercent(): 0.001143 # correct idle load
Load in [0, 1]: PASS