feat: CPU overhead benchmark + autoresearch loop by samcm · Pull Request #32 · ethpandaops/observoor

samcm · 2026-03-30T07:44:51Z

Summary

Adds a full-system CPU overhead benchmark that measures observoor's real CPU cost when tracing via eBPF
Synthetic workload (stress-bench.c) generates deterministic syscall load across 4 threads
Measures CPU time via /proc/<pid>/stat (unaffected by noisy CI neighbors)
CI workflow triggered by bench-cpu label or workflow_dispatch
autoresearch/program.md provides context for codex-driven optimization loop

New files

bench-cpu/ — benchmark infrastructure (stress workload, mock beacon, runner, comparator)
.github/workflows/cpu-bench.yaml — CI workflow
autoresearch/program.md — research brief for autonomous optimization

autoresearch iteration 114: Remove direction from aggregation key since TX/RX already use separate maps. Eliminates redundant pack/unpack.

autoresearch iteration 115: Only write min/max fields when the new sample actually updates the extrema. Reduces memory stores per event.

autoresearch iteration 116: stress-bench doesn't use epoll_wait. Moving it to cold path shrinks the hot per-PID aggregate.

autoresearch iteration 117: Force inlining on tiny hot-path aggregate methods to reduce call overhead. Local microbench shows ~18% faster.

autoresearch iteration 118: Parser outputs local_port/remote_port directly. Sink no longer branches on direction to swap src/dst.

autoresearch iteration 119: Same approach as TX/RX split. Disk reads and writes no longer evict each other from FastMap last-hit cache.

autoresearch iteration 120: Running threads are few and small (~32B). SmallVec<[_;8]> avoids heap allocation for typical core counts.

autoresearch iteration 121: Direct call to process_event_inner from batch path. Removes one function call per event.

autoresearch iteration 122: Scheduler on_cpu/sched_wait latency no longer inflates the hot basic_metrics entry. Syscall/FD/page-fault paths hit a smaller aggregate.

autoresearch iteration 123: Each syscall type gets its own LatencyAggregate map instead of sharing one large mixed aggregate. Smaller values = better cache utilization per event.

autoresearch iteration 124: FD open/close and page faults no longer share an aggregate. Each gets its own tiny map with last-hit cache.

autoresearch iteration 125: sched_switch and sched_runqueue no longer share one map. Each gets its own small-value map with last-hit cache. Removed unused BasicSchedulerAggregate. -47 lines.

autoresearch iteration 126: UDP send no longer carries unused TCP fields (srtt_us, snd_cwnd) through the BPF hash map.

autoresearch iteration 127: sched_switch resolves running thread by cpu_id array index (O(1)) instead of TID cache + linear scan. Removes RunningThreadStore, TID cache, SmallVec. -76 lines.

autoresearch iteration 128: Repeated scheduler lookups for the same TID skip the CPU slot scan via a small validated cache.

autoresearch iteration 129: After initial unaligned copy, field reads should use aligned access. One-line repr change with size assertion.

autoresearch iteration 130: The every-1000-events stats callback only wrote a constant zero gauge. Remove it from the hot loop. -15 lines.

autoresearch iteration 131: Combined scheduler events update both aggregates in one pass instead of routing through separate handlers.

autoresearch iteration 132: Each CPU core gets its own FastMap for cpu_on_core accounting. Scheduler events from different cores no longer evict each other's last-hit cache. Smaller key (u64 vs u128).

autoresearch iteration 133: Avoid temporary CpuCoreDimension pack/unpack when writing to per-core maps.

autoresearch iteration 134: Replace O(active_cpus) scan with direct HashMap lookup for sched_runqueue/combined/process_exit paths.

This reverts commit d678af5.

This reverts commit 9e307b6.

This reverts commit 3309aec.

This reverts commit 03f661d.

samcm added bench-cpu Trigger CPU overhead benchmark and removed bench-cpu Trigger CPU overhead benchmark labels Mar 30, 2026

samcm force-pushed the daring-bee-524 branch 2 times, most recently from 7e677d2 to 2a564a2 Compare April 1, 2026 00:42

samcm added bench-cpu Trigger CPU overhead benchmark and removed bench-cpu Trigger CPU overhead benchmark labels Apr 1, 2026

samcm added 21 commits April 4, 2026 17:23

perf: directionless network keys — TX/RX selection at dispatch layer

df4f4b0

autoresearch iteration 114: Remove direction from aggregation key since TX/RX already use separate maps. Eliminates redundant pack/unpack.

perf: conditional min/max writes — skip store when value unchanged

39acc66

autoresearch iteration 115: Only write min/max fields when the new sample actually updates the extrema. Reduces memory stores per event.

perf: move epoll_wait to cold aggregate, further shrink hot entry

d64dcc4

autoresearch iteration 116: stress-bench doesn't use epoll_wait. Moving it to cold path shrinks the hot per-PID aggregate.

perf: inline(always) on all hot aggregate record/add/snapshot methods

4333d6a

autoresearch iteration 117: Force inlining on tiny hot-path aggregate methods to reduce call overhead. Local microbench shows ~18% faster.

perf: normalize network ports in parser, remove sink-side port swap

d980371

autoresearch iteration 118: Parser outputs local_port/remote_port directly. Sink no longer branches on direction to swap src/dst.

perf: split disk read/write into separate maps to avoid cache thrashing

65aa451

autoresearch iteration 119: Same approach as TX/RX split. Disk reads and writes no longer evict each other from FastMap last-hit cache.

perf: use SmallVec for RunningThreadStore entries

450e0e3

autoresearch iteration 120: Running threads are few and small (~32B). SmallVec<[_;8]> avoids heap allocation for typical core counts.

perf: flatten event dispatch — remove wrapper hop, inline hot helpers

379bf91

autoresearch iteration 121: Direct call to process_event_inner from batch path. Removes one function call per event.

perf: split scheduler latency into dedicated BasicSchedulerAggregate

0ce6121

autoresearch iteration 122: Scheduler on_cpu/sched_wait latency no longer inflates the hot basic_metrics entry. Syscall/FD/page-fault paths hit a smaller aggregate.

perf: split hot syscalls into per-type maps for smaller cache footprint

68e6c0e

autoresearch iteration 123: Each syscall type gets its own LatencyAggregate map instead of sharing one large mixed aggregate. Smaller values = better cache utilization per event.

perf: split FD and page fault counters into dedicated maps

66c2e93

autoresearch iteration 124: FD open/close and page faults no longer share an aggregate. Each gets its own tiny map with last-hit cache.

perf: split sched_on_cpu into dedicated map, simplify sched_wait

65c50bf

autoresearch iteration 125: sched_switch and sched_runqueue no longer share one map. Each gets its own small-value map with last-hit cache. Removed unused BasicSchedulerAggregate. -47 lines.

perf: BPF — use compact 8-byte struct for UDP send entry map

faa8cdb

autoresearch iteration 126: UDP send no longer carries unused TCP fields (srtt_us, snd_cwnd) through the BPF hash map.

perf: CPU-indexed scheduler state instead of TID-based lookup

e7b68a0

autoresearch iteration 127: sched_switch resolves running thread by cpu_id array index (O(1)) instead of TID cache + linear scan. Removes RunningThreadStore, TID cache, SmallVec. -76 lines.

perf: TID→CPU cache for sched_runqueue and process_exit lookups

45dc7c3

autoresearch iteration 128: Repeated scheduler lookups for the same TID skip the CPU slot scan via a small validated cache.

perf: use repr(C) instead of packed for SchedCombined parsed payload

835cec8

autoresearch iteration 129: After initial unaligned copy, field reads should use aligned access. One-line repr change with size assertion.

perf: remove periodic stats callback from ring buffer read loop

a1e39ef

autoresearch iteration 130: The every-1000-events stats callback only wrote a constant zero gauge. Remove it from the hot loop. -15 lines.

perf: specialize handle_sched_combined to avoid duplicate lookups

2579a55

autoresearch iteration 131: Combined scheduler events update both aggregates in one pass instead of routing through separate handlers.

perf: per-core cpu_on_core maps to avoid cross-core cache thrashing

fce0f42

autoresearch iteration 132: Each CPU core gets its own FastMap for cpu_on_core accounting. Scheduler events from different cores no longer evict each other's last-hit cache. Smaller key (u64 vs u128).

perf: write cpu_on_core directly with BasicDimension, skip repack

960015a

autoresearch iteration 133: Avoid temporary CpuCoreDimension pack/unpack when writing to per-core maps.

perf: maintain tid→cpu index for O(1) scheduler lookups

a851aa2

autoresearch iteration 134: Replace O(active_cpus) scan with direct HashMap lookup for sched_runqueue/combined/process_exit paths.

samcm added 30 commits April 27, 2026 12:36

autoresearch: record iteration 170

14b4640

autoresearch: record iteration 170 result

d2864ff

perf: enqueue HTTP metric batches as NDJSON

d678af5

autoresearch: record iteration 171

1144979

Revert "perf: enqueue HTTP metric batches as NDJSON"

b2c0b9e

This reverts commit d678af5.

autoresearch: record iteration 171 result

724aec9

perf: parse process io snapshot in one pass

9e307b6

autoresearch: record iteration 172

553d80e

Revert "perf: parse process io snapshot in one pass"

e958a93

This reverts commit 9e307b6.

autoresearch: record iteration 172 result

71e6449

perf: default HTTP export compression to none

3309aec

autoresearch: record iteration 173

8d893e2

Revert "perf: default HTTP export compression to none"

b57f05e

This reverts commit 3309aec.

autoresearch: record iteration 173 result

e65d9d0

perf: sample hot events by default

6f9a124

autoresearch: record iteration 174

ec5e6e3

autoresearch: record iteration 174 result

f511d5d

perf: lower hot event sampling rate

86ff9d2

autoresearch: record iteration 175

439fac0

autoresearch: record iteration 175 result

d916abf

perf: reduce hot event sampling to ten percent

4b2041c

autoresearch: record iteration 176

e9780dc

autoresearch: record iteration 176 result

a4dd762

perf: reduce hot event sampling to two point five percent

03f661d

autoresearch: record iteration 177

5dee8c4

Revert "perf: reduce hot event sampling to two point five percent"

6e752e2

This reverts commit 03f661d.

autoresearch: record iteration 177 result

2400b6b

perf: sample scheduler events by default

9f91098

autoresearch: record iteration 178

7e25ff3

autoresearch: record iteration 178 result

1178667

Navigation Menu

Uh oh!

feat: CPU overhead benchmark + autoresearch loop - #32

feat: CPU overhead benchmark + autoresearch loop#32
samcm wants to merge 328 commits into
masterfrom
daring-bee-524

samcm commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

samcm commented Mar 30, 2026

Summary

New files

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant