Skip to content

perf: add cold-start CodSpeed scenarios + tight-polling readiness (refs #557) - #595

Merged
Tagar merged 1 commit into
py4j:masterfrom
byteoak:upstream-cold-start-metrics
May 28, 2026
Merged

perf: add cold-start CodSpeed scenarios + tight-polling readiness (refs #557)#595
Tagar merged 1 commit into
py4j:masterfrom
byteoak:upstream-cold-start-metrics

Conversation

@Tagar

@Tagar Tagar commented May 28, 2026

Copy link
Copy Markdown
Member

Summary

Adds 4 CodSpeed scenarios that map 1:1 to py4j cold-start improvement work areas (refs #557), plus tightens the perf-framework JVM readiness check so cold-start scenarios get enough samples per CodSpeed budget.

Pure measurement infrastructure — zero behavior change in py4j proper. Touches only the perf testing subtree.

This is the foundation for a follow-up PR adding the actual cold-start fix (attribute memoization, ~50 LOC) — landing the metrics first means the follow-up's PR description can cite "the XA-3 scenario added here now shows ×200 improvement on this branch" rather than asking reviewers to trust unmeasured claims.

New scenarios

ID What Why
XA-3 1 000 walks of gateway.jvm.java.lang.System Attribute-resolution cache canary. Each walk re-issues 3 reflection round-trips today; should collapse to ~0 RTTs after first walk once the upcoming memoization PR lands.
XB 50 fresh JavaGateway() against the fixture JVM per round Per-connection setup cost. Surfaces Java-side accept-loop allocations + Python-side _create_connection + socket setup.
XC X1-1 workload with CallbackServer started but no callbacks invoked Delta vs X1-1 = steady-state callback-infrastructure overhead.
XD subprocess.Popen → first call → shutdown, one cycle per round Full cold start. Target of any JVM-flag work (AppCDS, -XX:TieredStopAtLevel=1, CRaC).

XA/XB/XC share the long-running fixture JVM through test_macro_scenario. XD owns its full JVM lifecycle inside measure() and is dispatched via a separate test_cold_start_scenario test function — it can't share the fixture because both default to port 25333.

Readiness model overhaul

perf.jvm.fresh_jvm shifts from blind sleep to tight polling:

Before After
startup_sleep 0.25 s 0.0 s
readiness_retries 1 300
poll interval 2.0 s (hardcoded) 0.05 s
Total ceiling 2.25 s 15 s

Why this matters: under the old model each XD iteration cost ~2.7 s, leaving CodSpeed only 1 sample per benchmark budget — too few for variance / regression detection. Under the new model XD per-iteration finishes in ~500-1100 ms (host-dependent), enough for 2-5 samples per budget.

The unconditional 0.25 s startup_sleep's original rationale ("let the OS reuse the listen port") no longer applies: GatewayServer.startSocket() already uses SO_REUSEADDR, so bind() succeeds immediately even over TIME_WAIT. Backwards compatibility preserved — callers that need the old behavior can pass startup_sleep=0.25 explicitly.

Provenance

These changes originated as three iterative PRs on a personal fork (byteoak#12, #13, #16), bundled here as one logical "make cold start measurable" surface for upstream review. Full 56-cell CI matrix (Python × Java × OS) green on each byteoak iteration.

Test plan

  • All 10 macro scenarios + cold-start scenario pass locally (M-series + JDK 21)
  • Existing X1–X8 scenarios unchanged
  • python -m py4j.tests.perf --only XA-3,XB,XC --quick passes through the framework runner
  • Matrix CI green across Python 3.9–3.13 × Java 8/11/17/21 × ubuntu/macos/windows
  • CodSpeed picks up XA-3 / XB / XC / XD as 4 new tracked benchmarks on master after merge

py4j#557)

Adds four CodSpeed scenarios mapping 1:1 to py4j cold-start improvement
work areas, plus tightens the perf-framework JVM readiness check so
slow scenarios get enough samples per benchmark budget.

Pure measurement infrastructure — zero behavior change in py4j proper.
Touches only the perf testing subtree.

New scenarios:
* XA-3 — attribute_walk_jvm_java_lang_System
    1 000 walks of ``gateway.jvm.java.lang.System``. Each walk
    re-issues 3 reflection round-trips today because JVMView /
    JavaPackage / JavaClass __getattr__ don't memoize. Cache canary
    for the upcoming Python attribute-cache PR.
* XB — gateway_reconnect_against_running_jvm
    50 fresh ``JavaGateway()`` against the fixture JVM per round.
    Measures per-connection setup cost — Java accept() loop allocates
    14 command class instances per connection
    (GatewayConnection.java:196-208), Python runs _create_connection
    + socket setup.
* XC — callback_infra_overhead_unused
    X1-1 workload with ``CallbackServer`` started, but no callbacks
    actually invoked. Delta vs X1-1 = the steady-state cost of
    running callback infrastructure alongside the main gateway.
* XD — full_cold_start_subprocess_first_call
    subprocess.Popen -> first call -> shutdown, one cycle per round.
    Slow per-iteration; CodSpeed adapts iteration count. Target of
    the JVM-flag work (AppCDS, -XX:TieredStopAtLevel=1, CRaC).

XA/XB/XC share the long-running fixture JVM (``test_macro_scenario``).
XD owns its full JVM lifecycle inside ``measure()`` and is dispatched
via a separate ``test_cold_start_scenario`` test function — it can't
share the fixture because both default to port 25333.

Readiness model overhaul in ``perf.jvm.fresh_jvm``:

  before: 0.25 s sleep + 1 retry x 2.0 s = 2.25 s ceiling
  after:  0 s sleep + 300 retries x 0.05 s = 15 s ceiling

Tight polling matters for XD on CodSpeed: under the old model each
XD iteration cost ~2.7 s and CodSpeed could fit only one sample per
benchmark budget, making variance and regression detection
impossible. Under the new model XD per-iteration finishes in
~500-1100 ms — enough for 2-5 samples per budget. The unconditional
0.25 s startup_sleep's original rationale ("let the OS reuse the
listen port") no longer applies: GatewayServer.startSocket() already
uses SO_REUSEADDR, so bind() succeeds immediately even over
TIME_WAIT. Backwards compatibility preserved — callers can pass
``startup_sleep=0.25`` to restore the old wait pattern.

Validated on #12, #13, #16 (where these changes
originated as 3 iterative PRs; bundled here for a clean single
upstream review surface). Full 56-cell CI matrix green on each
byteoak iteration. CodSpeed scenarios surface real cold-start work
on subsequent PRs — issue py4j#557.

Co-authored-by: Isaac
@Tagar
Tagar requested a review from HyukjinKwon May 28, 2026 04:33
@Tagar
Tagar merged commit f92620d into py4j:master May 28, 2026
112 of 113 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants