bench(concurrency): add ragged paired benchmark - #596
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 75 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 75 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
93c9c5a to
cca54e7
Compare
cca54e7 to
65d4c1c
Compare
There was a problem hiding this comment.
All reported issues were addressed
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
65d4c1c to
1e67ef2
Compare
1e67ef2 to
8fdfb24
Compare
There was a problem hiding this comment.
All reported issues were addressed across 86 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
8fdfb24 to
b6479c9
Compare
b6479c9 to
f9b5e53
Compare
|
Added canonical workload concurrency coverage in b98d3c3:
Single-repeat Strix Halo screening on the exact raw HumanEval corpus (server binary SHA-256 31664e6f184b7cc37df9e719179d34c2b527ab36e02fd518cf08aef45aa65389):
All rows are 10/10 requests with exactly 128 output tokens. This is a screen, not a publication result; use REPEATS=5 on a clean target PR build for final numbers. The flat acceptance makes the C>=5 crossover an implementation-cost signal rather than a draft-quality failure. |
There was a problem hiding this comment.
All reported issues were addressed across 12 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
C=3/C=4 follow-up (Strix Halo, Qwen3.6-27B Q4_K_M target + Q8_0/SWA=2048 blog drafter, budget 22, 96 forced output tokens, five repeats):
An additional five-repeat C=4 adaptive-DDTree screen measured 30.36 tok/s on GSM8K (-4.4% vs AR) and 31.53 tok/s on Math500 (-1.0% vs AR). It reduces the low-yield GSM penalty, but did not beat AR in either C=4 suite. Practical result: fixed blog-DDTree is clearly best through C=3 in these reasoning workloads. At C=4, fixed DDTree is worthwhile only for a known high-acceptance Math-like workload; AR is the safer choice for GSM8K or unknown/mixed traffic. Existing HumanEval screening favors AR at C>=4. All requests completed with exact 96-token accounting and no failures. Output hashes varied across repeats for both AR and DDTree, so this is explicitly reported as a concurrent reproducibility warning rather than hidden. These remain screening measurements because the measured local PR605 binary/worktree was not a clean publication build. Commit 4527e85 adds CASE_LIMIT for exact full C=3 waves, adaptive-ddtree as an independently selectable variant, robust interrupt cleanup, output-stability reporting, tests, and the measured guidance. |
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
4527e85 to
b47d17f
Compare
There was a problem hiding this comment.
4 issues found across 12 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="harness/benchmarks/concurrency/canonical_concurrent_benchmark.py">
<violation number="1" location="harness/benchmarks/concurrency/canonical_concurrent_benchmark.py:121">
P2: When any wave request fails, this filter drops it and still attaches a DDTree proof for the remaining requests. Reject the proof before attachment when the measured level has any failed request, so the JSON report and printed acceptance metrics cannot describe a partial run.</violation>
</file>
<file name="harness/benchmarks/concurrency/summarize_concurrency.py">
<violation number="1" location="harness/benchmarks/concurrency/summarize_concurrency.py:181">
P2: A report with multiple client levels is silently reduced to `levels[0]`, so the summary can discard measurements without warning. Reject reports unless `levels` contains exactly one level before indexing it.</violation>
<violation number="2" location="harness/benchmarks/concurrency/summarize_concurrency.py:225">
P2: When prompt token usage is incomplete, `fixed_token_workload_valid` can still be true because canonical aggregation checks completion-token validity but not `prompt_token_count_complete`; the summary then crashes while taking the prompt-rate median. Require both token-count completeness flags before aggregating.</violation>
<violation number="3" location="harness/benchmarks/concurrency/summarize_concurrency.py:237">
P2: Repeated canonical reports can use different prompt files while sharing this grouping key, so the summarizer takes medians across incompatible workloads and evaluates stability as if they were repeats. Compare `report["prompt_file_sha256"]` and reject missing or mismatched hashes within each suite/C/case-limit family before aggregating.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| for level in report.get("levels", []) | ||
| for wave in level.get("wave_results", []) | ||
| for request in wave.get("requests_detail", []) | ||
| if request.get("error") is None |
There was a problem hiding this comment.
P2: When any wave request fails, this filter drops it and still attaches a DDTree proof for the remaining requests. Reject the proof before attachment when the measured level has any failed request, so the JSON report and printed acceptance metrics cannot describe a partial run.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/benchmarks/concurrency/canonical_concurrent_benchmark.py, line 121:
<comment>When any wave request fails, this filter drops it and still attaches a DDTree proof for the remaining requests. Reject the proof before attachment when the measured level has any failed request, so the JSON report and printed acceptance metrics cannot describe a partial run.</comment>
<file context>
@@ -85,6 +87,64 @@ def wait_for_retirement(path: Path, response_ids: list[str], timeout: float) ->
+ for level in report.get("levels", [])
+ for wave in level.get("wave_results", [])
+ for request in wave.get("requests_detail", [])
+ if request.get("error") is None
+ ]
+ totals = {key: 0 for key in DDTREE_COUNTERS}
</file context>
|
|
||
|
|
||
| def output_signature(report: dict[str, Any]) -> tuple[tuple[str, str, str], ...] | None: | ||
| level = report["levels"][0] |
There was a problem hiding this comment.
P2: A report with multiple client levels is silently reduced to levels[0], so the summary can discard measurements without warning. Reject reports unless levels contains exactly one level before indexing it.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/benchmarks/concurrency/summarize_concurrency.py, line 181:
<comment>A report with multiple client levels is silently reduced to `levels[0]`, so the summary can discard measurements without warning. Reject reports unless `levels` contains exactly one level before indexing it.</comment>
<file context>
@@ -172,15 +173,133 @@ def summarize(reports: list[dict]) -> str:
+
+
+def output_signature(report: dict[str, Any]) -> tuple[tuple[str, str, str], ...] | None:
+ level = report["levels"][0]
+ rows = []
+ for wave in level.get("wave_results", []):
</file context>
| level = report["levels"][0] | |
| levels = report.get("levels") | |
| if not isinstance(levels, list) or len(levels) != 1: | |
| raise ValueError(f"expected exactly one client level: {path}") | |
| level = levels[0] |
| raise ValueError(f"invalid case_limit: {path}") | ||
| if isinstance(repeat, bool) or not isinstance(repeat, int) or repeat < 1: | ||
| raise ValueError(f"missing or invalid repeat id: {path}") | ||
| if level["failures"] or level["fixed_token_workload_valid"] is not True: |
There was a problem hiding this comment.
P2: When prompt token usage is incomplete, fixed_token_workload_valid can still be true because canonical aggregation checks completion-token validity but not prompt_token_count_complete; the summary then crashes while taking the prompt-rate median. Require both token-count completeness flags before aggregating.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/benchmarks/concurrency/summarize_concurrency.py, line 225:
<comment>When prompt token usage is incomplete, `fixed_token_workload_valid` can still be true because canonical aggregation checks completion-token validity but not `prompt_token_count_complete`; the summary then crashes while taking the prompt-rate median. Require both token-count completeness flags before aggregating.</comment>
<file context>
@@ -172,15 +173,133 @@ def summarize(reports: list[dict]) -> str:
+ raise ValueError(f"invalid case_limit: {path}")
+ if isinstance(repeat, bool) or not isinstance(repeat, int) or repeat < 1:
+ raise ValueError(f"missing or invalid repeat id: {path}")
+ if level["failures"] or level["fixed_token_workload_valid"] is not True:
+ raise ValueError(f"invalid measured report: {path}")
+ if variant.endswith("ddtree"):
</file context>
| if level["failures"] or level["fixed_token_workload_valid"] is not True: | |
| if ( | |
| level["failures"] | |
| or level.get("token_count_complete") is not True | |
| or level.get("prompt_token_count_complete") is not True | |
| or level["fixed_token_workload_valid"] is not True | |
| ): |
| or proof.get("requests_proven") != requests | ||
| ): | ||
| raise ValueError(f"missing positive DDTree proof: {path}") | ||
| key = (suite, clients, variant, case_limit, requests) |
There was a problem hiding this comment.
P2: Repeated canonical reports can use different prompt files while sharing this grouping key, so the summarizer takes medians across incompatible workloads and evaluates stability as if they were repeats. Compare report["prompt_file_sha256"] and reject missing or mismatched hashes within each suite/C/case-limit family before aggregating.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At harness/benchmarks/concurrency/summarize_concurrency.py, line 237:
<comment>Repeated canonical reports can use different prompt files while sharing this grouping key, so the summarizer takes medians across incompatible workloads and evaluates stability as if they were repeats. Compare `report["prompt_file_sha256"]` and reject missing or mismatched hashes within each suite/C/case-limit family before aggregating.</comment>
<file context>
@@ -172,15 +173,133 @@ def summarize(reports: list[dict]) -> str:
+ or proof.get("requests_proven") != requests
+ ):
+ raise ValueError(f"missing positive DDTree proof: {path}")
+ key = (suite, clients, variant, case_limit, requests)
+ if repeat in repeat_ids[key]:
+ raise ValueError(f"duplicate repeat id {repeat} for {key}")
</file context>
1 participant
Dependency
Depends on #595 (packed ragged prefill), transitively on #594.
Scope
Adds benchmark-only tooling under
harness/benchmarks/concurrency/for reproducible Qwen3.6 concurrency measurements:K=8, LuceboxK=1, and llama.cpp continuous batching.Benchmark setup
run_qwen36_concurrency.shgenerates 29 deterministic, disjoint prompts forC=1,4,8,16. Each case starts a fresh server, waits for health, runs a same-concurrency warmup, then records the measurement. Variant order is rotated; the runner rejects prompt reuse and ambient GPU/backend tuning variables.run_qwen36_canonical_concurrency.shruns the checked-in prompt suites in complete fixed-width waves:C=1,2,5,10for ten-case suites andC=1,2,3,6for the six-case agent suite. It defaults toVARIANTS=ar;blog-ddtreeandadaptive-ddtreeare optional and require a server build that emits per-response[concurrency-metrics]telemetry.Each run writes prompts, server logs, warmup and benchmark reports, exact server commands, binary/model/prompt hashes, shared-library hashes, and output summaries under
OUT. Use at least five paired repeats for a publication-quality claim; the defaults are intended for screening.Running the benchmarks
Paired ragged workloads
Decode-heavy short-prompt comparison:
Canonical suites
Optional DDTree run:
Both scripts accept
--help. The main options are:MODELrequiredMODELrequired; optionalDRAFT_MODELfor DDTreeLUCE_SERVER_BIN,LLAMA_SERVER_BINSERVER_BINWORKLOADS=short,medium,longSUITES=he-raw,he,gsm,math,agentVARIANTS=luce-k8,luce-k1,llamaVARIANTS=ar,blog-ddtree, oradaptive-ddtreeCLIENTS=1,4,8,16CLIENTS=1,2,5,10by default; agent uses1,2,3,6REPEATS=1by defaultREPEATS=1by defaultMAX_TOKENS=64,WARMUP_TOKENS=8MAX_TOKENS=128,WARMUP_TOKENS=8CASE_LIMIT=<n>; use complete wavesOUT=...OUT=...SLOTS=16,PORT=18114,HEALTH_TIMEOUT_SECONDS=600,COOLDOWN_SECONDS=3SLOTS=16,PORT=18116,HEALTH_TIMEOUT_SECONDS=600,COOLDOWN_SECONDS=3Metrics
Output goodputis exact completion tokens divided by level wall time. It includes queueing, prefill, and decode; it is not decode-only throughput.Output-window tok/sstarts at the earliest observed first output and ends at the final request completion. It removes the initial all-prefill interval but can include staggered prefill, so it is decode-facing rather than pure kernel decode throughput.Request decode tok/sis the median per-request estimate(completion_tokens - 1) / (end - first_output).Strix Halo screening measurements vs llama.cpp
One fresh screening repeat was run on Strix Halo (Radeon 8060S) after integrating the gfx1151 hipBLASLt default and Q4_K 64-row/four-warp RDNA MMQ specialization from #595. It used Qwen3.6-27B Q4_K_M, Q4_0 K/V cache, greedy sampling, and fixed 64-token outputs. Every row uses a fresh server process and same-concurrency warmup.
All 116 measured requests completed successfully with exact 64-token usage counts. These are one-repeat screening measurements, not publication-quality estimates; use at least five paired repeats for a final performance claim.
Decode-heavy Strix Halo screening
A fresh one-repeat paired screen used the same short ragged cohorts with 256 forced output tokens after integrating the gfx1151 hipBLASLt default and Q4_K RDNA MMQ specialization. Generation therefore dominates more of each request.
Validation
git diff --check