feat: add HTTP/2 load testing infrastructure#798
Open
jrvb-rl wants to merge 2 commits into
Open
Conversation
Adds a loadtest/ directory with scripts for benchmarking HTTP/2 transport throughput against the Runloop API. Includes baselines for raw node:http2, undici, and the SDK client at various concurrency levels. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
CodeAnt AI is reviewing your PR. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
codeant-ai
Bot
added
the
size:XL
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
| return { latencyMs: performance.now() - start, status: res.status }; | ||
| }); | ||
|
|
||
| const results = await Promise.all(promises); |
Contributor
There was a problem hiding this comment.
Suggestion: Using fail-fast aggregation here means one transient network failure rejects the entire run and discards all other completed results, which makes the benchmark unstable and non-comparable. Collect per-request failures into result objects (as done in the SDK loadtest script) instead of failing the whole batch. [logic error]
Severity Level: Major ⚠️
- ❌ undici HTTP/2 baseline test aborts on single network error.
- ⚠️ Successful requests' metrics are lost, reducing benchmark stability.Steps of Reproduction ✅
1. Run `REQUEST_COUNT=1000 npx tsx loadtest/undici-test.ts`, which executes `main()`
defined in `loadtest/undici-test.ts:22-88`.
2. In `main()`, an undici `Agent` dispatcher is created at
`loadtest/undici-test.ts:23-31`, and `REQUEST_COUNT` fetch promises are built at lines
47-61, each issuing a POST to `${BASE_URL}/v1/devboxes`.
3. All per-request promises are aggregated with `const results = await
Promise.all(promises);` at `loadtest/undici-test.ts:63`; if any `fetch(...)` rejects due
to a network/TLS error or similar transport issue, `Promise.all` rejects immediately.
4. The rejection from `Promise.all` prevents computation of latencies and status counts at
`loadtest/undici-test.ts:69-73`, and the outer `main().catch(...)` at lines 90-92 logs the
error and exits, discarding all completed results and leaving the benchmark run with no
throughput/latency metrics.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** loadtest/undici-test.ts
**Line:** 63:63
**Comment:**
*Logic Error: Using fail-fast aggregation here means one transient network failure rejects the entire run and discards all other completed results, which makes the benchmark unstable and non-comparable. Collect per-request failures into result objects (as done in the SDK loadtest script) instead of failing the whole batch.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
Comment on lines
+56
to
+60
| const clients = await Promise.all( | ||
| Array.from({ length: NUM_CONNECTIONS }, () => connectH2(url.origin)), | ||
| ); | ||
|
|
||
| const maxStreams = clients[0].remoteSettings?.maxConcurrentStreams; |
Contributor
There was a problem hiding this comment.
Suggestion: NUM_CONNECTIONS is fully user-configurable, but zero is not validated; with 0, no clients are created and this access dereferences undefined, crashing before any test runs. Validate that connection count is at least 1 before building clients and reading clients[0]. [incorrect condition logic]
Severity Level: Major ⚠️
- ❌ HTTP/2 baseline script crashes with NUM_CONNECTIONS=0.
- ⚠️ Load testing workflow fails before sending any requests.Steps of Reproduction ✅
1. In `/workspace/api-client-ts/loadtest/h2-test.ts`, `NUM_CONNECTIONS` is parsed from
`process.env.NUM_CONNECTIONS` at line 4 with default `"10"`.
2. From the `loadtest/` directory, run the HTTP/2 baseline script with zero connections,
e.g. `RUNLOOP_API_KEY=... NUM_CONNECTIONS=0 REQUEST_COUNT=10 npx tsx h2-test.ts`.
3. In `main()` (lines 50–57), `NUM_CONNECTIONS` is 0, so `Array.from({ length:
NUM_CONNECTIONS }, ...)` at line 57 produces an empty array and `Promise.all(...)`
resolves `clients` to `[]`.
4. At line 60, the code reads `clients[0].remoteSettings?.maxConcurrentStreams`; since
`clients[0]` is `undefined`, this throws a `TypeError` and the script crashes before any
HTTP/2 requests are sent or throughput is measured.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** loadtest/h2-test.ts
**Line:** 56:60
**Comment:**
*Incorrect Condition Logic: `NUM_CONNECTIONS` is fully user-configurable, but zero is not validated; with `0`, no clients are created and this access dereferences `undefined`, crashing before any test runs. Validate that connection count is at least 1 before building clients and reading `clients[0]`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix| console.log(`Wall clock: ${(wallMs / 1000).toFixed(2)}s`); | ||
| console.log(`Throughput: ${(REQUEST_COUNT / (wallMs / 1000)).toFixed(1)} req/s`); | ||
| console.log(`\nLatency (ms):`); | ||
| console.log(` min: ${latencies[0].toFixed(1)}`); |
Contributor
There was a problem hiding this comment.
Suggestion: If request count is zero, no latency values exist and this call throws because latencies[0] is undefined. Add an empty-result guard before printing latency statistics. [type error]
Severity Level: Major ⚠️
- ❌ HTTP/2 metrics crash when REQUEST_COUNT resolves to zero.
- ⚠️ No summary stats produced for zero-request dry runs.Steps of Reproduction ✅
1. In `/workspace/api-client-ts/loadtest/h2-test.ts`, `REQUEST_COUNT` is parsed from
`process.env.REQUEST_COUNT` at line 3 with default `"10000"`.
2. From the `loadtest/` directory, run the script with zero requests, e.g.
`RUNLOOP_API_KEY=... REQUEST_COUNT=0 npx tsx h2-test.ts` (or set `REQUEST_COUNT` to a
non-numeric string so `parseInt` yields `NaN`, which produces an array of length 0).
3. In `main()` (lines 71–81), `Array.from({ length: REQUEST_COUNT }, ...)` produces an
empty `promises` array, so `results` resolves to `[]` and `latencies` at line 87 becomes
an empty array after mapping and sorting.
4. At line 99, the script executes `console.log(\` min: ${latencies[0].toFixed(1)}\`);`;
since `latencies[0]` is `undefined`, calling `.toFixed(1)` throws a `TypeError`, causing
the metrics reporting step to crash instead of cleanly handling the zero-request case.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** loadtest/h2-test.ts
**Line:** 99:99
**Comment:**
*Type Error: If request count is zero, no latency values exist and this call throws because `latencies[0]` is `undefined`. Add an empty-result guard before printing latency statistics.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix| ); | ||
| console.log(""); | ||
| console.log("Latency (ms):"); | ||
| console.log(` min: ${latencies[0].toFixed(1)}`); |
Contributor
There was a problem hiding this comment.
Suggestion: When REQUEST_COUNT resolves to 0 (or an invalid value that yields zero requests), latencies is empty and this toFixed call throws at runtime because latencies[0] is undefined. Add a guard for empty results before computing percentile/min/max metrics. [type error]
Severity Level: Major ⚠️
- ❌ SDK-level load test crashes for REQUEST_COUNT resolving to zero.
- ⚠️ No metrics summary available for zero-request configuration runs.Steps of Reproduction ✅
1. In `/workspace/api-client-ts/loadtest/loadtest.ts`, `REQUEST_COUNT` is parsed from
`process.env.REQUEST_COUNT` at line 3 with default `"100000"`.
2. From the `loadtest/` directory, run the SDK-level load test with zero requests, e.g.
`RUNLOOP_API_KEY=... REQUEST_COUNT=0 npm test` (which runs `tsx loadtest.ts`), or set
`REQUEST_COUNT` to a non-numeric string so `parseInt` yields `NaN`.
3. In `main()` at lines 130–139, `Array.from({ length: REQUEST_COUNT }, ...)` evaluates to
an empty array when `REQUEST_COUNT` is 0/NaN, so no `sendRequest()` calls are made and
`results` resolves to `[]`.
4. `printMetrics(results, wallClockMs)` is called at line 144; inside `printMetrics`
(lines 73–100), `latencies` becomes an empty array at line 74, and the log statement at
line 90 executes `latencies[0].toFixed(1)`, which throws a `TypeError` because
`latencies[0]` is `undefined`, causing the load test to crash instead of gracefully
handling the zero-request scenario.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** loadtest/loadtest.ts
**Line:** 90:90
**Comment:**
*Type Error: When `REQUEST_COUNT` resolves to `0` (or an invalid value that yields zero requests), `latencies` is empty and this `toFixed` call throws at runtime because `latencies[0]` is `undefined`. Add a guard for empty results before computing percentile/min/max metrics.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
Contributor
|
CodeAnt AI finished reviewing your PR. |
Add res.on('error', reject) so the promise settles if the server
aborts mid-response, instead of hanging indefinitely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Summary
Scripts
Usage
Test plan
Generated with Claude Code
CodeAnt-AI Description
Add HTTP/2 load testing scripts for the Runloop API
What Changed
Impact
✅ Faster HTTP/2 performance comparisons✅ Clearer load test results✅ Easier API transport benchmarking💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.