Skip to content

docs: update incremental benchmarks (3.4.0) #627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlos-alm merged 4 commits into from
Mar 26, 2026
Merged

docs: update incremental benchmarks (3.4.0) #627

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fca7cef
docs: update incremental benchmarks (3.4.0)
github-actions[bot] Mar 26, 2026
93d94fd
Merge branch 'main' into benchmark/incremental-v3.4.0-20260326-073741
carlos-alm Mar 26, 2026
3e19b2e
Merge branch 'main' into benchmark/incremental-v3.4.0-20260326-073741
carlos-alm Mar 26, 2026
b05bdd5
Merge branch 'main' into benchmark/incremental-v3.4.0-20260326-073741
carlos-alm Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 63 additions & 11 deletions generated/benchmarks/INCREMENTAL-BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Import resolution: native batch vs JS fallback throughput.

| Version | Engine | Files | Full Build | No-op | 1-File | Resolve (native) | Resolve (JS) |
|---------|--------|------:|-----------:|------:|-------:|------------------:|-------------:|
| 3.4.0 | native | 473 | 2.3s ↓4% | 12ms ↑9% | 448ms ↑29% | 4ms ↓58% | 12ms ↓54% |
| 3.4.0 | wasm | 473 | 5.0s ↑15% | 12ms ↑20% | 617ms ↑21% | 4ms ↓58% | 12ms ↓54% |
Comment on lines +9 to +10
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Disproportionate 1-file rebuild regression

The 1-file rebuild time increased +29% for native (346ms → 448ms) and +21% for WASM (508ms → 617ms), but the file count only grew by ~7% (442 → 473 files). For a single-file incremental rebuild the total file count should have minimal impact on latency, so this regression is disproportionate to the codebase growth.

The WASM full build also regressed +15% (4.4s → 5.0s), widening the native/WASM gap from ~1.85× to ~2.21×.

These regressions are faithfully recorded here and the numbers are correct — this is just a heads-up that the underlying performance delta in 3.4.0 may be worth investigating before the next release.

Copy link
Copy Markdown
Contributor

@carlos-alm carlos-alm Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — the 1-file rebuild regression (+29% native, +21% WASM) is disproportionate to the ~7% file count growth and worth investigating. Since this is an auto-generated benchmark recording and the numbers are correct, the fix belongs in the codebase (likely in the incremental rebuild path), not in this PR. Will track separately.

Copy link
Copy Markdown
Contributor

@carlos-alm carlos-alm Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracked in #635 for follow-up investigation. The regression is real and worth looking into, but as you noted, the benchmark data here is correct — the fix belongs in the incremental rebuild path, not in this docs PR.

| 3.3.1 | native | 442 | 2.4s ↓22% | 11ms ↑10% | 346ms ↓17% | 9ms ↓6% | 26ms ~ |
| 3.3.1 | wasm | 442 | 4.4s ↑2% | 10ms ↓9% | 508ms ↑6% | 9ms ↓6% | 26ms ~ |
| 3.3.0 | native | 429 | 3.0s ↑141% | 10ms ↑67% | 417ms ↑74% | 10ms ↑700% | 26ms ↑888% |
Expand Down Expand Up @@ -35,37 +37,87 @@ Import resolution: native batch vs JS fallback throughput.

### Latest results

**Version:** 3.3.1 | **Files:** 442 | **Date:** 2026-03-20
**Version:** 3.4.0 | **Files:** 473 | **Date:** 2026-03-26

#### Native (Rust)

| Metric | Value |
|--------|------:|
| Full build | 2.4s |
| No-op rebuild | 11ms |
| 1-file rebuild | 346ms |
| Full build | 2.3s |
| No-op rebuild | 12ms |
| 1-file rebuild | 448ms |

#### WASM

| Metric | Value |
|--------|------:|
| Full build | 4.4s |
| No-op rebuild | 10ms |
| 1-file rebuild | 508ms |
| Full build | 5.0s |
| No-op rebuild | 12ms |
| 1-file rebuild | 617ms |

#### Import Resolution

| Metric | Value |
|--------|------:|
| Import pairs | 665 |
| Native batch | 9ms |
| JS fallback | 26ms |
| Import pairs | 891 |
| Native batch | 4ms |
| JS fallback | 12ms |
| Per-import (native) | 0ms |
| Per-import (JS) | 0ms |
| Speedup ratio | 2.9x |
| Speedup ratio | 3.1x |

<!-- INCREMENTAL_BENCHMARK_DATA
[
{
"version": "3.4.0",
"date": "2026-03-26",
"files": 473,
"wasm": {
"fullBuildMs": 5033,
"noopRebuildMs": 12,
"oneFileRebuildMs": 617,
"oneFilePhases": {
"setupMs": 1.9,
"parseMs": 223,
"insertMs": 14.1,
"resolveMs": 1.6,
"edgesMs": 13.6,
"structureMs": 29,
"rolesMs": 69.1,
"astMs": 0.6,
"complexityMs": 0.7,
"cfgMs": 0.5,
"dataflowMs": 0.5,
"finalizeMs": 14.6
}
},
"native": {
"fullBuildMs": 2276,
"noopRebuildMs": 12,
"oneFileRebuildMs": 448,
"oneFilePhases": {
"setupMs": 2,
"parseMs": 73.5,
"insertMs": 14.3,
"resolveMs": 1.6,
"edgesMs": 24.5,
"structureMs": 39.6,
"rolesMs": 54.3,
"astMs": 0.6,
"complexityMs": 0.6,
"cfgMs": 0.4,
"dataflowMs": 0.4,
"finalizeMs": 9.7
}
},
"resolve": {
"imports": 891,
"nativeBatchMs": 3.8,
"jsFallbackMs": 11.8,
"perImportNativeMs": 0,
"perImportJsMs": 0
}
},
{
"version": "3.3.1",
"date": "2026-03-20",
Expand Down
Loading