Skip to content

docs: update incremental benchmarks (3.6.0) #714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlos-alm merged 4 commits into from
Mar 31, 2026
+51 −8
Merged

docs: update incremental benchmarks (3.6.0) #714

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e820c28
docs: update incremental benchmarks (3.6.0)
github-actions[bot] Mar 30, 2026
839aa86
docs(benchmarks): explain missing native engine row for 3.6.0
carlos-alm Mar 30, 2026
e77a2bd
Merge branch 'main' into benchmark/incremental-v3.6.0-20260330-164447
carlos-alm Mar 31, 2026
5f43842
Merge branch 'main' into benchmark/incremental-v3.6.0-20260330-164447
carlos-alm Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 51 additions & 8 deletions generated/benchmarks/INCREMENTAL-BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Import resolution: native batch vs JS fallback throughput.

| Version | Engine | Files | Full Build | No-op | 1-File | Resolve (native) | Resolve (JS) |
|---------|--------|------:|-----------:|------:|-------:|------------------:|-------------:|
| 3.6.0 | wasm | 514 | 6.1s | 13ms | 545ms | 4ms ↓3% | 12ms ↑9% |
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing native engine row with no explanation

3.6.0 has "native": null in the embedded JSON data, so only a wasm row appears in the summary table. For 3.5.0, the workflow generated a <!-- NOTES_START --> ... <!-- NOTES_END --> block explaining why both engines were null (napi-rs serialization bug, WAL race condition). That note has now been removed since 3.6.0 is the latest.

However, native was working in 3.4.1, and readers of this file have no indication of why the native engine is absent for 3.6.0. It may be worth adding a brief note in the "Latest results" section (or a comment in the JSON entry) explaining whether the native engine was intentionally skipped, failed during the benchmark run, or is being phased out.

Copy link
Copy Markdown
Contributor

@carlos-alm carlos-alm Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Added a blockquote note in the "Latest results" section explaining that the native engine crashed with SQLITE_CORRUPT during the incremental rebuild tier (WAL corruption in detect-changes / purgeAndAddReverseDeps). See 839aa86.

| 3.4.1 | native | 473 | 2.4s ↑3% | 13ms ↑8% | 331ms ↓26% | 4ms ~ | 13ms ↑8% |
| 3.4.1 | wasm | 473 | 5.1s ~ | 13ms ↑8% | 511ms ↓17% | 4ms ~ | 13ms ↑8% |
| 3.4.0 | native | 473 | 2.3s ↓4% | 12ms ↑9% | 448ms ↑29% | 4ms ↓58% | 12ms ↓54% |
Expand Down Expand Up @@ -39,25 +40,67 @@ Import resolution: native batch vs JS fallback throughput.

### Latest results

**Version:** 3.5.0 | **Files:** 0 | **Date:** 2026-03-30
**Version:** 3.6.0 | **Files:** 514 | **Date:** 2026-03-30

> **Note:** The native engine is absent for 3.6.0 because the native worker
> crashed with `SQLITE_CORRUPT` during the incremental rebuild tier
> (WAL corruption in `detect-changes` → `purgeAndAddReverseDeps`).
> WASM results are unaffected. Tracked as a known WAL race condition
> between native and JS connections.

#### WASM

| Metric | Value |
|--------|------:|
| Full build | 6.1s |
| No-op rebuild | 13ms |
| 1-file rebuild | 545ms |

#### Import Resolution

| Metric | Value |
|--------|------:|
| Import pairs | 896 |
| Import pairs | 904 |
| Native batch | 4ms |
| JS fallback | 11ms |
| JS fallback | 12ms |
| Per-import (native) | 0ms |
| Per-import (JS) | 0ms |
| Speedup ratio | 2.7x |

<!-- NOTES_START -->
**Note (3.5.0):** No build/rebuild metrics for this release (both engines null, files: 0) — only import resolution data was collected. The native engine crashed during `insertNodes` due to a napi-rs serialization bug where null `visibility` fields on parameter/child nodes fail conversion at the Rust boundary (`Failed to convert JavaScript value 'Null' into rust type 'String' on InsertNodesDefinition.visibility`). The WASM engine crashed during `buildStructure` with `database disk image is malformed` — a WAL deferred-close race condition in the benchmark's forked child processes where the first child's async `db.close()` corrupts the WAL for subsequent processes. Import resolution metrics are unaffected as they don't depend on a full build. Both issues will be fixed in the next release.
<!-- NOTES_END -->
| Speedup ratio | 3.0x |

<!-- INCREMENTAL_BENCHMARK_DATA
[
{
"version": "3.6.0",
"date": "2026-03-30",
"files": 514,
"wasm": {
"fullBuildMs": 6128,
"noopRebuildMs": 13,
"oneFileRebuildMs": 545,
"oneFilePhases": {
"setupMs": 2.5,
"parseMs": 246.6,
"insertMs": 17.7,
"resolveMs": 1.6,
"edgesMs": 19,
"structureMs": 28.8,
"rolesMs": 45.2,
"astMs": 12.9,
"complexityMs": 0.6,
"cfgMs": 0.3,
"dataflowMs": 0.3,
"finalizeMs": 4.3
}
},
"native": null,
"resolve": {
"imports": 904,
"nativeBatchMs": 3.9,
"jsFallbackMs": 11.7,
"perImportNativeMs": 0,
"perImportJsMs": 0
}
},
{
"version": "3.5.0",
"date": "2026-03-30",
Expand Down
Loading