docs: update query benchmarks (3.6.0)#713
Conversation
Greptile SummaryThis PR adds version 3.6.0 query benchmark data to the benchmark document, providing both native (Rust) and WASM rows in the summary table and a fully updated "Latest results" section. It directly addresses the prior review finding about missing delta indicators on the 3.6.0 native row. Key changes:
Confidence Score: 5/5Safe to merge — documentation-only change with no logic, no code execution, and all data verified as internally consistent. The previously flagged P1 concern (missing native delta indicators) is fully resolved in commit 1f9aa73. All delta percentages are mathematically correct, the JSON data block matches the markdown tables exactly, the note accurately explains the non-standard 3.4.1 baseline and the mid-target change driving the ↑41% WASM diffImpact. No P0 or P1 findings remain. No files require special attention. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["3.6.0 native row\n(new)"] -- "baseline: 3.4.1 native\n(3.5.0 was wasm-only)" --> B["delta indicators\n↑6-10%"]
C["3.6.0 wasm row\n(new)"] -- "baseline: 3.5.0 wasm" --> D["delta indicators\n↑7-41%"]
D -- "↑41% diffImpact\nexplained by mid-target\nchange: db → node" --> E["Note (3.6.0)\nadded in NOTES section"]
B --> E
E --> F["JSON data block\n(source of truth)"]
F -- "values match\nmarkdown table" --> A
F -- "values match\nmarkdown table" --> C
Reviews (2): Last reviewed commit: "docs(benchmarks): add native delta indic..." | Re-trigger Greptile |
|
|
||
| | Version | Engine | fnDeps d1 | fnDeps d3 | fnDeps d5 | fnImpact d1 | fnImpact d3 | fnImpact d5 | diffImpact | | ||
| |---------|--------|----------:|----------:|----------:|------------:|------------:|------------:|-----------:| | ||
| | 3.6.0 | native | 9.4 | 9.6 | 9.4 | 3.4 | 3.4 | 3.3 | 8.3ms | |
There was a problem hiding this comment.
Missing delta indicators for 3.6.0 native row
Every other row in the summary table includes delta indicators (↑%, ↓%, or ~) relative to the previous entry of the same engine. The 3.6.0 native row shows only raw values, with no comparison against 3.4.1 native (the last version that had native data). This makes the row visually inconsistent and harder to interpret at a glance.
Based on the 3.4.1 native baseline, expected deltas would be approximately:
- fnDeps d1: 9.4 vs 8.9 → ↑6%
- fnDeps d3: 9.6 vs 8.9 → ↑8%
- fnDeps d5: 9.4 vs 8.8 → ↑7%
- fnImpact d1: 3.4 vs 3.2 → ↑6%
- fnImpact d3: 3.4 vs 3.1 → ↑10%
- fnImpact d5: 3.3 vs 3.1 → ↑6%
- diffImpact: 8.3ms vs 8.1ms → ~
If the generator intentionally skips deltas when the immediately preceding version lacks native data (3.5.0 was wasm-only), that behavior is worth documenting. If it's a generation bug, it should be fixed in the workflow before subsequent releases exhibit the same gap.
There was a problem hiding this comment.
Fixed in 1f9aa73. Added delta indicators to the 3.6.0 native row, computed against 3.4.1 native (the last version with native data):
- fnDeps d1: 9.4 vs 8.9 → ↑6%
- fnDeps d3: 9.6 vs 8.9 → ↑8%
- fnDeps d5: 9.4 vs 8.8 → ↑7%
- fnImpact d1: 3.4 vs 3.2 → ↑6%
- fnImpact d3: 3.4 vs 3.1 → ↑10%
- fnImpact d5: 3.3 vs 3.1 → ↑6%
- diffImpact: 8.3ms vs 8.1ms → ~
Also added a note explaining that native deltas are relative to 3.4.1 since 3.5.0 was wasm-only.
Add missing delta indicators to the 3.6.0 native row (vs 3.4.1, the last version with native data) and add an explanatory note for the mid-target change (db → node) that accounts for the 41% WASM diffImpact increase.
|
Both Greptile findings addressed in 1f9aa73:
|
|
@greptileai review |
Automated query benchmark update for 3.6.0 from workflow run #538.