Skip to content

benchmark: parallelize cases for CI performance#19780

Draft
hai-x wants to merge 24 commits intomainfrom
worker-benchmark
Draft

benchmark: parallelize cases for CI performance#19780
hai-x wants to merge 24 commits intomainfrom
worker-benchmark

Conversation

@hai-x
Copy link
Copy Markdown
Member

@hai-x hai-x commented Aug 10, 2025

Summary

What kind of change does this PR introduce?

The full benchmark task takes ~50 mins to complete. Now the cases are run in parallel to reduce the CI time.

Did you add tests for your changes?

No

Does this PR introduce a breaking change?

No

If relevant, what needs to be documented once your changes are merged or what have you already documented?

No

Use of AI

No

@hai-x hai-x force-pushed the worker-benchmark branch 5 times, most recently from 8e735de to 430655e Compare August 10, 2025 15:55
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Aug 10, 2025

Merging this PR will degrade performance by 98.11%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 25 improved benchmarks
❌ 16 regressed benchmarks
✅ 79 untouched benchmarks
⏩ 24 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation benchmark "three-long", scenario '{"name":"mode-production","mode":"production"}' 3.1 s 2.5 s +21.4%
Simulation benchmark "asset-modules-inline", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 50.2 ms 67.7 ms -25.83%
Simulation benchmark "asset-modules-bytes", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 34.6 ms 60.2 ms -42.54%
Simulation benchmark "lodash", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 32.9 ms 27.1 ms +21.48%
Simulation benchmark "many-chunks-commonjs", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 47.6 ms 61 ms -21.98%
Memory benchmark "concatenate-modules", scenario '{"name":"mode-development","mode":"development"}' 706.1 KB 1,130.8 KB -37.56%
Memory benchmark "wasm-modules-async", scenario '{"name":"mode-production","mode":"production"}' 71.1 MB 4 MB ×18
Memory benchmark "wasm-modules-sync", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 864.4 KB 306.6 KB ×2.8
Memory benchmark "asset-modules-source", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 390.1 KB 587.2 KB -33.56%
Memory benchmark "many-chunks-esm", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 709.7 KB 976.7 KB -27.33%
Memory benchmark "asset-modules-bytes", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 944.4 KB 237.8 KB ×4
Memory benchmark "lodash", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 695.6 KB 353 KB +97.04%
Memory benchmark "asset-modules-bytes", scenario '{"name":"mode-development","mode":"development"}' 999 KB 681.6 KB +46.58%
Memory benchmark "context-esm", scenario '{"name":"mode-production","mode":"production"}' 9.4 MB 7.6 MB +24.05%
Memory benchmark "wasm-modules-async", scenario '{"name":"mode-development","mode":"development"}' 30.3 MB 71.1 MB -57.37%
Memory benchmark "devtool-source-map", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 735.6 KB 985.2 KB -25.34%
Memory benchmark "context-esm", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 1,157.4 KB 633.5 KB +82.68%
Memory benchmark "json-modules", scenario '{"name":"mode-development","mode":"development"}' 534.1 KB 1,471 KB -63.69%
Memory benchmark "cache-filesystem", scenario '{"name":"mode-production","mode":"production"}' 9.2 MB 5.2 MB +75.12%
Memory benchmark "devtool-eval", scenario '{"name":"mode-development","mode":"development"}' 2.2 MB 1.3 MB +66.56%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing worker-benchmark (6ee9e23) with main (9211be0)

Open in CodSpeed

Footnotes

  1. 24 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@hai-x hai-x force-pushed the worker-benchmark branch 4 times, most recently from f4a9617 to 1235c28 Compare August 10, 2025 17:55
@hai-x hai-x force-pushed the worker-benchmark branch 7 times, most recently from ec6261e to 2776482 Compare August 28, 2025 15:31
@hai-x hai-x changed the title refactor: benchmark refactor: parallel benchmarks Oct 11, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Nov 11, 2025

This PR is packaged and the instant preview is available (6ee9e23).

Install it locally:

  • npm
npm i -D webpack@https://pkg.pr.new/webpack@6ee9e23
  • yarn
yarn add -D webpack@https://pkg.pr.new/webpack@6ee9e23
  • pnpm
pnpm add -D webpack@https://pkg.pr.new/webpack@6ee9e23

@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Dec 27, 2025

⚠️ No Changeset found

Latest commit: 6ee9e23

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@hai-x hai-x force-pushed the worker-benchmark branch 3 times, most recently from 124b1f8 to e2def54 Compare March 21, 2026 13:19
@hai-x hai-x force-pushed the worker-benchmark branch from e2def54 to e9f0da7 Compare March 21, 2026 13:20
@hai-x hai-x changed the title refactor: parallel benchmarks benchmark: parallelize cases to reduce CI time Mar 21, 2026
@hai-x hai-x force-pushed the worker-benchmark branch from 3b49d9a to 654bb6b Compare March 21, 2026 16:59
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.40%. Comparing base (47bb540) to head (6ee9e23).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #19780      +/-   ##
==========================================
- Coverage   91.44%   91.40%   -0.04%     
==========================================
  Files         559      561       +2     
  Lines       55150    55359     +209     
  Branches    14556    14610      +54     
==========================================
+ Hits        50430    50603     +173     
- Misses       4720     4756      +36     
Flag Coverage Δ
integration 90.39% <ø> (-0.04%) ⬇️
test262 41.96% <ø> (+0.01%) ⬆️
unit 36.23% <ø> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hai-x hai-x force-pushed the worker-benchmark branch from 28902d5 to 968fcdf Compare April 9, 2026 16:59
@hai-x hai-x changed the title benchmark: parallelize cases to reduce CI time benchmark: parallelize cases for CI performance Apr 10, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Types Coverage

Coverage after merging worker-benchmark into main will be
98.93%
Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
bin
   webpack.js98.77%100%100%98.77%91
examples
   build-common.js100%100%100%100%
   buildAll.js100%100%100%100%
   examples.js100%100%100%100%
   template-common.js98.21%100%100%98.21%72
examples/custom-javascript-parser
   test.filter.js100%100%100%100%
examples/custom-javascript-parser/internals
   acorn-parse.js100%100%100%100%
   meriyah-parse.js100%100%100%100%
   oxc-parse.js91.30%100%100%91.30%140, 142–143, 145, 147, 153–154, 161, 168, 90
examples/markdown
   webpack.config.mjs100%100%100%100%
examples/typescript
   test.filter.js50%100%100%50%5
examples/virtual-modules
   test.filter.js100%100%100%100%
examples/wasm-bindgen-esm
   test.filter.js100%100%100%100%
examples/wasm-complex
   test.filter.js100%100%100%100%
examples/wasm-simple
   test.filter.js100%100%100%100%
examples/wasm-simple-source-phase
   test.filter.js100%100%100%100%
lib
   APIPlugin.js100%100%100%100%
   AbstractMethodError.js100%100%100%100%
   AsyncDependenciesBlock.js100%100%100%100%
   AsyncDependencyToInitialChunkError.js100%100%100%100%
   AutomaticPrefetchPlugin.js100%100%100%100%
   BannerPlugin.js100%100%100%100%
   Cache.js98.21%100%100%98.21%104
   CacheFacade.js100%100%100%100%
   CaseSensitiveModulesWarning.js100%100%100%100%
   Chunk.js99.72%100%100%99.72%37
   ChunkGraph.js100%100%100%100%
   ChunkGroup.js100%100%100%100%
   ChunkRenderError.js100%100%100%100%
   ChunkTemplate.js100%100%100%100%
   CleanPlugin.js98.72%100%100%98.72%206, 226, 382
   CodeGenerationError.js100%100%100%100%
   CodeGenerationResults.js100%100%100%100%
   CommentCompilationWarning.js100%100%100%100%
   CompatibilityPlugin.js100%100%100%100%
   Compilation.js98.55%100%100%98.55%1559, 1855, 1862, 1870, 1892, 2788, 3213, 3868, 3897, 3950–3951, 3955, 3960, 3976–3977, 3991–3992, 3997–3998, 4471, 4497, 498, 503, 5205, 5286, 5301, 5326–5327, 5329, 5651, 5656, 5662, 5665, 5677, 5679, 5683, 5699, 5714, 5746, 5800, 5824, 5938, 717–718
   Compiler.js99.56%100%100%99.56%1116–1117, 1125
   ConcatenationScope.js98.59%100%100%98.59%189
   ConcurrentCompilationError.js100%100%100%100%
   ConditionalInitFragment.js100%100%100%100%
   ConstPlugin.js100%100%100%100%
   ContextExclusionPlugin.js100%100%100%100%
   ContextModule.js100%100%100%100%
   ContextModuleFactory.js97.75%100%100%97.75%258, 393, 418, 443, 447, 458
   ContextReplacementPlugin.js100%100%100%100%
   CssModule.js81.32%100%100%81.32%157, 162–177
   DefinePlugin.js98.92%100%100%98.92%158–159, 175, 194, 268
   DelegatedModule.js95.24%100%100%95.24%253–257
   DelegatedModuleFactoryPlugin.js98.15%100%100%98.15%106
   DelegatedPlugin.js100%100%100%100%
   DependenciesBlock.js100%100%100%100%
   Dependency.js98.13%100%100%98.13%371, 404
   DependencyTemplate.js100%100%100%100%
   DependencyTemplates.js100%100%100%100%
   DllEntryPlugin.js100%100%100%100%
   DllModule.js100%100%100%100%
   DllModuleFactory.js100%100%100%100%
   DllPlugin.js100%100%100%100%
   DllReferencePlugin.js100%100%100%100%
   DotenvPlugin.js97.88%100%100%97.88%237, 378, 391–392
   DynamicEntryPlugin.js100%100%100%100%
   EntryOptionPlugin.js100%100%100%100%
   EntryPlugin.js100%100%100%100%
   Entrypoint.js100%100%100%100%
   EnvironmentNotSupportAsyncWarning.js100%100%100%100%
   EnvironmentPlugin.js97.14%100%100%97.14%49
   ErrorHelpers.js100%100%100%100%
   EvalDevToolModulePlugin.js100%100%100%100%
   EvalSourceMapDevToolPlugin.js100%100%100%100%
   ExportsInfo.js100%100%100%100%
   ExportsInfoApiPlugin.js100%100%100%100%
   ExternalModule.js98.89%100%100%98.89%399–403, 542
   ExternalModuleFactoryPlugin.js100%100%100%100%
   ExternalsPlugin.js100%100%100%100%
   FalseIIFEUmdWarning.js100%100%100%100%
   FileSystemInfo.js99.49%100%100%99.49%182, 2213–2214, 2217, 2228, 2239, 2250, 278, 3596, 3611, 3635
   FlagAllModulesAsUsedPlugin.js100%100%100%100%
   FlagDependencyExportsPlugin.js98.74%100%100%98.74%399, 401, 405
   FlagDependencyUsagePlugin.js100%100%100%100%
   FlagEntryExportAsUsedPlugin.js100%100%100%100%
   Generator.js100%100%100%100%
   GraphHelpers.js100%100%100%100%
   HarmonyLinkingError.js100%100%100%100%
   HookWebpackError.js100%100%100%100%
   HotModuleReplacementPlugin.js100%100%100%100%
   HotUpdateChunk.js100%100%100%100%
   IgnoreErrorModuleFactory.js100%100%100%100%
   IgnorePlugin.js100%100%100%100%
   IgnoreWarningsPlugin.js100%100%100%100%
   InitFragment.js100%100%100%100%
   InvalidDependenciesModuleWarning.js100%100%100%100%
   JavascriptMetaInfoPlugin.js100%100%100%100%
   LibManifestPlugin.js97.14%100%100%97.14%117, 120
   LibraryTemplatePlugin.js100%100%100%100%
   LoaderOptionsPlugin.js100%100%100%100%
   LoaderTargetPlugin.js100%100%100%100%
   MainTemplate.js100%100%100%100%
   ManifestPlugin.js100%100%100%100%
   Module.js98.50%100%100%98.50%1303, 1308, 1369, 1383, 1445, 1454
   ModuleBuildError.js100%100%100%100%
   ModuleDependencyError.js100%100%100%100%
   ModuleDependencyWarning.js100%100%100%100%
   ModuleError.js100%100%100%100%
   ModuleFactory.js100%100%100%100%
   ModuleFilenameHelpers.js98.85%100%100%98.85%106, 108
   ModuleGraph.js99.73%100%100%99.73%1004
   ModuleGraphConnection.js100%100%100%100%
   ModuleHashingError.js100%100%100%100%
   ModuleInfoHeaderPlugin.js100%100%100%100%
   ModuleNotFoundError.js100%100%100%100%
   ModuleParseError.js100%100%100%100%
   ModuleProfile.js100%100%100%100%
   ModuleRestoreError.js100%100%100%100%
   ModuleSourceTypeConstants.js100%100%100%100%
   ModuleStoreError.js100%100%100%100%
   ModuleTemplate.js100%100%100%100%
   ModuleTypeConstants.js100%100%100%100%
   ModuleWarning.js100%100%100%100%
   MultiCompiler.js99.69%100%100%99.69%645
   MultiStats.js100%100%100%100%
   MultiWatching.js100%100%100%100%
   NoEmitOnErrorsPlugin.js100%100%100%100%
   NoModeWarning.js100%100%100%100%
   NodeStuffInWebError.js100%100%100%100%
   NodeStuffPlugin.js100%100%100%100%
   NormalModule.js97.78%100%100%97.78%1002, 1036, 1052, 1139, 1781, 1786–1796, 214, 726, 744, 761
   NormalModuleFactory.js99.47%100%100%99.47%1067, 1376, 466, 478
   NormalModuleReplacementPlugin.js100%100%100%100%
   NullFactory.js100%100%100%100%
   OptimizationStages.js100%100%100%100%
   OptionsApply.js100%100%100%100%
   Parser.js100%100%100%100%
   PlatformPlugin.js100%100%100%100%
   PrefetchPlugin.js100%100%100%100%
   ProgressPlugin.js98.75%100%100%98.75%448–449, 454, 456, 520
   ProvidePlugin.js100%100%100%100%
   RawModule.js100%100%100%100%
   RecordIdsPlugin.js100%100%100%100%
   RequestShortener.js100%100%100%100%
   RequireJsStuffPlugin.js100%100%100%100%
   ResolverFactory.js100%100%100%100%
   RuntimeGlobals.js100%100%100%100%
   RuntimeModule.js100%100%100%100%
   RuntimePlugin.js100%100%100%100%
   RuntimeTemplate.js100%100%100%100%
   SelfModuleFactory.js100%100%100%100%
   SingleEntryPlugin.js100%100%100%100%
   SizeFormatHelpers.js100%100%100%100%
   SourceMapDevToolModuleOptionsPlugin.js100%100%100%100%
   SourceMapDevToolPlugin.js99.16%100%100%99.16%267–268, 610
   Stats.js100%100%100%100%
   Template.js100%100%100%100%
   TemplatedPathPlugin.js98.86%100%100%98.86%134–135
   UnhandledSchemeError.js100%100%100%100%
   UnsupportedFeatureWarning.js100%100%100%100%
   UseStrictPlugin.js100%100%100%100%
   WarnCaseSensitiveModulesPlugin.js100%100%100%100%
   WarnDeprecatedOptionPlugin.js100%100%100%100%
   WarnNoModeSetPlugin.js100%100%100%100%
   WatchIgnorePlugin.js100%100%100%100%
   Watching.js100%100%100%100%
   WebpackError.js96.97%100%100%96.97%44
   WebpackIsIncludedPlugin.js100%100%100%100%
   WebpackOptionsApply.js100%100%100%100%
   WebpackOptionsDefaulter.js100%100%100%100%
   buildChunkGraph.js99.87%100%100%99.87%326
   cli.js98.71%100%100%98.71%117, 469, 501, 543, 813
   formatLocation.js100%100%100%100%
   index.js100%100%100%100%
   validateSchema.js94.67%100%100%94.67%100, 87, 89, 98
   webpack.js97.22%100%100%97.22%196, 218, 220
lib/asset
   AssetBytesGenerator.js100%100%100%100%
   AssetBytesParser.js100%100%100%100%
   AssetGenerator.js100%100%100%100%
   AssetModulesPlugin.js97.77%100%100%97.77%285, 309, 312, 364, 40
   AssetParser.js100%100%100%100%
   AssetSourceGenerator.js100%100%100%100%
   AssetSourceParser.js100%100%100%100%
   RawDataUrlModule.js100%100%100%100%
lib/async-modules
   AsyncModuleHelpers.js100%100%100%100%
   AwaitDependenciesInitFragment.js100%100%100%100%
   InferAsyncModulesPlugin.js100%100%100%100%
lib/cache
   AddBuildDependenciesPlugin.js100%100%100%100%
   AddManagedPathsPlugin.js100%100%100%100%
   IdleFileCachePlugin.js97.92%100%100%97.92%71, 83, 91
   MemoryCachePlugin.js95.83%100%100%95.83%33
   MemoryWithGcCachePlugin.js93.15%100%100%93.15%106, 113–114, 122, 89
   PackFileCacheStrategy.js<

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant