-
-
Notifications
You must be signed in to change notification settings - Fork 34.8k
Move ESM loaders off-thread #44710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
aduh95
merged 126 commits into
nodejs:main
from
JakobJingleheimer:feat/esm_off-thread-loaders
Apr 13, 2023
Merged
Move ESM loaders off-thread #44710
Changes from 1 commit
Commits
Show all changes
126 commits
Select commit
Hold shift + click to select a range
03f6bb4
esm: move hook execution to separate thread
JakobJingleheimer a0bb205
esm: resolve optionally returns import assertions
GeoffreyBooth cd16ccd
port updates to `worker_thread`
JakobJingleheimer 01f1191
move `HooksProxy` to `hooks.js`
JakobJingleheimer 5f02297
`ESMLoader::addCustomLoaders` → `ESMLoader::registerCustomLoaders`
JakobJingleheimer 820aa55
fix startup issues in hooks worker
JakobJingleheimer 231b964
WIP: move importing custom loaders back to main thread
JakobJingleheimer ed412d5
🎉 `/test-esm-example-loader.mjs` is passing!!
JakobJingleheimer 4776a6d
Fix tests that are loading modules/esm/worker.js on the main thread
GeoffreyBooth 2aed49a
WIP: convert resolve to sync
JakobJingleheimer 78a2889
Fix test
GeoffreyBooth 5746d17
Restore parentURL validation to defaultResolve (and therefore also to…
GeoffreyBooth c1afb3d
Improve deserialization error (sometimes)
GeoffreyBooth 666ee10
Rename #hooks to #chains
GeoffreyBooth 8198a13
Subclass customized module loader
GeoffreyBooth 2d39e13
Fix import.meta being undefined (WIP)
GeoffreyBooth dd6a5d3
Remove undocumented globalPreload setImportMetaCallback (WIP)
GeoffreyBooth 1415d6e
fs.write(1, message) works at least in Mac and Windows for printing f…
GeoffreyBooth a374bd1
Standardize names, fix lint
GeoffreyBooth 98b5512
Exceptions thrown inside loader hooks should be returned to the main …
GeoffreyBooth 7be92f3
Rewrite test-esm-loader-hooks test to spawn child process and inspect…
GeoffreyBooth 382753a
Send loader initialization errors up to the main thread so they can b…
GeoffreyBooth 2b3e6a5
Remove debugging code (fixes 3 tests!)
GeoffreyBooth c73001f
Catch another worker thread exception so that the process doesn't hang
GeoffreyBooth b4cccd4
Add error handling around hooks method responses being undefined or h…
GeoffreyBooth b272d31
separate worker `data` & `lock` buffers
JakobJingleheimer 0ec518a
chunks working?? (code is very rough!)
JakobJingleheimer 11c1471
Fix import.meta.resolve
GeoffreyBooth 0dd5114
More carefully pass errors back to the main thread
GeoffreyBooth d15b425
Some errors span multiple chunks, apparently
GeoffreyBooth b796c02
Fix chunking by separating metadata into separate buffers; fix error …
GeoffreyBooth 599a616
unlock worker thread
targos 42d485e
use syncCommPort to exchange data too
targos 8e3e0c6
fix --import
targos 76e23d8
fix doclint
targos 75fe62b
remove internal check
targos 350351f
refactor worker exception handling
targos f3b2ed9
errors during load should just be rethrown
targos 7e79b2f
run global preload scripts in the main thread
targos c0142d6
nits
aduh95 5d7e3b4
nits
aduh95 685590c
fix linter errors
aduh95 49a4088
Add `HookProxy.prototype.makeAsyncRequest`
aduh95 d40834d
Wrap messages passed from worker to main thread so that we can carry …
GeoffreyBooth 47bd0bf
Emit experimental warning only once
GeoffreyBooth 17ff524
Don't re-throw uncaught exceptions and then re-catch them ad infinitum
GeoffreyBooth 2065222
fix linter errors
aduh95 b0cc21f
transfer `ArrayBuffer`s
aduh95 29d019f
improve exception serialization
aduh95 8706cbd
chaining loader hooks
aduh95 04ebc5c
make things more consistent
aduh95 8aae17e
remove unnecessary parenthesis
aduh95 b74dcaa
simplify error cloning
aduh95 3312b1c
small optimisation: read error properties only once when cloning
aduh95 7e8be7c
skip already cloned properties
aduh95 7b7e5be
DRY
aduh95 3f4c9bf
fix `test-esm-loader-chaining`
aduh95 140c03b
update docs
aduh95 020b9d7
fix `waitAsync` usage
aduh95 6899203
simplify `ArrayBuffer` transfer
aduh95 9d392ef
do not die waiting on worker to respond,
aduh95 1c7dd71
`import()` should not block the main thread, even for the resolve step
aduh95 acfa8d9
rework docs
aduh95 1821301
do not keep the worker alive when awaiting hooks to resolve
aduh95 b1dcccc
keep the worker thread alive longer
aduh95 0cfc683
let uncatcught exceptinnon be uncaught exceptions
aduh95 7fd6fa1
use `serializeError` instead of custom logic
aduh95 a3982ee
fix unserializable errors handling
aduh95 47b4d83
only count async messages
aduh95 d518e53
remove global error handler only at the end of the sync messages
aduh95 531c9a3
use `'beforeExit'` event to detect when threads are ready to die
aduh95 3bcf18c
ensure there are no pending messages before letting worker die
aduh95 b071a89
linter fixes
aduh95 0911d15
get rid of the tick counter timeout
aduh95 0b1c22b
Use `Atomics.load` for better results
aduh95 88686f5
add `shared_constants` internal module to make code clearer
aduh95 1b1583e
rename `ModuleLoader` -> `createModuleLoader`
aduh95 3f4934a
lazy load esmLoader
aduh95 800d151
nits
aduh95 a5a798b
fix test-esm-loader
aduh95 f9371ae
unref worker more often
aduh95 0b79809
add more comments
aduh95 0cfec42
fix main thread refusing to die waiting on a response that will never…
aduh95 281e943
remove unrelated/unnecessary changes in `test/`
aduh95 3e31a1c
remove unrelated change in `src/`
aduh95 595a95f
restore changes related to sync resolve
aduh95 10bf7b8
Revert "unref worker more often"
aduh95 106b400
nits
aduh95 cc9373a
use `console.log` instead `fs.writeSync`
aduh95 a67af11
fix lint
aduh95 aab57d7
add comment to describe the shared memory organisation
aduh95 638b497
Remove unnecessary `Atomics.store` calls
aduh95 e340d24
s/BORED/IDLE/
aduh95 018ab18
add `SHARED_MEMORY_BYTE_LENGTH` to shared_constants
aduh95 80cd74c
add definition for idleness
aduh95 817d0e8
Update doc/api/esm.md
aduh95 0590df4
Update lib/internal/modules/esm/shared_constants.js
aduh95 cee03fa
add/fix comments + nits
aduh95 258cf8a
for..of -> classic for
aduh95 8863cb9
s/NUMBER_OF_INCOMING_MESSAGES/NUMBER_OF_MESSAGES_IN_TRANSIT/
aduh95 13f3d8d
add explanation for how `process.cwd()` can fail
JakobJingleheimer 4731362
Apply suggestions from code review
aduh95 12717f7
fix worker permissions for internal workers
aduh95 61ad2a5
fix dead lock
aduh95 fa29a70
Apply suggestions from code review
aduh95 159ca6f
Apply suggestions from code review
aduh95 b4a8a2f
`git mv lib/internal/modules/esm/worker.js lib/internal/main/loader_t…
aduh95 6177afa
add comment
aduh95 a85288d
revert `git mv lib/internal/modules/esm/worker.js lib/internal/main/l…
aduh95 be8090e
never-settling `import.meta.resolve`
aduh95 f6679bc
reuse the public port instead of creating an additional communication…
aduh95 722fd0e
fix: disable test's case concurrency to avoid deadlock ~1/10 times
JakobJingleheimer 544bf89
fix rare deadlock
JakobJingleheimer 88f9042
update Hooks entry in to note use of threads and caveat for async ops
JakobJingleheimer 6d8bfc6
chore: de-lint md
JakobJingleheimer 31e7d5d
switch test fixtures from `console.log` → `writeSync(1, …)`
JakobJingleheimer 372065e
fix loaders with node:fs
targos 244f916
use more writeSync
targos bc8e970
remove remaining `await` in front of `import.meta.resolve`
aduh95 f44a246
resolve TODOs for edge cases
aduh95 221f2e2
fix race condition that could cause a dead lock when dealing with nev…
aduh95 9436c97
reduce the size of the sab and introduce a `never-setlle` message status
aduh95 ad73db6
fix race condition?
aduh95 95052c6
tidy code, comments, and docs
JakobJingleheimer acc0900
Update lib/internal/worker.js
aduh95 f02012f
Update lib/internal/worker.js
aduh95 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
chaining loader hooks
- Loading branch information
commit 8706cbd6ada10af7fd77d97529a8ae87fcc3b75c
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
test/fixtures/es-module-loaders/loader-load-receiving-modified-context.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
test/fixtures/es-module-loaders/loader-resolve-receiving-modified-context.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.