-
-
Notifications
You must be signed in to change notification settings - Fork 34.2k
fs: allow correct handling of burst in fs-events with AsyncIterator #58490
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #58490 +/- ##
==========================================
- Coverage 90.16% 90.15% -0.01%
==========================================
Files 636 636
Lines 187891 187923 +32
Branches 36884 36882 -2
==========================================
+ Hits 169408 169429 +21
- Misses 11246 11247 +1
- Partials 7237 7247 +10
🚀 New features to boost your workflow:
|
63d31e7 to
3512480
Compare
16c2f33 to
d5d6d5d
Compare
|
This almost LGTM % the suggestions. Though I'd appreciate if others from @nodejs/fs or maybe @jasnell @benjamingr can take a look and see if the new options look good. |
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Other than the option values, this LGTM
joyeecheung
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % missing updates to the docs
|
Thanks for all the help, guidance & input @joyeecheung & @Ethan-Arrowood ❤️ |
|
Landed in 5f7dbf4 |
This addresses a bug in
fs.watchwhen used as an AsyncIterator.The issue is that when consuming the AsyncIteractor returned by
fs.watchit yields a value. When using that value and in turn awaiting an asynchrounous action any events happening in the meantime will go missing. The reason is that between exiting the watch function by yielding and reentering it through the next round, thepromiseinside watch is already resolved. So any events generated will be duplicate resolutions of that promise and therefore ignored.More reaslistically than this minimal example is when the found files are actually read via
await fs.readFile. Then there will be a lag. If the file events happening are happening close together, then the second one will be missed.To fix this issue I added a queue to the watch function that new file events get pushed onto. The promise is no longer resolved with a value, but is simply the gate gating whether or not there are any events in the queue. The iterator awaits the promise and then yields the items from the queue so long as there are any. When the queue is empty and the watch is still running, then a new promise is created and awaited upon. This whould eliminate the problem entirely and one can now go asynchronous in the loop as long as one wants without missing events.
Verified that the added test fails with v24.0.0 and passes after the fix.
Based on feedback the queuing was made configurable with the
maxQueue(default2048) option determining the maximum size of the queue and theoverflowoption deciding to eitherignorethe issue orthrowan Error (default:'ignore'). To effectively get back the previous behavior one would have to pass{ maxQueue: 1, overflow: 'ignore' }.