docs(config): note cacheToDisk=false as fd-exhaustion escape hatch#700
Merged
RapidPoseidon merged 1 commit intoJul 21, 2026
Merged
Conversation
The "Too many open files" guidance covered lowering cacheShards/maxWorkers and raising ulimit, but not the simplest lever: turning the disk cache off entirely. cacheToDisk=False switches file uploads to an in-memory cache that opens no cache file descriptors at all — the biggest single reduction for a tight limit — at the mild cost of losing cross-run upload dedup (within-run dedup still works). Document it as an option in config.md and mention it in the error_handling.md pointer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
LinoGiger
approved these changes
Jul 21, 2026
LinoGiger
marked this pull request as ready for review
July 21, 2026 12:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #696. Adds the simplest escape hatch to the "Too many open files" guidance: turning the disk cache off entirely.
cacheToDisk=Falseswitches file uploads to an in-memory cache that opens no cache file descriptors at all — the single biggest FD reduction available, useful when descriptors are still tight after loweringcacheShards/maxWorkers.Tradeoff (mild): the upload cache no longer persists across runs, so re-running the same upload re-uploads files it would otherwise skip. Dedup within a single run still works.
Changes (docs only)
docs/config.md— new bullet under "Too many open files" showingrapidata_config.upload.cacheToDisk = False/RAPIDATA_cacheToDisk=falseand the tradeoff.docs/error_handling.md— parity mention in the pointer to the config section.No code change.
mkdocs buildsucceeds.🔗 Session: https://node-b899e465.poseidon.rapidata.internal/