From d45dcd2345771ccbd1af06323d786b18a321be9e Mon Sep 17 00:00:00 2001 From: RapidPoseidon Date: Tue, 21 Jul 2026 12:07:04 +0000 Subject: [PATCH] docs(config): note cacheToDisk=false as fd-exhaustion escape hatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "Too many open files" guidance covered lowering cacheShards/maxWorkers and raising ulimit, but not the simplest lever: turning the disk cache off entirely. cacheToDisk=False switches file uploads to an in-memory cache that opens no cache file descriptors at all — the biggest single reduction for a tight limit — at the mild cost of losing cross-run upload dedup (within-run dedup still works). Document it as an option in config.md and mention it in the error_handling.md pointer. Co-Authored-By: Claude Opus 4.8 Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com> --- docs/config.md | 13 +++++++++++-- docs/error_handling.md | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/config.md b/docs/config.md index b29400ea4..98e5cfddb 100644 --- a/docs/config.md +++ b/docs/config.md @@ -71,7 +71,7 @@ Any field left as `None` falls back to the server-side default. Currently applie Uploading local files opens file descriptors — for the on-disk upload cache (one set of handles per `cacheShards`), the worker pool (`maxWorkers`), and the HTTP connections. On systems with a low `ulimit -n` (1024 is common), a large or highly concurrent upload can exhaust the limit and fail with `OSError: [Errno 24] Too many open files`. -Two ways to resolve it: +Ways to resolve it: - **Raise the OS limit** (per shell): `ulimit -n 8192`. - **Lower the SDK's footprint** — reduce the cache shards and/or the worker pool: @@ -83,7 +83,16 @@ Two ways to resolve it: `cacheShards` is immutable at runtime, so set it via the environment variable (or a `.env` file); `maxWorkers` can also be set in code (`rapidata_config.upload.maxWorkers = 10`). -The default `cacheShards` of 32 keeps a single upload well under a 1024 limit; lower it further only if you run many upload processes concurrently against the same limit. +- **Turn the disk cache off entirely** — the simplest fix if descriptors are still tight. `cacheToDisk=False` switches file uploads to an in-memory cache, which opens **no** cache file descriptors at all: + + ```python + from rapidata import rapidata_config + rapidata_config.upload.cacheToDisk = False # or RAPIDATA_cacheToDisk=false + ``` + + The tradeoff is only mild: the upload cache no longer persists across runs, so re-running the same upload re-uploads files it would otherwise have skipped (dedup within a single run still works). For a one-off large or highly concurrent submission this is usually the easiest way out. + +The default `cacheShards` of 32 keeps a single upload well under a 1024 limit; lower it further, or turn off `cacheToDisk`, only if you run many upload processes concurrently against the same limit. ## Environment Variables diff --git a/docs/error_handling.md b/docs/error_handling.md index 2b5df1027..1216fd929 100644 --- a/docs/error_handling.md +++ b/docs/error_handling.md @@ -179,4 +179,4 @@ Note that this only re-uploads the datapoints; it does not create the job defini If uploads fail with `OSError: [Errno 24] Too many open files`, the process has hit its file-descriptor limit — the upload cache, worker pool, and HTTP connections all consume descriptors. When the SDK detects this it appends a hint to the `FailedUploadException` message pointing at the relevant knobs. -Raise the OS limit (`ulimit -n 8192`) or lower the SDK's footprint with `RAPIDATA_cacheShards` / `RAPIDATA_maxWorkers`. See [Too many open files](config.md#too-many-open-files) in the configuration guide for details. +Raise the OS limit (`ulimit -n 8192`), lower the SDK's footprint with `RAPIDATA_cacheShards` / `RAPIDATA_maxWorkers`, or turn the disk cache off entirely with `cacheToDisk=False` (no cache descriptors at all, at the cost of cross-run upload dedup). See [Too many open files](config.md#too-many-open-files) in the configuration guide for details.