diff --git a/docs/config.md b/docs/config.md index b29400ea4..98e5cfddb 100644 --- a/docs/config.md +++ b/docs/config.md @@ -71,7 +71,7 @@ Any field left as `None` falls back to the server-side default. Currently applie Uploading local files opens file descriptors — for the on-disk upload cache (one set of handles per `cacheShards`), the worker pool (`maxWorkers`), and the HTTP connections. On systems with a low `ulimit -n` (1024 is common), a large or highly concurrent upload can exhaust the limit and fail with `OSError: [Errno 24] Too many open files`. -Two ways to resolve it: +Ways to resolve it: - **Raise the OS limit** (per shell): `ulimit -n 8192`. - **Lower the SDK's footprint** — reduce the cache shards and/or the worker pool: @@ -83,7 +83,16 @@ Two ways to resolve it: `cacheShards` is immutable at runtime, so set it via the environment variable (or a `.env` file); `maxWorkers` can also be set in code (`rapidata_config.upload.maxWorkers = 10`). -The default `cacheShards` of 32 keeps a single upload well under a 1024 limit; lower it further only if you run many upload processes concurrently against the same limit. +- **Turn the disk cache off entirely** — the simplest fix if descriptors are still tight. `cacheToDisk=False` switches file uploads to an in-memory cache, which opens **no** cache file descriptors at all: + + ```python + from rapidata import rapidata_config + rapidata_config.upload.cacheToDisk = False # or RAPIDATA_cacheToDisk=false + ``` + + The tradeoff is only mild: the upload cache no longer persists across runs, so re-running the same upload re-uploads files it would otherwise have skipped (dedup within a single run still works). For a one-off large or highly concurrent submission this is usually the easiest way out. + +The default `cacheShards` of 32 keeps a single upload well under a 1024 limit; lower it further, or turn off `cacheToDisk`, only if you run many upload processes concurrently against the same limit. ## Environment Variables diff --git a/docs/error_handling.md b/docs/error_handling.md index 2b5df1027..1216fd929 100644 --- a/docs/error_handling.md +++ b/docs/error_handling.md @@ -179,4 +179,4 @@ Note that this only re-uploads the datapoints; it does not create the job defini If uploads fail with `OSError: [Errno 24] Too many open files`, the process has hit its file-descriptor limit — the upload cache, worker pool, and HTTP connections all consume descriptors. When the SDK detects this it appends a hint to the `FailedUploadException` message pointing at the relevant knobs. -Raise the OS limit (`ulimit -n 8192`) or lower the SDK's footprint with `RAPIDATA_cacheShards` / `RAPIDATA_maxWorkers`. See [Too many open files](config.md#too-many-open-files) in the configuration guide for details. +Raise the OS limit (`ulimit -n 8192`), lower the SDK's footprint with `RAPIDATA_cacheShards` / `RAPIDATA_maxWorkers`, or turn the disk cache off entirely with `cacheToDisk=False` (no cache descriptors at all, at the cost of cross-run upload dedup). See [Too many open files](config.md#too-many-open-files) in the configuration guide for details.