fix(upload): reduce disk-cache file-descriptor use to avoid EMFILE

[RapidPoseidon]

Problem

Submitting local-file jobs on a common ulimit -n 1024 can crash with OSError: [Errno 24] Too many open files. Reported from integration (SDK 3.16.5, Python 3.12, Linux, Compare task); the workaround was ulimit -n 8192 + sequential submission.

Root cause (measured)

The upload worker pool is already bounded (maxWorkers=25). The dominant descriptor consumer is the on-disk upload cache — diskcache.FanoutCache(shards=128):

cacheShards	FDs at init	FDs under 25-worker load
128 (old default)	384	639
32 (new default)	96	317
16	48	~260

A single default-config process already holds ~640 descriptors for the cache alone, before open files + HTTP connections — right at the edge of a 1024 limit.

Change

• cacheShards default 128 → 32. Shards only reduce SQLite write-lock contention, and cache writes are tiny key → filename strings written after the network upload. At 25 workers, 32 shards (≥ worker count) costs nothing measurable while cutting cache descriptors ~640 → ~317 — enough headroom for several concurrent processes under 1024. Benchmarked: fewer shards was not slower.
• Kept cacheShards decoupled from maxWorkers. Deriving shards from the worker count would change the key → shard mapping and silently invalidate the persistent cross-run upload cache every time workers changed (verified: 32→16 shards = ~57% cache miss). A stable static default reshuffles the on-disk cache exactly once (on upgrade).
• Made the failure discoverable. FailedUploadException now detects descriptor-exhaustion failures (EMFILE errno, with a message-text fallback) and appends a hint pointing at RAPIDATA_cacheShards / RAPIDATA_maxWorkers / ulimit -n and the config docs. New "Too many open files" sections in docs/config.md and docs/error_handling.md.
• No self-bumping of the OS limit (deliberate — a library shouldn't mutate process rlimits) and no new per-call max_upload_workers param (the global rapidata_config.upload.maxWorkers covers all upload paths uniformly). Both were discussed and decided with the requester.

Compatibility with #687

Lands cleanly on top of the sibling's merged #687 (atomic job creation). #687's only upload_config.py edit was additive (failureTolerance) — no conflict. All changes here sit below the JobDefinitionCreationMachine, inside its UPLOAD_DATAPOINTS step, so the concurrency bound already respects the atomic upload-then-persist ordering; no control flow was forked or reverted. #687 also gave the error hint a natural home — FailedUploadException is now the single chokepoint where mass-EMFILE failures surface.

One-time impact

Changing the shard count reshuffles the on-disk cache layout, so anyone with an existing warm cache re-uploads previously-cached assets once, then re-caches under the new layout. Not a correctness issue.

Testing

• New tests/rapidata_client/exceptions/test_failed_upload_exception.py (3 tests) covering the EMFILE hint (errno + message-fallback + negative case).
• pyright src/rapidata/rapidata_client — 0 errors (the CI gate).
• pytest tests/rapidata_client/job tests/rapidata_client/exceptions — 13 passed.
• mkdocs build — succeeds.

Note: skipped repo-wide black — the currently-pinned black>=25.11.0 reformats ~19 unrelated files vs. how the repo was last formatted, which would bury this change in churn. Added lines follow the surrounding style; CI enforces pyright only.

🔗 Session: https://node-b899e465.poseidon.rapidata.internal/