Reduce worker startup latency: fast env-activation entrypoint, lazy ML imports, slimmer bases

[arjunrajlab]

Why

NimbusImage runs each worker as a fresh docker run --rm container per job (images are already local — registry pull is not the cost). Two avoidable costs were paid on every invocation:

1. conda run -n worker … in the ENTRYPOINT spawns a second Python process (the conda CLI) just to activate the env and exec the target — ~0.8–1.0 s of pure overhead per job (~70% of a light worker's startup).
2. Heavy ML imports (torch/TF/cellpose/sam2/…) ran at module load, so even the lightweight interface request (the interactive, user-facing path) paid multi-second imports it never used.

Full audit, measurements, and methodology: todo/worker-startup-latency.md (TODO-002).

What changed

Fast entrypoint. New workers/base_docker_images/run_worker.sh activates the conda env in-process (PATH + activate.d hooks, preserving GDAL_DATA/PROJ_DATA/GDAL_DRIVER_PATH) and execs the env python — no second interpreter. Auto-detects miniforge (amd64) / miniconda (arm64).

• Baked into worker-base + image-processing-base as the ENTRYPOINT; the 19 shared-base workers inherit it. worker_client added to worker-base for parity.
• Rolled out to the 28 remaining production Dockerfiles (13 GPU workers + deconwolf + ai_analysis + blob_random_forest_classifier, incl. _M1).

Lazy imports. Heavy ML libs deferred into compute()/helpers across all GPU workers (+ sam_automatic_mask_generator); matplotlib deferred in the shared annotation_tools.py.

Refactors / cleanup. laplacian_of_gaussian, line_scan_worker, blob_colony_two_color_intensity_worker now build FROM worker-base (dropping inline base duplication, a stale Kitware/UPennContrast clone, and hardcoded x86_64 miniforge); obsolete _M1 files deleted. line_scan __main__ aligned to the canonical dispatch. Dropped unused r-base + added conda clean to both bases.

Results (arm64, full startup via --help, median of 5)

Worker	Before	After
blob_intensity	1.21 s	0.41 s
crop	1.22 s	0.44 s
blob_metrics	~1.2 s	0.40 s
registration	~1.2 s	0.51 s

Base images: worker-base 5.18 → 4.74 GB, image-processing-base 9.65 → 9.09 GB (disk only; does not affect startup).

Validation

• CPU workers built + tested: blob_metrics 9, registration 20, crop 14, blob_intensity 9, line_scan 7, blob_random_forest 4 — all pass. GDAL/PROJ env vars verified identical to the conda run reference.
• ⚠️ GPU workers are static-validated only (py_compile + import-usage analysis) — no local CUDA/torch. They must be built on the amd64 build host and smoke-tested (imports resolve at runtime, a compute run works) before deploy.

Deferred (not in this PR)

• Multi-stage prune of build-essential/git/dev headers (riskier image-size win).
• Deprecated / $BASE_IMAGE-ARG workers left on conda run (not in the production manifest).
• Trimming other always-on imports (tifffile/girder_client).

🤖 Generated with Claude Code

https://claude.ai/code/session_015xrXEVpvb4c4ScEVjN1VsM

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5cd5e96cd1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep run_worker inside the Piscis predict context

The Piscis compose file builds this Dockerfile with context: . from workers/annotations/piscis and dockerfile: ./predict/Dockerfile, while Docker COPY sources are resolved from the build context (Dockerfile docs; Compose build docs). Under that context ./workers/base_docker_images/run_worker.sh is absent/outside the build context, so docker compose build predict fails before the predict image is built; the rest of this Dockerfile still uses Piscis-local paths like ./environment.yml, confirming it is meant for the nested context.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Keep run_worker inside the Piscis train context

The Piscis compose file builds this Dockerfile with context: . from workers/annotations/piscis and dockerfile: ./train/Dockerfile, while Docker COPY sources are resolved from the build context (Dockerfile docs; Compose build docs). Under that context ./workers/base_docker_images/run_worker.sh is absent/outside the build context, so docker compose build train fails before the training image is built; the rest of this Dockerfile still uses Piscis-local paths like ./environment.yml, confirming it is meant for the nested context.

Useful? React with 👍 / 👎.