CI: stop rebuilding rserve on every PR run; pin R deps to a CRAN snapshot - #849
Merged
Conversation
install_and_verify pins the packages it names, but their dependencies resolved from live CRAN. A mid-day CRAN publish on 2026-07-16 (ggrepel requiring ggplot2 >= 3.5.2/gtable >= 0.3.6) broke rserve image rebuilds that had passed hours earlier, plus transient cloud.r-project.org download failures. Resolve everything from the 2026-07-15 snapshot: same inputs every build until the date is bumped deliberately. The __linux__/jammy path serves prebuilt binaries for the base image (Ubuntu 22.04 + R 4.4) - install_packages.R drops from ~480s to ~100s - with the plain source snapshot as fallback. Verified locally: docker build of docker/R completes, 13/13 packages install and load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every docker job rebuilt the rserve image from scratch: BuildKit (Docker 28 default) does not use pulled images as layer cache, so the docker-compose pull that used to make the build a cache no-op under the classic builder stopped helping. That cost ~10 min per run and exposed every PR to CRAN availability (see 2026-07-16 failure on #845's run). - PR runs build only web unless the PR changes docker/R (two-commit tree diff against the fetched base tip - works on the shallow merge-ref clone; any fetch/diff failure falls back to building everything). The pulled nrel/openstudio-rserve:latest, kept fresh by docker-upload on develop/master pushes, backs the stack instead - push runs still build everything (docker-upload deploys those images) - compose build config gains cache_from + BUILDKIT_INLINE_CACHE so images pushed from develop carry layer-cache metadata and can seed the builds that do still happen Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
brianlball
added a commit
that referenced
this pull request
Jul 29, 2026
* Update permissions for rspec_command and enhance README with MVP suitability details * Enhance external batch execution support: - Update developer guide to reflect UrbanOpt and custom gemfile support. - Modify README to clarify MVP suitability and resolved limitations. - Improve run_chunk.rb to support Windows scripts and enhance logging. - Update LHS sampling to generate histogram images using pure Ruby. - Refactor external_batch.rb to handle nil ENV variables gracefully. - Adjust ingester and packager to support UrbanOpt and gemfile analyses. - Revise tests to validate packaging for UrbanOpt and gemfile analyses. * Add Nomad executor implementation for external batch feature * Add Nomad job submission script for external batch executor * Add Nomad task wrapper script for external batch processing Create task_wrapper.sh that serves as the entrypoint for Nomad tasks in the external batch executor. The script: - Handles package retrieval from shared filesystem or S3 - Sets up execution environment (unsets Bundler vars, configures PATH) - Executes run_chunk.rb for assigned chunk index - Handles result storage back to shared filesystem or S3 - Implements proper error handling and logging - Supports configurable storage types via environment variables - Matches the interface expected by Nomad job templates * Add Nomad job template for batch processing * docs: add Nomad executor section to external_batch/README.md * Add Nomad executor unit tests for submit_nomad.rb and sync_results.rb. Includes tests for packaging, job submission, parameter handling, and result synchronization. Also adds a full pipeline test for Nomad executor. Ensures existing functionality remains unaffected. * external_batch/nomad: .env-based config, Ansible provisioning, task_wrapper fixes, doc cleanup BREAKING: docker-compose.yml no longer has hardcoded IP fallbacks for OS_SERVER_NOMAD_SSH_HOST or NOMAD_ADDR. Users must set these in .env. Changes: - docker-compose.yml: Remove hardcoded 10.60.126.125 from OS_SERVER_NOMAD_SSH_HOST; add NOMAD_ADDR env var to web and web-background services - .gitignore: Add .env (now un-tracked via git rm --cached) - .env.example: New template with placeholder values and instructions - Dockerfile: Add rsync package (required for SSH rsync bridge) - task_wrapper.sh: Use NOMAD_ALLOC_INDEX as fallback for --chunk-index; switch work dir to /tmp/nomad/task; use /usr/local/bin/run_chunk.rb; downgrade chunk failure to WARNING (non-fatal for individual data points) - job_array.hcl: raw_exec driver, nomad-client constraint, resources 2000MHz/4096MB/1000MB with tuning comments - ansible/: Complete provisioning playbook set (site.yml, 3 playbooks, 5 roles, create_cluster.py provisioner) - submit_nomad.rb / sync_results.rb: Update --ssh-host help text to use <NOMAD_SERVER_FLOATING_IP> placeholder - nomad/README.md: Update --ssh-host description, reference OS_SERVER_NOMAD_SSH_HOST - docs/external_batch_nomad_plan.md: Replace all hardcoded IPs with placeholders * ansible/README.md: document additive mode, --max, CLI flags, inventory merge - Quick Start: make auto-inventory path primary, manual copy secondary - Add additive mode section (auto-detect, skip server, offset indices, merge) - Add CLI flags reference table - Update 'what the script generates' to describe merge vs overwrite - Document --max, --no-auto-inventory, --inventory-output, --prefix flags * external batch Nomad/Ansible fixes: create_cluster refactor, task wrapper, job templates, .gitignore Ported from mvp-external-batch-fixes commit 3cc9d73 minus the parts that conflict with or duplicate develop: - dropped OpenStudio 3.10.0 / mongo 6.0.7 downgrades (develop stays 3.11.0 / mongo 8) - dropped bin/openstudio_meta bundler config-set rewrite (broke CI: persistent with/without group conflict between test and export install_gems runs) - dropped DJ/Resque job edits (hand-backports of fixes develop already has via #844/#845) - dropped empty ansible/tail file and personal .gitignore entries * Fix CI: rewrite nomad spec to script's real contract, ImageMagick-optional histograms - external_batch_nomad_spec: was stubbing a nomad CLI via --nomad-cmd, but submit_nomad.rb drives the Nomad HTTP API (/v1/jobs/parse -> /v1/jobs) and lays out <loc>/analysis_<id>/{package,results}; spec also died in before hook on an interpolating heredoc (NameError: output_flag), masking the mismatch. Rewritten against the HTTP API with an in-spec stub server. - nomad/sync_results.rb: NFS branch mirrors via FileUtils instead of rsync (absent on Windows dev boxes and slim containers; loop hung without it) - sampling/lhs.rb: preflight histogram attach non-fatal; paperclip needs ImageMagick identify, absent on ubuntu-24.04/macos-15 runners + PAT hosts (CRAN snapshot pin restore from the original commit is a no-op here: this branch is based on develop, which already has it via ba21522 / #849.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep tracked .env dev defaults from develop 71c2c39 moved compose config to .env.example + gitignored .env, but develop ships a tracked .env with dev-default credentials that docker-compose.yml requires out of the box (REDIS_PASSWORD etc). Keep both: .env.example documents the knobs, .env keeps a fresh clone bootable. Removing the tracked .env can be its own PR if wanted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * compose: drop runtime apt-get rsync hack; rsync is in the image now 71c2c39 installed rsync at container start because the published image lacked it; the same commit added rsync to the Dockerfile, and compose builds locally, so the runtime install is redundant. chmod of the optional ssh key moves to the operator (see external_batch/nomad/README). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review on #856 - sampling/lhs.rb: emit valid IHDR chunk (was IHLR + 12-byte pack, every histogram PNG invalid); require fileutils/tempfile explicitly - run_chunk.rb: bundle install once per runner process with completion marker keyed on Gemfile digest (was once per datapoint, racy on shared dirs); spawn .bat/.cmd via cmd.exe /c and .ps1 argv-style; stop rewriting CRLF->LF in Windows scripts - task_wrapper.sh: per-allocation work dir (NOMAD_TASK_DIR or mktemp, was hard-coded /tmp/nomad/task); dotfile/empty-dir-safe cp -a copies - submit_nomad.rb: validate --job-template/--package-location up front - external_batch.rb: require tmpdir; guard nil/empty sim_root_path - compose: mount tracked external_batch/nomad/ssh dir instead of gitignored key files (fresh-clone docker compose up worked never) - ansible: OpenStudio 3.10.0 -> 3.11.0 in examples/defaults per PR intent Verified: sampling_lhs + external_batch + external_batch_nomad specs (27 examples, 0 failures, 1 pending); generated PNG decodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ansible examples: OpenStudio 3.10.0 to match base branch 179 PR retargeted develop -> 179 (OS 3.10). Flip back to 3.11.0 when this work is promoted to develop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Alex Chapin <achapin@nrel.gov> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #847's CI work, prompted by the docker-job failure on #845's re-run (2026-07-16 ~19:03Z): the rserve image rebuild died in
Rscript install_packages.Ron transientcloud.r-project.orgdownload failures plus mid-day CRAN version drift (ggplot2 3.5.1 found, but >= 3.5.2 is required by 'ggrepel';gtable 0.3.5 loaded, but >= 0.3.6 required). The same rebuild had passed 2.5h earlier — nothing in the PR was at fault.Why every run rebuilt rserve
BuildKit (default since Docker 23, runners have 28) does not use pulled images as layer cache, so the
docker-compose pullbeforebuild— the original caching mechanism under the classic builder — stopped helping. Every docker job replayed the full R install (~10 min) against live CRAN.Changes
docker/R/**(two-commit tree diff against the fetched base tip, so it works on the shallow merge-ref clone; any fetch/diff failure falls back to building everything). The stack uses the pullednrel/openstudio-rserve:latest, whichdocker-uploadkeeps fresh on develop/master pushes. Push runs still build everything since those images get deployed.cache_from+BUILDKIT_INLINE_CACHEon all built services, so images pushed from develop carry layer-cache metadata and can seed the builds that do still happen.2026-07-15) instead of live CRAN —install_and_verifypinned its named packages but their dependencies floated, which is exactly what broke. The__linux__/jammypath serves prebuilt binaries for the base image (Ubuntu 22.04 + R 4.4):install_packages.Rdrops from ~480s to ~100s. Bump the snapshot date deliberately to take newer packages.Testing
docker build docker/Rwith the snapshot: completes, 13/13 packages install and pass their load check, R-install layer 102s (was 482s in CI).docker/R/**, so its own docker job exercises the "PR changed docker/R → rebuild" path end-to-end, including the snapshot install in CI.docker compose configvalidates; build args string-quoted for docker-compose 1.29's schema.🤖 Generated with Claude Code