Nomad executor + Ansible cluster provisioning for external batch - #856
Conversation
- Update developer guide to reflect UrbanOpt and custom gemfile support. - Modify README to clarify MVP suitability and resolved limitations. - Improve run_chunk.rb to support Windows scripts and enhance logging. - Update LHS sampling to generate histogram images using pure Ruby. - Refactor external_batch.rb to handle nil ENV variables gracefully. - Adjust ingester and packager to support UrbanOpt and gemfile analyses. - Revise tests to validate packaging for UrbanOpt and gemfile analyses.
Create task_wrapper.sh that serves as the entrypoint for Nomad tasks in the external batch executor. The script: - Handles package retrieval from shared filesystem or S3 - Sets up execution environment (unsets Bundler vars, configures PATH) - Executes run_chunk.rb for assigned chunk index - Handles result storage back to shared filesystem or S3 - Implements proper error handling and logging - Supports configurable storage types via environment variables - Matches the interface expected by Nomad job templates
…. Includes tests for packaging, job submission, parameter handling, and result synchronization. Also adds a full pipeline test for Nomad executor. Ensures existing functionality remains unaffected.
…rapper fixes, doc cleanup BREAKING: docker-compose.yml no longer has hardcoded IP fallbacks for OS_SERVER_NOMAD_SSH_HOST or NOMAD_ADDR. Users must set these in .env. Changes: - docker-compose.yml: Remove hardcoded 10.60.126.125 from OS_SERVER_NOMAD_SSH_HOST; add NOMAD_ADDR env var to web and web-background services - .gitignore: Add .env (now un-tracked via git rm --cached) - .env.example: New template with placeholder values and instructions - Dockerfile: Add rsync package (required for SSH rsync bridge) - task_wrapper.sh: Use NOMAD_ALLOC_INDEX as fallback for --chunk-index; switch work dir to /tmp/nomad/task; use /usr/local/bin/run_chunk.rb; downgrade chunk failure to WARNING (non-fatal for individual data points) - job_array.hcl: raw_exec driver, nomad-client constraint, resources 2000MHz/4096MB/1000MB with tuning comments - ansible/: Complete provisioning playbook set (site.yml, 3 playbooks, 5 roles, create_cluster.py provisioner) - submit_nomad.rb / sync_results.rb: Update --ssh-host help text to use <NOMAD_SERVER_FLOATING_IP> placeholder - nomad/README.md: Update --ssh-host description, reference OS_SERVER_NOMAD_SSH_HOST - docs/external_batch_nomad_plan.md: Replace all hardcoded IPs with placeholders
…y merge - Quick Start: make auto-inventory path primary, manual copy secondary - Add additive mode section (auto-detect, skip server, offset indices, merge) - Add CLI flags reference table - Update 'what the script generates' to describe merge vs overwrite - Document --max, --no-auto-inventory, --inventory-output, --prefix flags
…pper, job templates, .gitignore Ported from mvp-external-batch-fixes commit 3cc9d73 minus the parts that conflict with or duplicate develop: - dropped OpenStudio 3.10.0 / mongo 6.0.7 downgrades (develop stays 3.11.0 / mongo 8) - dropped bin/openstudio_meta bundler config-set rewrite (broke CI: persistent with/without group conflict between test and export install_gems runs) - dropped DJ/Resque job edits (hand-backports of fixes develop already has via #844/#845) - dropped empty ansible/tail file and personal .gitignore entries
…ional histograms
- external_batch_nomad_spec: was stubbing a nomad CLI via --nomad-cmd, but
submit_nomad.rb drives the Nomad HTTP API (/v1/jobs/parse -> /v1/jobs) and
lays out <loc>/analysis_<id>/{package,results}; spec also died in before
hook on an interpolating heredoc (NameError: output_flag), masking the
mismatch. Rewritten against the HTTP API with an in-spec stub server.
- nomad/sync_results.rb: NFS branch mirrors via FileUtils instead of rsync
(absent on Windows dev boxes and slim containers; loop hung without it)
- sampling/lhs.rb: preflight histogram attach non-fatal; paperclip needs
ImageMagick identify, absent on ubuntu-24.04/macos-15 runners + PAT hosts
(CRAN snapshot pin restore from the original commit is a no-op here: this
branch is based on develop, which already has it via ba21522 / #849.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
71c2c39 moved compose config to .env.example + gitignored .env, but develop ships a tracked .env with dev-default credentials that docker-compose.yml requires out of the box (REDIS_PASSWORD etc). Keep both: .env.example documents the knobs, .env keeps a fresh clone bootable. Removing the tracked .env can be its own PR if wanted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
71c2c39 installed rsync at container start because the published image lacked it; the same commit added rsync to the Dockerfile, and compose builds locally, so the runtime install is redundant. chmod of the optional ssh key moves to the operator (see external_batch/nomad/README). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the external-batch feature set on top of current develop, adding a Nomad-based executor path (job submission, task wrapper, result syncing, and Ansible provisioning) while also expanding external-batch capabilities to include UrbanOpt and per-analysis Gemfile workflows and enhancing Ruby LHS sampling with histogram generation.
Changes:
- Add Nomad executor tooling (submit/sync scripts, task wrapper, Nomad job templates) plus server-side spec coverage for a Nomad-shaped pipeline.
- Relax external-batch packaging constraints (UrbanOpt + custom gemfile allowed) and propagate those flags through the manifest for executor behavior.
- Add Ansible-based Nomad/NFS/OpenStudio cluster provisioning and update Docker image/compose for external-batch/Nomad integration.
Reviewed changes
Copilot reviewed 39 out of 40 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| server/spec/models/external_batch_spec.rb | Updates packager expectations to allow UrbanOpt/Gemfile packaging. |
| server/spec/models/external_batch_nomad_spec.rb | Adds a full Nomad-shaped pipeline spec using a stub Nomad HTTP API. |
| server/app/lib/external_batch/packager.rb | Removes UrbanOpt/Gemfile packaging rejections; adds manifest flags and require_relative. |
| server/app/lib/external_batch/ingester.rb | Adds require_relative to ensure ExternalBatch module availability. |
| server/app/lib/external_batch.rb | Makes external batch root resolution more defensive when ENV/APP_CONFIG are missing. |
| server/app/lib/analysis_library/sampling/lhs.rb | Adds Ruby-side histogram generation and attachment (pure-Ruby PNG creation). |
| external_batch/templates/nomad.nomad | Adds a generic Nomad job template supporting multiple execution styles. |
| external_batch/runner/run_chunk.rb | Extends runner to support Windows scripts and adds bundler install flows for UrbanOpt/Gemfile analyses. |
| external_batch/README.md | Expands documentation substantially and adds Nomad executor sections; updates stated limitations. |
| external_batch/nomad/templates/job_multiprocessor.hcl | Adds a Nomad template for a single-group multiprocessor approach. |
| external_batch/nomad/templates/job_array.hcl | Adds a Nomad template for one-task-per-chunk array-style execution. |
| external_batch/nomad/task_wrapper.sh | Adds Nomad task wrapper to stage package, run the chunk runner, and export results. |
| external_batch/nomad/sync_results.rb | Adds sync helper to mirror results to the server batch dir (S3/SSH/local modes). |
| external_batch/nomad/sync_results_spec.rb | Adds tests for sync_results.rb behavior using local directories. |
| external_batch/nomad/submit_nomad.rb | Adds Nomad submission helper (package push + HTTP API submit via /parse then /jobs). |
| external_batch/nomad/submit_nomad_spec.rb | Adds tests for submit_nomad.rb behavior using a stubbed Nomad CLI/logging approach. |
| external_batch/nomad/README.md | Adds Nomad executor documentation (usage, templates, result handling). |
| external_batch/nomad/ansible/roles/openstudio/tasks/main.yml | Installs OpenStudio CLI and deploys runner scripts to clients. |
| external_batch/nomad/ansible/roles/nomad/server/tasks/main.yml | Configures and validates a Nomad server node. |
| external_batch/nomad/ansible/roles/nomad/install/tasks/main.yml | Installs Nomad binary + systemd service across nodes. |
| external_batch/nomad/ansible/roles/nomad/install/handlers/main.yml | Adds handler to restart Nomad. |
| external_batch/nomad/ansible/roles/nomad/client/tasks/main.yml | Configures Nomad client nodes and environment defaults. |
| external_batch/nomad/ansible/roles/nfs/server/tasks/main.yml | Provisions NFS server export on the Nomad server. |
| external_batch/nomad/ansible/roles/nfs/server/handlers/main.yml | Adds exportfs handler for NFS exports. |
| external_batch/nomad/ansible/roles/nfs/client/tasks/main.yml | Provisions NFS client mounts on Nomad clients. |
| external_batch/nomad/ansible/roles/common/tasks/main.yml | Common node bootstrap (hostname, /etc/hosts, packages). |
| external_batch/nomad/ansible/README.md | Adds documentation for provisioning workflow and structure. |
| external_batch/nomad/ansible/playbooks/site.yml | Orchestrates full provisioning (server then clients). |
| external_batch/nomad/ansible/playbooks/nomad-server.yml | Provisions only the Nomad server role set. |
| external_batch/nomad/ansible/playbooks/nomad-clients.yml | Provisions all Nomad clients with serial batching. |
| external_batch/nomad/ansible/Makefile | Adds convenience targets for validation and deployment runs. |
| external_batch/nomad/ansible/inventory.yml.example | Provides example inventory and variables for provisioning. |
| external_batch/nomad/ansible/create_cluster.py | Adds OpenStack automation to create/extend a Nomad cluster and generate inventory. |
| external_batch/nomad/ansible/ansible.cfg | Provides Ansible defaults tuned for the provisioning workflow. |
| external_batch/DEVELOPER_GUIDE.md | Updates packager validation documentation to match new behavior. |
| docs/external_batch_nomad_plan.md | Adds an integration plan doc for a Nomad-based external batch pipeline. |
| Dockerfile | Adds rsync package installation into the image. |
| docker-compose.yml | Adds env vars and bind mounts for external_batch + Nomad SSH integration. |
| .gitignore | Adds ignores for .env, inventory, python cache, and Nomad SSH material. |
| .env.example | Adds a compose-focused environment template including Nomad external batch knobs. |
Comments suppressed due to low confidence (2)
docker-compose.yml:74
- The web-background service has the same issue as web: it bind-mounts ./external_batch/nomad/id_ed25519 and ./external_batch/nomad/known_hosts, but those files are not in the repo (and are gitignored). This breaks
docker compose upon fresh clones.
volumes:
- osdata:/mnt/openstudio
- ./external_batch:/opt/openstudio/external_batch:ro
- ./external_batch/nomad/id_ed25519:/config/ssh/id_rsync:ro
- ./external_batch/nomad/known_hosts:/config/ssh/known_hosts:ro
external_batch/nomad/task_wrapper.sh:200
cp -r "$RESULTS_DIR"/*has the same problems as the package copy: it skips dotfiles and fails if the results dir is empty (or if only dotfiles exist). Usecp -a src/. dest/to copy directory contents robustly.
log "Copying results to NFS location: $RESULTS_DIR -> $RESULTS_FINAL_DIR"
# Copy results to the final NFS location
cp -r "$RESULTS_DIR"/* "$RESULTS_FINAL_DIR/"
fi
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # See also https://openstudio.net/license | ||
| # ******************************************************************************* | ||
|
|
||
| require 'zlib' |
There was a problem hiding this comment.
Fixed in cc769dc — added explicit require 'fileutils' / require 'tempfile'.
| # IHLR chunk | ||
| ihlr_data = [width, height, 8, 2, 0, 0, 0].pack('N2C4') | ||
| ihlr_chunk = [ihlr_data.length].pack('N') + 'IHLR' + ihlr_data + [Zlib::crc32('IHLR' + ihlr_data)].pack('N') |
There was a problem hiding this comment.
Fixed in cc769dc — IHDR chunk name + pack('N2C5') (the old N2C4 also silently dropped the interlace byte, so the header was 12 bytes instead of 13). Generated PNG now passes a chunk/CRC walk, inflates to exactly h*(1+w*3) bytes, and decodes in System.Drawing.
| # Build batch command with arguments | ||
| cmd = ["\"#{script_path}\""] | ||
| cmd += args if args.any? | ||
| cmd = cmd.join(" ") | ||
|
|
||
| # Set up environment variables | ||
| env = { 'SCRIPT_ANALYSIS_ID' => analysis_id, 'SCRIPT_DATA_POINT_ID' => dp_id } | ||
|
|
||
| log_path = File.join(results_dp_dir, "#{script_name}.log") | ||
| File.chmod(0o755, script_path) | ||
| text = File.read(script_path) | ||
| File.open(script_path, 'wb') { |f| f.print text.gsub(/\r\n/m, "\n") } | ||
|
|
||
| pid = Process.spawn(env, cmd, [:out, :err] => [log_path, 'w']) |
There was a problem hiding this comment.
Fixed in cc769dc — .bat/.cmd now spawn argv-style via explicit cmd.exe /c, .ps1 via powershell.exe -ExecutionPolicy Bypass -File (no hand-built quoted strings), and the CRLF->LF rewrite + chmod are no longer applied to Windows scripts (rewrite kept for POSIX .sh only).
| # Set up directories | ||
| LOCAL_WORK_DIR="/tmp/nomad/task" | ||
| PACKAGE_DIR="$LOCAL_WORK_DIR/package" | ||
| RESULTS_DIR="$LOCAL_WORK_DIR/results" | ||
| mkdir -p "$PACKAGE_DIR" "$RESULTS_DIR" |
There was a problem hiding this comment.
Fixed in cc769dc — work dir is now $NOMAD_TASK_DIR/work (private per-allocation dir Nomad provides) when running under Nomad, else a mktemp -d dir with a cleanup trap for manual runs.
| volumes: | ||
| - osdata:/mnt/openstudio | ||
| - ./external_batch:/opt/openstudio/external_batch:ro | ||
| - ./external_batch/nomad/id_ed25519:/config/ssh/id_rsync:ro | ||
| - ./external_batch/nomad/known_hosts:/config/ssh/known_hosts:ro |
There was a problem hiding this comment.
Fixed in cc769dc — replaced the two file binds with a single tracked-directory mount ./external_batch/nomad/ssh:/config/ssh:ro. Only a README is in git (contents gitignored), so fresh-clone docker compose up works before any keys exist; users drop id_rsync/known_hosts into that dir to enable the rsync bridge.
| opt_parser.parse! | ||
|
|
||
| batch_dir = ARGV.shift | ||
| abort opt_parser.banner if batch_dir.nil? |
There was a problem hiding this comment.
Fixed in cc769dc — submit_nomad.rb now aborts with the usage banner when --job-template or --package-location is missing, and verifies the template file exists before reading it. README updated to list --package-location as required.
| if defined?(APP_CONFIG) && APP_CONFIG | ||
| sim_root_path = APP_CONFIG['sim_root_path'] | ||
| else | ||
| # Fallback to a reasonable default if APP_CONFIG is not available | ||
| sim_root_path = Dir.tmpdir # Use system temp directory as fallback | ||
| end | ||
| root = File.join(sim_root_path, 'external_batch') | ||
| else |
There was a problem hiding this comment.
Fixed in cc769dc — require 'tmpdir' at the top of the file, and the Dir.tmpdir fallback now kicks in whenever sim_root_path is nil/empty (not only when APP_CONFIG is undefined), so File.join can no longer see a nil.
| run_data_point_script(analysis_dir, manifest['analysis_id'], dp_id, 'initialize', tmp_results_dp, runner_log) | ||
|
|
||
| # Handle UrbanOpt bundling if this is an UrbanOpt analysis | ||
| if manifest['urbanopt'] | ||
| urbanopt_dir = File.join(analysis_dir, 'lib', 'urbanopt') | ||
| gemfile_path = File.join(urbanopt_dir, 'Gemfile') | ||
| if File.file?(gemfile_path) | ||
| bundle_log_path = File.join(tmp_results_dp, 'urbanopt_bundle.log') | ||
| runner_log.puts "Installing UrbanOpt bundle using gemfile #{gemfile_path}" | ||
|
|
||
| # Set up bundle environment | ||
| bundle_env = { | ||
| 'BUNDLE_GEMFILE' => gemfile_path, | ||
| 'BUNDLE_PATH' => File.join(urbanopt_dir, '.bundle'), | ||
| 'RUBYOPT' => nil, | ||
| 'BUNDLER_SETUP' => nil | ||
| } | ||
|
|
||
| cmd = "bundle install --gemfile=#{gemfile_path} --path=#{File.join(urbanopt_dir, '.bundle')} --retry 10" | ||
| runner_log.puts "Bundle install command: #{cmd}" | ||
|
|
||
| pid = Process.spawn(bundle_env, cmd, [:out, :err] => [bundle_log_path, 'w'], chdir: urbanopt_dir, **spawn_pgroup_opts) | ||
| begin | ||
| Timeout.timeout(4 * 3600) { Process.wait(pid) } # 4 hour timeout for bundle install | ||
| runner_log.puts "UrbanOpt bundle install exited with #{$?.exitstatus}" | ||
| if $?.exitstatus != 0 | ||
| runner_log.puts "UrbanOpt bundle install failed: #{File.read(bundle_log_path)}" | ||
| end | ||
| rescue Timeout::Error | ||
| kill_process_tree(pid, runner_log) if pid | ||
| runner_log.puts "UrbanOpt bundle install killed after 4h timeout" | ||
| raise "UrbanOpt bundle install timed out" | ||
| rescue StandardError => e | ||
| runner_log.puts "UrbanOpt bundle install failed: #{e.message}" | ||
| raise e | ||
| end | ||
| else | ||
| runner_log.puts "UrbanOpt analysis detected but no Gemfile found at #{gemfile_path}" | ||
| end | ||
| end | ||
|
|
||
| # Handle gemfile bundling if this analysis has a custom gemfile | ||
| if manifest['gemfile'] | ||
| gemfile_path = File.join(analysis_dir, 'Gemfile') | ||
| if File.file?(gemfile_path) | ||
| bundle_log_path = File.join(tmp_results_dp, 'bundle.log') | ||
| runner_log.puts "Installing gems using gemfile #{gemfile_path}" | ||
|
|
||
| # Set up bundle environment | ||
| bundle_env = { | ||
| 'BUNDLE_GEMFILE' => gemfile_path, | ||
| 'BUNDLE_PATH' => File.join(analysis_dir, 'gems'), | ||
| 'RUBYOPT' => nil, | ||
| 'BUNDLER_SETUP' => nil | ||
| } | ||
|
|
||
| cmd = "bundle install --gemfile=#{gemfile_path} --path=#{File.join(analysis_dir, 'gems')} --retry 3" | ||
| runner_log.puts "Bundle install command: #{cmd}" | ||
|
|
||
| pid = Process.spawn(bundle_env, cmd, [:out, :err] => [bundle_log_path, 'w'], chdir: analysis_dir, **spawn_pgroup_opts) | ||
| begin | ||
| Timeout.timeout(4 * 3600) { Process.wait(pid) } # 4 hour timeout for bundle install | ||
| runner_log.puts "Bundle install exited with #{$?.exitstatus}" | ||
| if $?.exitstatus != 0 | ||
| runner_log.puts "Bundle install failed: #{File.read(bundle_log_path)}" | ||
| end | ||
| rescue Timeout::Error | ||
| kill_process_tree(pid, runner_log) if pid | ||
| runner_log.puts "Bundle install killed after 4h timeout" | ||
| raise "Bundle install timed out" | ||
| rescue StandardError => e | ||
| runner_log.puts "Bundle install failed: #{e.message}" | ||
| raise e | ||
| end | ||
| else |
There was a problem hiding this comment.
Fixed in cc769dc — bundler installs are hoisted out of run_data_point to a single install_bundle call per runner process, guarded by a completion marker keyed on the Gemfile's SHA256 so other runner processes sharing the same analysis_dir skip the install instead of racing it. Install failure no longer raises mid-datapoint: dps still run and reach a terminal status.json (the OSCLI --bundle invocation fails fast with its own log).
| log "Copying package from NFS location: $PACKAGE_URI_NORMALIZED" | ||
| cp -r "$PACKAGE_URI_NORMALIZED"/* "$PACKAGE_DIR/" |
There was a problem hiding this comment.
Fixed in cc769dc — both copies now use cp -a "src/." "dest/" (dotfile-safe, empty-dir-safe, preserves permissions/timestamps).
- sampling/lhs.rb: emit valid IHDR chunk (was IHLR + 12-byte pack, every histogram PNG invalid); require fileutils/tempfile explicitly - run_chunk.rb: bundle install once per runner process with completion marker keyed on Gemfile digest (was once per datapoint, racy on shared dirs); spawn .bat/.cmd via cmd.exe /c and .ps1 argv-style; stop rewriting CRLF->LF in Windows scripts - task_wrapper.sh: per-allocation work dir (NOMAD_TASK_DIR or mktemp, was hard-coded /tmp/nomad/task); dotfile/empty-dir-safe cp -a copies - submit_nomad.rb: validate --job-template/--package-location up front - external_batch.rb: require tmpdir; guard nil/empty sim_root_path - compose: mount tracked external_batch/nomad/ssh dir instead of gitignored key files (fresh-clone docker compose up worked never) - ansible: OpenStudio 3.10.0 -> 3.11.0 in examples/defaults per PR intent Verified: sampling_lhs + external_batch + external_batch_nomad specs (27 examples, 0 failures, 1 pending); generated PNG decodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR retargeted develop -> 179 (OS 3.10). Flip back to 3.11.0 when this work is promoted to develop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What
Replaces
mvp-external-batch-fixes(and PR #854) with the same work rebuilt on currentdevelop.mvp-external-batch-fixesbranched from develop at #840 and duplicated the external-batch/LHS work that later merged (squashed) as #842, while missing every develop fix since (#844, #845, #847, #849, #852, #853). Its CI was failing for two reasons of its own making:3cc9d738rewrotebin/openstudio_metainstall_gemsto persistentbundle config set --local with/without; the CI test phase then poisoned the export phase (a group cannot be in both with & without simultaneously), killing linux-test and macos-test inexport_build_*.sh.How this branch was built
develop(keeps OpenStudio 3.11.0, mongo 8, Add external batch execution (AWS Batch) and a pure-Ruby LHS sampling backend #842–Fix nil-DataPoint crash masking real errors in ResqueJobs::RunSimulateDataPoint #853 intact).17fa57cf,7cc6c9e9,ff35886f..47c9d7e9.3cc9d738ported surgically: kept the Nomad/Ansible/task-wrapper/create_cluster work and.gitignorehygiene; dropped the OpenStudio 3.10.0 + mongo 6.0.7 downgrades, theopenstudio_metabundler rewrite (the CI breaker), the emptyansible/tailfile, and its DJ/Resque job edits (hand-backports of fixes develop already has via Fail fast on corrupt seed.zip: validate at upload, no futile retries, terminal 'failed' state (#841) #844/Harden RunSimulateDataPoint worker against permanent lock deadlock and stale-job re-dispatch #845).sync_results.rbworks without rsync, preflight histogram attach is non-fatal without ImageMagick..env(removing it breaks fresh-clonedocker-compose up);.env.exampledocuments the knobs.apt-get install rsynccompose command-override — the same commit adds rsync to the Dockerfile.Verified
external_batch_spec,external_batch_nomad_spec,external_batch_aws_spec,sampling_lhs_speclocally: 30 examples, 0 failures, 1 pending.🤖 Generated with Claude Code