Skip to content

Nomad executor + Ansible cluster provisioning for external batch - #856

Merged
brianlball merged 16 commits into
179from
external-batch-nomad
Jul 29, 2026
Merged

Nomad executor + Ansible cluster provisioning for external batch#856
brianlball merged 16 commits into
179from
external-batch-nomad

Conversation

@brianlball

Copy link
Copy Markdown
Contributor

What

Replaces mvp-external-batch-fixes (and PR #854) with the same work rebuilt on current develop.

mvp-external-batch-fixes branched from develop at #840 and duplicated the external-batch/LHS work that later merged (squashed) as #842, while missing every develop fix since (#844, #845, #847, #849, #852, #853). Its CI was failing for two reasons of its own making:

  • 3cc9d738 rewrote bin/openstudio_meta install_gems to persistent bundle config set --local with/without; the CI test phase then poisoned the export phase (a group cannot be in both with & without simultaneously), killing linux-test and macos-test in export_build_*.sh.
  • The branch predated CI: stop rebuilding rserve on every PR run; pin R deps to a CRAN snapshot #849, so the docker job rebuilt rserve against a drifting CRAN.

How this branch was built

Verified

external_batch_spec, external_batch_nomad_spec, external_batch_aws_spec, sampling_lhs_spec locally: 30 examples, 0 failures, 1 pending.

🤖 Generated with Claude Code

anchapin and others added 14 commits July 28, 2026 10:17
- Update developer guide to reflect UrbanOpt and custom gemfile support.
- Modify README to clarify MVP suitability and resolved limitations.
- Improve run_chunk.rb to support Windows scripts and enhance logging.
- Update LHS sampling to generate histogram images using pure Ruby.
- Refactor external_batch.rb to handle nil ENV variables gracefully.
- Adjust ingester and packager to support UrbanOpt and gemfile analyses.
- Revise tests to validate packaging for UrbanOpt and gemfile analyses.
Create task_wrapper.sh that serves as the entrypoint for Nomad tasks in the external batch executor. The script:
- Handles package retrieval from shared filesystem or S3
- Sets up execution environment (unsets Bundler vars, configures PATH)
- Executes run_chunk.rb for assigned chunk index
- Handles result storage back to shared filesystem or S3
- Implements proper error handling and logging
- Supports configurable storage types via environment variables
- Matches the interface expected by Nomad job templates
…. Includes tests for packaging, job submission, parameter handling, and result synchronization. Also adds a full pipeline test for Nomad executor. Ensures existing functionality remains unaffected.
…rapper fixes, doc cleanup

BREAKING: docker-compose.yml no longer has hardcoded IP fallbacks for
OS_SERVER_NOMAD_SSH_HOST or NOMAD_ADDR. Users must set these in .env.

Changes:
- docker-compose.yml: Remove hardcoded 10.60.126.125 from OS_SERVER_NOMAD_SSH_HOST;
  add NOMAD_ADDR env var to web and web-background services
- .gitignore: Add .env (now un-tracked via git rm --cached)
- .env.example: New template with placeholder values and instructions
- Dockerfile: Add rsync package (required for SSH rsync bridge)
- task_wrapper.sh: Use NOMAD_ALLOC_INDEX as fallback for --chunk-index;
  switch work dir to /tmp/nomad/task; use /usr/local/bin/run_chunk.rb;
  downgrade chunk failure to WARNING (non-fatal for individual data points)
- job_array.hcl: raw_exec driver, nomad-client constraint, resources
  2000MHz/4096MB/1000MB with tuning comments
- ansible/: Complete provisioning playbook set (site.yml, 3 playbooks,
  5 roles, create_cluster.py provisioner)
- submit_nomad.rb / sync_results.rb: Update --ssh-host help text to use
  <NOMAD_SERVER_FLOATING_IP> placeholder
- nomad/README.md: Update --ssh-host description, reference OS_SERVER_NOMAD_SSH_HOST
- docs/external_batch_nomad_plan.md: Replace all hardcoded IPs with placeholders
…y merge

- Quick Start: make auto-inventory path primary, manual copy secondary
- Add additive mode section (auto-detect, skip server, offset indices, merge)
- Add CLI flags reference table
- Update 'what the script generates' to describe merge vs overwrite
- Document --max, --no-auto-inventory, --inventory-output, --prefix flags
…pper, job templates, .gitignore

Ported from mvp-external-batch-fixes commit 3cc9d73 minus the parts that
conflict with or duplicate develop:
- dropped OpenStudio 3.10.0 / mongo 6.0.7 downgrades (develop stays 3.11.0 / mongo 8)
- dropped bin/openstudio_meta bundler config-set rewrite (broke CI: persistent
  with/without group conflict between test and export install_gems runs)
- dropped DJ/Resque job edits (hand-backports of fixes develop already has via #844/#845)
- dropped empty ansible/tail file and personal .gitignore entries
…ional histograms

- external_batch_nomad_spec: was stubbing a nomad CLI via --nomad-cmd, but
  submit_nomad.rb drives the Nomad HTTP API (/v1/jobs/parse -> /v1/jobs) and
  lays out <loc>/analysis_<id>/{package,results}; spec also died in before
  hook on an interpolating heredoc (NameError: output_flag), masking the
  mismatch. Rewritten against the HTTP API with an in-spec stub server.
- nomad/sync_results.rb: NFS branch mirrors via FileUtils instead of rsync
  (absent on Windows dev boxes and slim containers; loop hung without it)
- sampling/lhs.rb: preflight histogram attach non-fatal; paperclip needs
  ImageMagick identify, absent on ubuntu-24.04/macos-15 runners + PAT hosts

(CRAN snapshot pin restore from the original commit is a no-op here: this
branch is based on develop, which already has it via ba21522 / #849.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
71c2c39 moved compose config to .env.example + gitignored .env, but
develop ships a tracked .env with dev-default credentials that
docker-compose.yml requires out of the box (REDIS_PASSWORD etc).
Keep both: .env.example documents the knobs, .env keeps a fresh
clone bootable. Removing the tracked .env can be its own PR if wanted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
71c2c39 installed rsync at container start because the published image
lacked it; the same commit added rsync to the Dockerfile, and compose
builds locally, so the runtime install is redundant. chmod of the
optional ssh key moves to the operator (see external_batch/nomad/README).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the external-batch feature set on top of current develop, adding a Nomad-based executor path (job submission, task wrapper, result syncing, and Ansible provisioning) while also expanding external-batch capabilities to include UrbanOpt and per-analysis Gemfile workflows and enhancing Ruby LHS sampling with histogram generation.

Changes:

  • Add Nomad executor tooling (submit/sync scripts, task wrapper, Nomad job templates) plus server-side spec coverage for a Nomad-shaped pipeline.
  • Relax external-batch packaging constraints (UrbanOpt + custom gemfile allowed) and propagate those flags through the manifest for executor behavior.
  • Add Ansible-based Nomad/NFS/OpenStudio cluster provisioning and update Docker image/compose for external-batch/Nomad integration.

Reviewed changes

Copilot reviewed 39 out of 40 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
server/spec/models/external_batch_spec.rb Updates packager expectations to allow UrbanOpt/Gemfile packaging.
server/spec/models/external_batch_nomad_spec.rb Adds a full Nomad-shaped pipeline spec using a stub Nomad HTTP API.
server/app/lib/external_batch/packager.rb Removes UrbanOpt/Gemfile packaging rejections; adds manifest flags and require_relative.
server/app/lib/external_batch/ingester.rb Adds require_relative to ensure ExternalBatch module availability.
server/app/lib/external_batch.rb Makes external batch root resolution more defensive when ENV/APP_CONFIG are missing.
server/app/lib/analysis_library/sampling/lhs.rb Adds Ruby-side histogram generation and attachment (pure-Ruby PNG creation).
external_batch/templates/nomad.nomad Adds a generic Nomad job template supporting multiple execution styles.
external_batch/runner/run_chunk.rb Extends runner to support Windows scripts and adds bundler install flows for UrbanOpt/Gemfile analyses.
external_batch/README.md Expands documentation substantially and adds Nomad executor sections; updates stated limitations.
external_batch/nomad/templates/job_multiprocessor.hcl Adds a Nomad template for a single-group multiprocessor approach.
external_batch/nomad/templates/job_array.hcl Adds a Nomad template for one-task-per-chunk array-style execution.
external_batch/nomad/task_wrapper.sh Adds Nomad task wrapper to stage package, run the chunk runner, and export results.
external_batch/nomad/sync_results.rb Adds sync helper to mirror results to the server batch dir (S3/SSH/local modes).
external_batch/nomad/sync_results_spec.rb Adds tests for sync_results.rb behavior using local directories.
external_batch/nomad/submit_nomad.rb Adds Nomad submission helper (package push + HTTP API submit via /parse then /jobs).
external_batch/nomad/submit_nomad_spec.rb Adds tests for submit_nomad.rb behavior using a stubbed Nomad CLI/logging approach.
external_batch/nomad/README.md Adds Nomad executor documentation (usage, templates, result handling).
external_batch/nomad/ansible/roles/openstudio/tasks/main.yml Installs OpenStudio CLI and deploys runner scripts to clients.
external_batch/nomad/ansible/roles/nomad/server/tasks/main.yml Configures and validates a Nomad server node.
external_batch/nomad/ansible/roles/nomad/install/tasks/main.yml Installs Nomad binary + systemd service across nodes.
external_batch/nomad/ansible/roles/nomad/install/handlers/main.yml Adds handler to restart Nomad.
external_batch/nomad/ansible/roles/nomad/client/tasks/main.yml Configures Nomad client nodes and environment defaults.
external_batch/nomad/ansible/roles/nfs/server/tasks/main.yml Provisions NFS server export on the Nomad server.
external_batch/nomad/ansible/roles/nfs/server/handlers/main.yml Adds exportfs handler for NFS exports.
external_batch/nomad/ansible/roles/nfs/client/tasks/main.yml Provisions NFS client mounts on Nomad clients.
external_batch/nomad/ansible/roles/common/tasks/main.yml Common node bootstrap (hostname, /etc/hosts, packages).
external_batch/nomad/ansible/README.md Adds documentation for provisioning workflow and structure.
external_batch/nomad/ansible/playbooks/site.yml Orchestrates full provisioning (server then clients).
external_batch/nomad/ansible/playbooks/nomad-server.yml Provisions only the Nomad server role set.
external_batch/nomad/ansible/playbooks/nomad-clients.yml Provisions all Nomad clients with serial batching.
external_batch/nomad/ansible/Makefile Adds convenience targets for validation and deployment runs.
external_batch/nomad/ansible/inventory.yml.example Provides example inventory and variables for provisioning.
external_batch/nomad/ansible/create_cluster.py Adds OpenStack automation to create/extend a Nomad cluster and generate inventory.
external_batch/nomad/ansible/ansible.cfg Provides Ansible defaults tuned for the provisioning workflow.
external_batch/DEVELOPER_GUIDE.md Updates packager validation documentation to match new behavior.
docs/external_batch_nomad_plan.md Adds an integration plan doc for a Nomad-based external batch pipeline.
Dockerfile Adds rsync package installation into the image.
docker-compose.yml Adds env vars and bind mounts for external_batch + Nomad SSH integration.
.gitignore Adds ignores for .env, inventory, python cache, and Nomad SSH material.
.env.example Adds a compose-focused environment template including Nomad external batch knobs.
Comments suppressed due to low confidence (2)

docker-compose.yml:74

  • The web-background service has the same issue as web: it bind-mounts ./external_batch/nomad/id_ed25519 and ./external_batch/nomad/known_hosts, but those files are not in the repo (and are gitignored). This breaks docker compose up on fresh clones.
    volumes:
      - osdata:/mnt/openstudio
      - ./external_batch:/opt/openstudio/external_batch:ro
      - ./external_batch/nomad/id_ed25519:/config/ssh/id_rsync:ro
      - ./external_batch/nomad/known_hosts:/config/ssh/known_hosts:ro

external_batch/nomad/task_wrapper.sh:200

  • cp -r "$RESULTS_DIR"/* has the same problems as the package copy: it skips dotfiles and fails if the results dir is empty (or if only dotfiles exist). Use cp -a src/. dest/ to copy directory contents robustly.
    log "Copying results to NFS location: $RESULTS_DIR -> $RESULTS_FINAL_DIR"
    # Copy results to the final NFS location
    cp -r "$RESULTS_DIR"/* "$RESULTS_FINAL_DIR/"
fi

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

# See also https://openstudio.net/license
# *******************************************************************************

require 'zlib'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — added explicit require 'fileutils' / require 'tempfile'.

Comment on lines +240 to +242
# IHLR chunk
ihlr_data = [width, height, 8, 2, 0, 0, 0].pack('N2C4')
ihlr_chunk = [ihlr_data.length].pack('N') + 'IHLR' + ihlr_data + [Zlib::crc32('IHLR' + ihlr_data)].pack('N')

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — IHDR chunk name + pack('N2C5') (the old N2C4 also silently dropped the interlace byte, so the header was 12 bytes instead of 13). Generated PNG now passes a chunk/CRC walk, inflates to exactly h*(1+w*3) bytes, and decodes in System.Drawing.

Comment thread external_batch/runner/run_chunk.rb Outdated
Comment on lines +154 to +167
# Build batch command with arguments
cmd = ["\"#{script_path}\""]
cmd += args if args.any?
cmd = cmd.join(" ")

# Set up environment variables
env = { 'SCRIPT_ANALYSIS_ID' => analysis_id, 'SCRIPT_DATA_POINT_ID' => dp_id }

log_path = File.join(results_dp_dir, "#{script_name}.log")
File.chmod(0o755, script_path)
text = File.read(script_path)
File.open(script_path, 'wb') { |f| f.print text.gsub(/\r\n/m, "\n") }

pid = Process.spawn(env, cmd, [:out, :err] => [log_path, 'w'])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — .bat/.cmd now spawn argv-style via explicit cmd.exe /c, .ps1 via powershell.exe -ExecutionPolicy Bypass -File (no hand-built quoted strings), and the CRLF->LF rewrite + chmod are no longer applied to Windows scripts (rewrite kept for POSIX .sh only).

Comment thread external_batch/nomad/task_wrapper.sh Outdated
Comment on lines +79 to +83
# Set up directories
LOCAL_WORK_DIR="/tmp/nomad/task"
PACKAGE_DIR="$LOCAL_WORK_DIR/package"
RESULTS_DIR="$LOCAL_WORK_DIR/results"
mkdir -p "$PACKAGE_DIR" "$RESULTS_DIR"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — work dir is now $NOMAD_TASK_DIR/work (private per-allocation dir Nomad provides) when running under Nomad, else a mktemp -d dir with a cleanup trap for manual runs.

Comment thread docker-compose.yml Outdated
Comment on lines +42 to +46
volumes:
- osdata:/mnt/openstudio
- ./external_batch:/opt/openstudio/external_batch:ro
- ./external_batch/nomad/id_ed25519:/config/ssh/id_rsync:ro
- ./external_batch/nomad/known_hosts:/config/ssh/known_hosts:ro

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — replaced the two file binds with a single tracked-directory mount ./external_batch/nomad/ssh:/config/ssh:ro. Only a README is in git (contents gitignored), so fresh-clone docker compose up works before any keys exist; users drop id_rsync/known_hosts into that dir to enable the rsync bridge.

opt_parser.parse!

batch_dir = ARGV.shift
abort opt_parser.banner if batch_dir.nil?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — submit_nomad.rb now aborts with the usage banner when --job-template or --package-location is missing, and verifies the template file exists before reading it. README updated to list --package-location as required.

Comment thread server/app/lib/external_batch.rb Outdated
Comment on lines +21 to +28
if defined?(APP_CONFIG) && APP_CONFIG
sim_root_path = APP_CONFIG['sim_root_path']
else
# Fallback to a reasonable default if APP_CONFIG is not available
sim_root_path = Dir.tmpdir # Use system temp directory as fallback
end
root = File.join(sim_root_path, 'external_batch')
else

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — require 'tmpdir' at the top of the file, and the Dir.tmpdir fallback now kicks in whenever sim_root_path is nil/empty (not only when APP_CONFIG is undefined), so File.join can no longer see a nil.

Comment thread external_batch/runner/run_chunk.rb Outdated
Comment on lines +252 to +326
run_data_point_script(analysis_dir, manifest['analysis_id'], dp_id, 'initialize', tmp_results_dp, runner_log)

# Handle UrbanOpt bundling if this is an UrbanOpt analysis
if manifest['urbanopt']
urbanopt_dir = File.join(analysis_dir, 'lib', 'urbanopt')
gemfile_path = File.join(urbanopt_dir, 'Gemfile')
if File.file?(gemfile_path)
bundle_log_path = File.join(tmp_results_dp, 'urbanopt_bundle.log')
runner_log.puts "Installing UrbanOpt bundle using gemfile #{gemfile_path}"

# Set up bundle environment
bundle_env = {
'BUNDLE_GEMFILE' => gemfile_path,
'BUNDLE_PATH' => File.join(urbanopt_dir, '.bundle'),
'RUBYOPT' => nil,
'BUNDLER_SETUP' => nil
}

cmd = "bundle install --gemfile=#{gemfile_path} --path=#{File.join(urbanopt_dir, '.bundle')} --retry 10"
runner_log.puts "Bundle install command: #{cmd}"

pid = Process.spawn(bundle_env, cmd, [:out, :err] => [bundle_log_path, 'w'], chdir: urbanopt_dir, **spawn_pgroup_opts)
begin
Timeout.timeout(4 * 3600) { Process.wait(pid) } # 4 hour timeout for bundle install
runner_log.puts "UrbanOpt bundle install exited with #{$?.exitstatus}"
if $?.exitstatus != 0
runner_log.puts "UrbanOpt bundle install failed: #{File.read(bundle_log_path)}"
end
rescue Timeout::Error
kill_process_tree(pid, runner_log) if pid
runner_log.puts "UrbanOpt bundle install killed after 4h timeout"
raise "UrbanOpt bundle install timed out"
rescue StandardError => e
runner_log.puts "UrbanOpt bundle install failed: #{e.message}"
raise e
end
else
runner_log.puts "UrbanOpt analysis detected but no Gemfile found at #{gemfile_path}"
end
end

# Handle gemfile bundling if this analysis has a custom gemfile
if manifest['gemfile']
gemfile_path = File.join(analysis_dir, 'Gemfile')
if File.file?(gemfile_path)
bundle_log_path = File.join(tmp_results_dp, 'bundle.log')
runner_log.puts "Installing gems using gemfile #{gemfile_path}"

# Set up bundle environment
bundle_env = {
'BUNDLE_GEMFILE' => gemfile_path,
'BUNDLE_PATH' => File.join(analysis_dir, 'gems'),
'RUBYOPT' => nil,
'BUNDLER_SETUP' => nil
}

cmd = "bundle install --gemfile=#{gemfile_path} --path=#{File.join(analysis_dir, 'gems')} --retry 3"
runner_log.puts "Bundle install command: #{cmd}"

pid = Process.spawn(bundle_env, cmd, [:out, :err] => [bundle_log_path, 'w'], chdir: analysis_dir, **spawn_pgroup_opts)
begin
Timeout.timeout(4 * 3600) { Process.wait(pid) } # 4 hour timeout for bundle install
runner_log.puts "Bundle install exited with #{$?.exitstatus}"
if $?.exitstatus != 0
runner_log.puts "Bundle install failed: #{File.read(bundle_log_path)}"
end
rescue Timeout::Error
kill_process_tree(pid, runner_log) if pid
runner_log.puts "Bundle install killed after 4h timeout"
raise "Bundle install timed out"
rescue StandardError => e
runner_log.puts "Bundle install failed: #{e.message}"
raise e
end
else

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — bundler installs are hoisted out of run_data_point to a single install_bundle call per runner process, guarded by a completion marker keyed on the Gemfile's SHA256 so other runner processes sharing the same analysis_dir skip the install instead of racing it. Install failure no longer raises mid-datapoint: dps still run and reach a terminal status.json (the OSCLI --bundle invocation fails fast with its own log).

Comment thread external_batch/nomad/task_wrapper.sh Outdated
Comment on lines +102 to +103
log "Copying package from NFS location: $PACKAGE_URI_NORMALIZED"
cp -r "$PACKAGE_URI_NORMALIZED"/* "$PACKAGE_DIR/"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc769dc — both copies now use cp -a "src/." "dest/" (dotfile-safe, empty-dir-safe, preserves permissions/timestamps).

Comment thread external_batch/nomad/ansible/inventory.yml.example
- sampling/lhs.rb: emit valid IHDR chunk (was IHLR + 12-byte pack, every
  histogram PNG invalid); require fileutils/tempfile explicitly
- run_chunk.rb: bundle install once per runner process with completion
  marker keyed on Gemfile digest (was once per datapoint, racy on shared
  dirs); spawn .bat/.cmd via cmd.exe /c and .ps1 argv-style; stop
  rewriting CRLF->LF in Windows scripts
- task_wrapper.sh: per-allocation work dir (NOMAD_TASK_DIR or mktemp,
  was hard-coded /tmp/nomad/task); dotfile/empty-dir-safe cp -a copies
- submit_nomad.rb: validate --job-template/--package-location up front
- external_batch.rb: require tmpdir; guard nil/empty sim_root_path
- compose: mount tracked external_batch/nomad/ssh dir instead of
  gitignored key files (fresh-clone docker compose up worked never)
- ansible: OpenStudio 3.10.0 -> 3.11.0 in examples/defaults per PR intent

Verified: sampling_lhs + external_batch + external_batch_nomad specs
(27 examples, 0 failures, 1 pending); generated PNG decodes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@brianlball
brianlball changed the base branch from develop to 179 July 29, 2026 12:58
PR retargeted develop -> 179 (OS 3.10). Flip back to 3.11.0 when this
work is promoted to develop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@brianlball
brianlball merged commit efa8c66 into 179 Jul 29, 2026
7 of 8 checks passed
@brianlball
brianlball deleted the external-batch-nomad branch July 29, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants