Add external batch execution (AWS Batch) and a pure-Ruby LHS sampling backend - #842
Merged
Conversation
…deployments - AnalysisLibrary::Sampling: Ruby LHS (randomLHS strata + qnorm/qlnorm/qunif/ qtriangle/qdiscrete) replicating the R backend; lhs.rb backend switch via config.x.sampling_backend (:rserve default, :ruby on local-server/local-test); Variable.pivot_array works without an R session. LHS now runs on openstudio_meta start_local (no Docker/Redis/Rserve). - ExternalBatchRun analysis type (sibling of batch_run): packages datapoints with pre-translated OSWs, marks queued, incrementally ingests results. - ExternalBatch::Packager/Ingester: worker-layout package + worker-equivalent Mongo ingest (results, sdp_log_file truncation, ResultFiles, status flags). - external_batch/ runner (plain Ruby, SLURM/AWS-Batch/local chunk indexing), local mock executor, Kestrel Apptainer sbatch template, README. - openstudio_meta run_analysis --batch-run-method flag. - Specs: distributions math, Ruby LHS sampling + end-to-end lhs perform, packager/ingester/ExternalBatchRun, and full package->executor->ingest pipeline against a stub OpenStudio CLI. All green; spec/models 67/67. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aws/Dockerfile + task_wrapper.sh: runner image (nrel/openstudio + AWS CLI); each array task syncs the package from S3, runs its chunk, pushes results back (partial pushes safe: status.json written last, ingester skips others). - aws/submit_batch.rb: push package to S3 + submit array job sized to the chunk count (single chunk -> plain job with CHUNK_INDEX); creds via the standard AWS CLI chain; containers use an IAM task role, no keys. - aws/sync_results.rb: mirrors s3 results down to the local batch dir until all chunk done-markers arrive, feeding the unchanged server ingest loop. - aws/infra/main.tf: one-time Terraform (S3 + lifecycle, ECR, scoped 3-role IAM, on-demand BEST_FIT_PROGRESSIVE compute env on m6idn/m5d, queue, job definition 1 vCPU/7500MB, retry=2). - local_executor.rb: create results dir (synced batch dirs lack it - S3 has no empty dirs); found by the new AWS pipeline spec. - Specs: shared fixture/stub helpers extracted to spec/support; stub `aws` CLI emulates S3 as a local folder; full AWS-shaped pipeline covered (package -> submit -> array tasks -> sync down -> ingest). 37 examples, 0 failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… doc - ExternalBatch.root_dir: expand_path the root so backslash Windows paths from OS_SERVER_EXTERNAL_BATCH_ROOT don't act as glob escapes in the ingester's reports Dir[] (found in the real-AWS smoke test: eplustbl.html not attached). Spec now sets the root with native separators to keep this covered. - external_batch/aws/WALKTHROUGH.md: first-time user guide verified against a real AWS run (tool installs, access keys, terraform, image push, run, console navigation/monitoring, troubleshooting, cleanup). Verified end-to-end on real AWS Batch 2026-07-02: 2-task array job on m5d.xlarge, EnergyPlus sims succeeded, results round-tripped via S3, ingested to completed datapoints with correct LHS-sampled values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
State files carry account IDs/ARNs; keep them out of any future git add -A. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- spec/features/aws_batch_smoke_spec.rb: runs the algo-spec SEB discrete LHS calibration project on the user's own AWS Batch infra (env-gated via AWS_BATCH_SMOKE_* vars; skips cleanly otherwise, so CI is unaffected) and asserts the calibration metrics match the docker_stack_algo_spec golden values. Validated against real AWS: 1 example, 0 failures, values identical to the classic docker/R path. The mocked CI twin remains spec/models/external_batch_aws_spec.rb (stub aws CLI + stub OpenStudio). - doc/aws_smoke_runbook.md: sanitized reproduction record of the first two real-AWS smoke runs (exact commands, gotchas, timings, outcomes). - WALKTHROUGH.md: billing section (budget alert via console + CLI; where cost actually appears - Batch GUI shows none; Cost Explorer + runtime math) and official-smoke-spec section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an external “batch executor” workflow (AWS Batch now, SLURM/local by template) where the server packages datapoints to a shared folder and later ingests completed results, and it adds a pure-Ruby Latin Hypercube Sampling (LHS) backend so LHS can run without R/Rserve in start_local-style deployments.
Changes:
- Add pure-Ruby sampling primitives (
Distributions) and an LHS sampler with a configurable backend switch (:rservevs:ruby), plus makeVariable.pivot_arrayusable without an R session. - Add server-side external batch packaging/ingestion and a new
external_batch_runanalysis method to wait/ingest until datapoints are terminal. - Add executor tooling under
external_batch/(runner, local executor, AWS Batch helper scripts + Terraform + docs) and expose--batch-run-methodinopenstudio_meta.
Reviewed changes
Copilot reviewed 30 out of 31 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| server/spec/support/external_batch_helpers.rb | Spec helpers for stub CLIs |
| server/spec/models/sampling_lhs_spec.rb | Ruby LHS backend specs |
| server/spec/models/sampling_distributions_spec.rb | Quantile/LHS math specs |
| server/spec/models/external_batch_spec.rb | External batch unit + pipeline specs |
| server/spec/models/external_batch_aws_spec.rb | Stubbed AWS-shaped pipeline spec |
| server/spec/features/aws_batch_smoke_spec.rb | Real-AWS gated smoke test |
| server/config/environments/local-test.rb | Default sampling backend for local-test |
| server/config/environments/local-server.rb | Default sampling backend for local-server |
| server/config/application.rb | Global sampling backend config default |
| server/app/models/variable.rb | Optional R session for pivot integer sequences |
| server/app/lib/external_batch/packager.rb | Server-side package builder |
| server/app/lib/external_batch/ingester.rb | Server-side results ingester |
| server/app/lib/external_batch.rb | External batch paths + schema version |
| server/app/lib/analysis_library/sampling/lhs.rb | Pure-Ruby LHS sampler |
| server/app/lib/analysis_library/sampling/distributions.rb | Pure-Ruby distribution functions |
| server/app/lib/analysis_library/lhs.rb | Backend switch (Rserve vs Ruby) |
| server/app/lib/analysis_library/external_batch_run.rb | External batch run loop + ingestion |
| external_batch/templates/kestrel_array.sbatch | SLURM/Apptainer array template |
| external_batch/runner/run_chunk.rb | Stdlib-only chunk runner |
| external_batch/README.md | External batch architecture + contracts |
| external_batch/local_executor.rb | Local/mock array executor |
| external_batch/aws/WALKTHROUGH.md | AWS setup walkthrough |
| external_batch/aws/task_wrapper.sh | AWS Batch task entrypoint |
| external_batch/aws/sync_results.rb | S3→local mirroring helper |
| external_batch/aws/submit_batch.rb | Package push + Batch submit helper |
| external_batch/aws/README.md | AWS executor documentation |
| external_batch/aws/infra/main.tf | AWS infra (S3/ECR/Batch/IAM) |
| external_batch/aws/infra/.gitignore | Terraform state ignore rules |
| external_batch/aws/Dockerfile | Runner image build |
| doc/aws_smoke_runbook.md | Real-AWS validation runbook |
| bin/openstudio_meta | Add --batch-run-method flag |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+74
to
+80
| ::Zip::File.open(zip_path) do |zf| | ||
| zf.each do |f| | ||
| f_path = File.join(analysis_pkg_dir, f.name) | ||
| FileUtils.mkdir_p(File.dirname(f_path)) | ||
| zf.extract(f, f_path) unless File.exist?(f_path) | ||
| end | ||
| end |
Contributor
Author
There was a problem hiding this comment.
Fixed in 334cc66 — each entry name is resolved with expand_path against the package dir and refused if it lands outside. Spec proves a crafted ../.. entry raises and nothing is written outside the package.
Comment on lines
+94
to
+106
| def data_point_json(dp) | ||
| h = dp.as_json | ||
| h['set_variable_values_names'] = {} | ||
| h['set_variable_values_display_names'] = {} | ||
| (h['set_variable_values'] || {}).each do |k, v| | ||
| var = Variable.where(_id: k).first | ||
| next unless var | ||
|
|
||
| h['set_variable_values_names'][var.name] = v | ||
| h['set_variable_values_display_names'][var.display_name] = v | ||
| end | ||
| { data_point: h } | ||
| end |
Contributor
Author
There was a problem hiding this comment.
Fixed in 48d6db9 — one memoized Variable.where(:_id.in => ...) map for the whole package instead of a query per entry per datapoint. Spec asserts a single query and the exact name→value mapping.
Comment on lines
+125
to
+138
| def attach(dp, path, type, display_name = nil) | ||
| return unless File.exist?(path) | ||
|
|
||
| display_name ||= File.basename(path, '.*') | ||
| file = File.open(path, 'rb') | ||
| begin | ||
| rf = ResultFile.new(display_name: display_name, type: type) | ||
| rf.attachment = file | ||
| dp.result_files << rf | ||
| dp.save! | ||
| ensure | ||
| file.close | ||
| end | ||
| end |
Contributor
Author
There was a problem hiding this comment.
Fixed in bd4bc12 — ResultFiles are now built in memory with a display_name/type de-dupe guard and persisted once by ingest_data_point's final dp.save!, so an ingest killed mid-pass can't leave partial or duplicated attachments. Spec simulates an interrupted pass and asserts re-ingest doesn't duplicate.
Comment on lines
+147
to
154
| def sampling_backend | ||
| override = (@analysis.problem || {}).dig('algorithm', 'sampling_backend') | ||
| return override.to_sym if override.present? | ||
|
|
||
| configured = Rails.application.config.x.sampling_backend | ||
| configured.present? ? configured.to_sym : :rserve | ||
| end | ||
| end |
Contributor
Author
There was a problem hiding this comment.
Fixed in 43ad4d9 — safe coercion plus an rserve/ruby allowlist. Went with a clear raise on unknown values rather than falling back to the configured default: an explicit-but-mistyped override silently switching backends seemed worse than failing loudly. Resolution also moved inside perform's rescue so a bad override marks the analysis errored instead of crashing the delayed job.
| timeout_s = 28_800 unless timeout_s.positive? | ||
|
|
||
| begin | ||
| pid = Process.spawn(oscli_env_unset, cmd, [:err, :out] => [process_log, 'w']) |
Contributor
Author
Comment on lines
+128
to
+138
| def kill_process_tree(pid, runner_log) | ||
| if Gem.win_platform? | ||
| system("taskkill /pid #{pid} /f /T >NUL 2>&1") | ||
| else | ||
| child_pid = `ps -o pid= --ppid "#{pid}"`.to_i | ||
| Process.kill('KILL', child_pid) if child_pid > 0 | ||
| Process.kill('KILL', pid) | ||
| end | ||
| rescue StandardError => e | ||
| runner_log.puts "Error killing process #{pid}: #{e.message}" | ||
| end |
Contributor
Author
There was a problem hiding this comment.
Fixed in 38b48fc — Process.kill('KILL', -pid) on the process group, with a single-pid fallback; no ps dependency. POSIX-only spec drives run_chunk.rb with a stub CLI whose heartbeating grandchild must die with the timeout (verified red/green on Linux in Docker).
A crafted seed zip entry name (../..) could escape the package directory and overwrite arbitrary files. Resolve each entry against the package dir and refuse entries that land outside it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
data_point_json issued one Variable query per set_variable_values entry per datapoint (N+1); large analyses paid thousands of round trips during packaging. Build one id->Variable map for the whole package instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
attach saved the DataPoint per attached file, so an ingest killed mid-pass left partial attachments that the next pass duplicated (dp never reached status completed). Build embedded ResultFiles in memory, skip ones an earlier pass already persisted, and rely on ingest_data_point's final dp.save! to persist everything together. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Link to DEVELOPER_GUIDE.md for technical details - Include visual aids for workflows and pipelines - Add getting started section for new developers - Provide versioning information for OpenStudio Server 3.11.0+ - Expand FAQ section with common issues and solutions - Include expanded code examples for implementation tasks
override.to_sym raised NoMethodError on non-string OSA values (numeric, boolean), and any typo silently selected the Ruby backend. Coerce safely, validate against rserve/ruby with a clear error, and resolve the backend inside perform's rescue so a bad override marks the analysis errored instead of crashing the delayed job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OSCLI command string goes through a shell and OSCLI spawns EnergyPlus children, so a timeout kill of the spawned pid alone leaves the tree running. Spawn OSCLI and the initialize/finalize scripts with pgroup: true on POSIX and route the script timeout through kill_process_tree; the group kill itself lands in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kill_process_tree shelled out to ps and killed only the first child, so grandchildren (EnergyPlus under OSCLI under the shell) survived a timeout; ps also isn't present in minimal images. Signal the process group instead (children are spawned with pgroup: true), falling back to the single pid. POSIX-only spec drives run_chunk.rb with a stub CLI whose heartbeating grandchild must die with the timeout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to external_batch
- feature introduced 3.11.0 not 3.8.0; ruby 3.2 not 2.5+ - spec filenames: external_batch_local_spec.rb doesn't exist; core coverage is external_batch_spec.rb (real Mongo, stub CLI, nothing mocked) - package layout: results/ is sibling of package/ under batch_dir, not nested - download-flag conditionals live in runner, not ingester attachment - no configurable timeout for missing dps: done-marker driven; re-submit dead chunk - runner is stdlib ruby; OpenStudio CLI version is what must match - drop content-free executor selection flowchart + README 'Enhanced with' blurb Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to external_batch
Limitations back to one-liners, keeping the two new facts (why urbanopt/gemfile are unsupported; shared-filesystem executors like a Nomad job need no transport step). Deployment planning content belongs in an issue/design doc, not the feature README — see PR discussion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
@anchapin heads-up on two docs commits I pushed: db61e11 trims the Nomad MVP sections from
f7f6f4a fixed factual errors in The Nomad suitability analysis itself looks right to me — shared storage + the local_executor pattern is exactly how I'd run it. Just want the README to stay contract-only. |
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR lets an OpenStudio Server analysis run its simulations on an external
executor — AWS Batch today, SLURM/Apptainer (Kestrel) by template, or a local
process pool for dev/CI — instead of the server's own simulation workers. It
also adds a pure-Ruby LHS sampling backend, so LHS analyses work on
openstudio_meta start_localdeployments (mongod + Rails + delayed_job) withno Docker, no Redis, and no Rserve.
The design is deliberately dumb: the server and the executor never talk to each
other. They share a folder —
package/(server writes once: extracted analysiszip,
manifest.json, and one pre-translateddata_point.oswperdatapoint) and
results/(executor writes one folder per finished datapoint;the server ingests incrementally). Remote execution is therefore only ever a
file-transport problem (
aws s3 sync, rsync, or a shared mount).What's included
Sampling (no behavior change for existing deployments)
AnalysisLibrary::Sampling::{Distributions,Lhs}— pure-Ruby LHS:randomLHSstrata, qnorm (Acklam + Halley refinement), qlnorm/qunif/qtriangle/qdiscrete,
replicating the R backend's formulas verbatim (including its lognormal
parameterization) so the two backends produce statistically equivalent designs.
AnalysisLibrary::Lhsgains a backend switch viaconfig.x.sampling_backend::rserveremains the default everywhere Docker/R exists;local-serverandlocal-testdefault to:ruby; per-analysis override viaproblem.algorithm.sampling_backend. The R code path is untouched.Variable.pivot_arrayno longer requires an R session for integer-sequencepivots (R session argument now optional; R path unchanged when provided).
External batch core (server)
AnalysisLibrary::ExternalBatchRun— sibling ofbatch_run: packages thenadatapoints, marks them queued, then incrementally ingests results untilevery datapoint is terminal (missing datapoints are errored once all chunk
done-markers arrive). Stop-analysis semantics match
batch_run.ExternalBatch::Packager— package layout mirrors the worker's analysisdirectory so the OSWs' relative
../measures,../weather,../seedspathsresolve unchanged; OSW translation is the same
Translator::Workflowcall theworker makes, moved to package time.
ExternalBatch::Ingester— server-side, idempotent; applies the sameDataPoint mutations
DjJobs::RunSimulateDataPointperforms (results,truncated
sdp_log_file, ResultFiles with identical display names/types,worker_logs, status flags, timestamps). Only ingests result folders that
contain
status.json, which the runner writes last — partial transfers areinvisible, retries are safe.
Executors (repo-root
external_batch/)runner/run_chunk.rb— plain-Ruby-stdlib runner, no server/DB access; chunkindex from
--chunk/SLURM_ARRAY_TASK_ID/AWS_BATCH_JOB_ARRAY_INDEX.One file runs identically under the local executor, Apptainer, or AWS Batch.
local_executor.rb— the dev/CI mock of an array job (one runner process perchunk,
--parallel N).aws/— Dockerfile + task wrapper (image = nrel/openstudio + AWS CLI),submit_batch.rb(S3 push + array-job submit),sync_results.rb(mirrorloop feeding the ingester), Terraform for one-time infra (S3 + lifecycle,
ECR, on-demand BEST_FIT_PROGRESSIVE compute env on NVMe families, job queue,
1 vCPU/7.5 GB job definition, least-privilege 3-role IAM — containers use a
task role scoped to the run bucket; no credentials in images or jobs).
templates/kestrel_array.sbatch— SLURM/Apptainer reference (follow-up work).openstudio_meta run_analysisgains an additive--batch-run-methodflag(default
batch_run, unchanged).Testing
lhsperform end-to-end;stub OpenStudio CLI;
a stub
awsCLI that emulates S3 as a local folder.spec/features/aws_batch_smoke_spec.rb— env-gated (AWS_BATCH_SMOKE_*),skips in CI; runs the docker-stack algo-spec SEB discrete LHS calibration
project on the user's own AWS Batch infra and asserts the calibration
CVRMSE/NMBE metrics match that spec's golden values.
EnergyPlus calibration runs, results round-tripped via S3 and ingested;
metrics matched the docker/R CI golden rows exactly — the external path is
value-identical to the classic worker path.
spec/modelssuite green (67 examples); the onespec/requestsfailurepre-exists on
develop(admin page requiresOPENSTUDIO_EXE_PATH).Not in scope / follow-ups
gemfile(packager raises).external path; re-ingest appends result files).
execution, unchanged.
Docs
external_batch/README.md— architecture, contracts, executor matrix.external_batch/aws/README.md— AWS setup/usage/failure handling.external_batch/aws/WALKTHROUGH.md— first-time user guide verified againsta real account: installs, access keys, console navigation and monitoring,
billing alerts and cost visibility, troubleshooting, cleanup.
doc/aws_smoke_runbook.md— sanitized command-by-command record of thereal-AWS validation runs.
🤖 Generated with Claude Code