Keep DataPoint under BSON cap; terminal state on worker failure (fixes #851)

[brianlball]

Fixes #851.

Problem

A datapoint whose simulation SUCCEEDS can still be lost: an oversized artifact (observed: generic_qaqc summary_report, ~19MB inlined via results) pushes the DataPoint document over mongo's 16MB BSON cap. The save raises, every later save of the now-dirty document re-raises, the Resque wrapper's catch-all rescue only logs, and the dp is left status: started forever. R-driven algorithms (morris/sobol/nsga2) then poll to timeout, fill the design with 1e+18 penalties, and crash in result assembly — one oversized measure report kills the whole analysis with no failure recorded.

Full forensics in #851 (docker 3.11.0 images, 27-zone model, morris r=6).

Changes

• prune_oversized_results (dj job): when measure_attributes.json exceeds MAX_INLINE_RESULTS_BYTES (5MB), per-measure results larger than MAX_INLINE_RESULTS_MEASURE_BYTES (1MB) are replaced with a stub. Small results — objective values the algorithms need — are preserved. Complements the existing MAX_INLINE_SDP_LOG_* truncation.
• dj rescue path: reload the DataPoint before terminal-state saves so a dirty oversized document can't re-raise the same persistence error out of the rescue.
• Resque wrapper: on unhandled exception, set error flag + complete state (after reload) so the dp reaches a terminal state (completed / datapoint failure) instead of zombie-ing as started.

Testing

• ruby -c on both files.
• Failure mode reproduced end-to-end on the 3.11.0 docker stack before the fix (see Oversized datapoint artifacts (e.g. 19MB generic_qaqc log) exceed BSON 16MB cap -> dp zombies as 'started', R algorithms crash; successful sims lost #851); with oversized results stubbed the same workflow completes (verified by disabling generic_qaqc checks client-side — equivalent payload reduction).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens datapoint execution against MongoDB’s 16MB BSON document limit and ensures datapoints reliably reach a terminal state when workers fail, preventing “zombie” datapoints that remain started indefinitely and destabilize downstream R-driven analyses.

Changes:

• Add terminal-state handling in the Resque wrapper on unhandled exceptions (reload + error + complete) to avoid leaving datapoints stuck in started.
• Prune oversized inlined results (from measure_attributes.json) by stubbing per-measure payloads above a configured threshold when the overall results file is large.
• Reload the DataPoint in the DJ rescue path before terminal-state saves so a dirty/oversized in-memory document doesn’t cause repeated persistence failures.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
server/app/jobs/resque_jobs/run_simulate_data_point.rb	Ensures unhandled Resque worker exceptions attempt to put the DataPoint into a terminal failure state after reload.
server/app/jobs/dj_jobs/run_simulate_data_point.rb	Adds oversized-results pruning before saving results, and reloads the DataPoint before terminal-state saves in the rescue path.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[brianlball]

Addressed: rebased onto develop with #853, which guards both rescue paths — a nil d now logs via Rails.logger with the datapoint id and re-raises the original error before these log calls, so the terminal-state block only runs with d bound (safe navigation dropped accordingly).