2026-03-05 Engineering Team

[joecastiglione]

Agenda

• Merge Explicit Error Terms
• Status of remaining PRs from prior tasks
• Memory usage

[dhensle]

Would like to add as an agenda item the following memory profile produced by TransLink running their full sample with sharrow on. TransLink developed a script to view the total system memory of the machine. Shows similar behavior as reported in ActivitySim/activitysim#850

cc @bwentl

[jpn--]

@dhensle @bwentl

Would it be possible to replicate the memory trace run reported above, ideally on the same machine and same model, but with the number of multiprocess processes cut in half? Thanks

[jpn--]

ActivitySim Engineering Meeting Notes

Meeting Overview

Agenda

• Testing documentation check-in and timeline
• Explicit Error Terms (EET) pull request status and merge strategy
• Memory usage and Sharrow performance profiling
• Open issues and Phase 11B/11C status

Participants

• Joe Castiglione (@joecastiglione) — Zephyr Foundation
• Jeff Newman (@jpn--) — Driftless
• David Hensle (@dhensle) — RSG
• Jan (@janzill) — Outer Loop
• Sijia Wang (@i-am-sijia) — WSP
• Feng Xie (@fxie-mwcog) — MWCOG

Duration

Approximately 45 minutes

---

Executive Summary

The team discussed the merge strategy for the Explicit Error Terms (EET) feature branch, agreeing to defer merging to the main branch until tests and documentation are completed, expected within a four-week window. The PR for skipping failed households was reviewed and the group had several proponents of the idea that the new skip-households feature should default to off, with the final call deferred to the product team. A significant Sharrow memory leak was identified in multi-process runs, with Jeff Newman taking ownership of investigating the root cause. The team also briefly addressed open issues and chunking for simple simulate models, with plans to close out remaining Phase 11B items in upcoming weeks.

---

Meeting Notes

Testing Documentation

The group discussed pacing check-ins on the testing documentation. David raised the concern that some closure on the testing approach is needed before Phase 11 work begins, as it affects how task orders are scoped. Jeff proposed agendizing a more concrete proposal for the next meeting, where David will present a draft component test for the park-and-ride scenario as a worked example of the new testing scheme. The group agreed not to slow other work while awaiting the new testing framework — work completed before the new regime is adopted will follow the old approach, and the new rules will apply going forward.

Explicit Error Terms (EET) Feature Branch

Jeff reported that after reviewing the EET pull request, he estimates he could complete the merge in about a week. However, Jan raised concerns about merging to the main branch before tests are written and a few known inconsistencies are resolved. Jan noted that some functions in the ET implementation are incomplete, and that test coverage — given how broadly the feature touches the codebase — would take approximately four weeks to establish.

The group agreed on the following approach:

• Jeff will sync the EET feature branch with the latest main branch changes (without merging to main).
• Jan and Outer Loop will use the four-week task order timeline to write tests, address known issues, and add documentation.
• After four weeks of progress, the branch will be in a position to merge to main as a feature that is off by default for users who don't opt in.

Skip Failed Households Default Setting

David reviewed the PR for this and flagged one substantive decision: whether the new skip-households feature should be on or off by default. David argued it should be off by default, since it changes how ActivitySim operates relative to its historical behavior, and agencies upgrading should not encounter unexpected changes without explicitly opting in.

Sijia joined mid-discussion and offered a nuanced counterpoint — that turning the feature on could helpfully surface previously masked household failures, and that knowing the scale of a problem is valuable. Jeff acknowledged both perspectives but noted this is a product decision rather than an engineering one, and agreed to bring it to the product team (Bhargava Sana's team) for a final call. In the meantime, the consensus in the room leaned toward off by default for consistency with existing behavior.

Sharrow Memory Usage and Performance Profiling

David shared memory profiling results contributed by Bo Wen (TransLink), who developed a system-level memory monitoring approach that captures full-sample, multi-process runs — an improvement over the existing per-process profiling method.

The profile revealed a significant and sustained memory increase during multi-process Sharrow runs, with memory not recovering between model steps as expected. Jeff analyzed the pattern and concluded this is consistent with a memory leak in Numba code within Sharrow, rather than a true unbounded leak — the memory is recovered by the OS when sub-processes terminate, which explains why it drops at the end of the run.

Jeff noted that each model step should add only a modest amount of memory (reflecting new columns added to the state), but the observed profile shows far more accumulation. He will investigate the root cause. Jan offered to consult a colleague with relevant Numba expertise for a second opinion.

A short-term workaround was discussed — subdividing the multi-process household group into multiple segments so the OS reclaims leaked memory between segments — but Feng Xie indicated that MWCOG would prefer to wait for a proper fix rather than implement a temporary workaround. Jeff agreed a real fix is preferable.

Chunking for Simple Simulate Models

David flagged a GitHub issue on explicit chunking support for simple simulate models. The chunking logic exists internally but is not exposed to users via settings. Jeff will add his thoughts to the issue; the preliminary expectation is that this is a low-priority addition — the memory footprint from simple simulate is typically small — but a PR would be accepted if someone wants to wire it up.

Open Issues and Phase 11B Wrap-Up

Jeff noted he has a significant number of hours allocated under general engineering and support for closing out open issues, and plans to work through them in the coming months. Joe clarified there is no hard budget deadline of June 30th for spending these hours — they can carry over to support future work if needed.

---

Action Items

• Jeff Newman: Present a concrete component test proposal for the park-and-ride scenario at next week's meeting.
• Jeff Newman: Sync the EET feature branch with the latest main branch changes and prepare it for eventual merge (without merging to main yet).
• Jan Zill: As tasked: write tests, fix known EET inconsistencies, and add documentation over the next four weeks, targeting readiness to merge to main.
• Jeff Newman: Message Bhargava Sana to flag the skip-households default setting (on vs. off) as a product team decision for the AD Global Options PR.
• Sijia Wang: Review David's comments on the skipping failures PR and address the minor feedback items.
• Jeff Newman: Investigate the Sharrow memory leak in Numba code; consider bringing in a second opinion via Jan's colleague.
• Jan Zill: Check with a Numba-experienced colleague about availability to review the Sharrow memory issue.
• Jeff Newman: Comment on the open GitHub issue regarding chunking for simple simulate models.
• Jeff Newman: Continue working through open issues allocated under general engineering and support hours.

[bwentl]

@dhensle @bwentl

Would it be possible to replicate the memory trace run reported above, ideally on the same machine and same model, but with the number of multiprocess processes cut in half? Thanks

@jpn-- - I am happy to do another profile with fewer mp processes (4 instead of our current 8) next week.

Is there an existing issue on this that I can share my result on? If not, shall I create one?

[jpn--]

Discussion on sharrow memory usage can be moved to ActivitySim/activitysim#1053