Skip to content

Fix: Prevent MPI communicator growth in LCAO matrix output - #7676

Merged
mohanchen merged 1 commit into
deepmodeling:developfrom
AsTonyshment:fix/MPI_comm_leak
Jul 23, 2026
Merged

Fix: Prevent MPI communicator growth in LCAO matrix output#7676
mohanchen merged 1 commit into
deepmodeling:developfrom
AsTonyshment:fix/MPI_comm_leak

Conversation

@AsTonyshment

Copy link
Copy Markdown
Collaborator

What's changed?

This bug was discovered by @ESROAMER when an LCAO RT-TDDFT calculation failed at approximately electronic step 2370 with:

Too many communicators (0/32768 free on this process; ignore_id=0)

LCAO matrix-output code created a temporary BLACS process grid for each serial gather and then immediately called set_serial(). set_serial() overwrote the BLACS context with -1, so the temporary context could no longer be released. Repeated DMR or H/S(R) output during RT-TDDFT therefore consumed MPI communicator contexts until MPI failed.

The affected output paths now construct their temporary Parallel_Orbitals distributions directly with set_serial(). This is sufficient for gatherParallels(), which performs the data transfer independently of a BLACS context. The same redundant pattern was removed from DMR, H/S(R), derivative-H, decomposed-H, DeePKS H(R), and asynchronous-overlap output paths.

There is no change to matrix values, output formats, filenames, output frequency semantics, numerical solver behavior, or INPUT parameters.

Unit Tests and/or Case Tests for my changes

image

The figure is plotted directly from the PMPI logs. In two-rank, nine-step RT-TDDFT runs, DMR and H/S(R) output previously added 4 and 8 live communicators per electronic step, respectively, or 12 when combined. After the fix, all output cases match the control and finish with created=41, freed=41, live=0 on each rank.

The generated DMR and H/S(R) CSR files are byte-for-byte unchanged.

Governance Notes

  • INPUT/docs changes: None.
  • Core module impact: The numerical ESolver, HSolver, ElecState, Hamilt, and Psi paths are unchanged. The only operator-side change is in the optional asynchronous-overlap CSR output path.
  • Exceptions requested: None.

This comment was marked as spam.

@mohanchen mohanchen added Bugs Bugs that only solvable with sufficient knowledge of DFT Refactor Refactor ABACUS codes labels Jul 23, 2026

@mohanchen mohanchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice fix!

@mohanchen
mohanchen merged commit f26d766 into deepmodeling:develop Jul 23, 2026
17 checks passed
@AsTonyshment
AsTonyshment deleted the fix/MPI_comm_leak branch July 24, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bugs Bugs that only solvable with sufficient knowledge of DFT Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants