possibility of memory leak : job is killed by SIGNAL 9 after several scf steps.

[QG-phy]

Describe the bug

when running abacus jobs. I found some of my jobs were killed by SIGNAL 9 after many SCF steps.

The error message goes like this:

Therefore, I am wondering maybe there is a possibility of a memory leak issue.

Expected behavior

fig it out and solve it maybe.

To Reproduce

InSb.tar.gz

Environment

env and image:

registry.dp.tech/deepmodeling/abacus-intel:latest

machine:

"bohrium": {
"scass_type": "c32_m128_cpu",
"job_type": "container",
"platform": "ali"
},

command:

#!/bin/bash
source /opt/intel/oneapi/setvars.sh
export OMP_NUM_THREADS=1
cp ./scf/* ./
mpirun -np 16 abacus

Additional Context

No response

Task list for Issue attackers

• Verify the issue is not a duplicate.
• Describe the bug.
• Steps to reproduce.
• Expected behavior.
• Error message.
• Environment details.
• Additional context.
• Assign a priority level (low, medium, high, urgent).
• Assign the issue to a team member.
• Label the issue with relevant tags.
• Identify possible related issues.
• Create a unit test or automated test to reproduce the bug (if applicable).
• Fix the bug.
• Test the fix.
• Update documentation (if necessary).
• Close the issue and inform the reporter (if applicable).

[hongriTianqi]

OK, we need to detect whether the memory increase is normal or abnormal

[hongriTianqi]

The gnu machine is ok with the same type of machine.
ABACUS v3.2

           Atomic-orbital Based Ab-initio Computation at UStc                    

                 Website: http://abacus.ustc.edu.cn/                             
           Documentation: https://abacus.deepmodeling.com/                       
              Repository: https://github.com/abacusmodeling/abacus-develop       
                          https://github.com/deepmodeling/abacus-develop         

Tue Oct 31 12:41:43 2023
MAKE THE DIR : OUT.ABACUS/

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Warning: the number of valence electrons in pseudopotential > 3 for In: [Kr] 4d10 5s2 5p1
Warning: the number of valence electrons in pseudopotential > 5 for Sb: [Kr] 4d10 5s2 5p3
Pseudopotentials with additional electrons can yield (more) accurate outcomes, but may be less efficient.
If you're confident that your chosen pseudopotential is appropriate, you can safely ignore this warning.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

UNIFORM GRID DIM : 180 * 200 * 180
UNIFORM GRID DIM(BIG): 45 * 40 * 45
DONE(1.39257 SEC) : SETUP UNITCELL
DONE(1.51377 SEC) : SYMMETRY
DONE(1.67186 SEC) : INIT K-POINTS

Self-consistent calculations for electrons

SPIN KPOINTS PROCESSORS NBASE
1 36 16 2400

Use Systematically Improvable Atomic bases

ELEMENT ORBITALS NBASE NATOM XC
In 2s2p2d1f-7au 25 48
Sb 2s2p2d1f-7au 25 48

Initial plane wave basis and FFT box

---

SELF-CONSISTENT :

START CHARGE : atomic
DONE(30.473 SEC) : INIT SCF
ITER ETOT(eV) EDIFF(eV) DRHO TIME(s)
GE1 -1.639180e+05 0.000000e+00 3.989e-02 1.705e+02
GE2 -1.639183e+05 -3.836051e-01 1.641e-02 1.660e+02
GE3 -1.638702e+05 4.810953e+01 2.314e-02 1.662e+02
GE4 -1.638471e+05 2.315318e+01 4.421e-02 1.674e+02
GE5 -1.639208e+05 -7.370479e+01 1.282e-02 1.667e+02
GE6 -1.639277e+05 -6.927459e+00 8.503e-03 1.662e+02
GE7 -1.639287e+05 -9.728276e-01 4.888e-03 1.671e+02
GE8 -1.639284e+05 2.916989e-01 3.985e-03 1.669e+02
GE9 -1.639284e+05 2.072873e-02 3.086e-03 1.671e+02
GE10 -1.639285e+05 -9.234770e-02 2.066e-03 1.677e+02
GE11 -1.639284e+05 4.144906e-02 1.235e-03 1.674e+02
GE12 -1.639284e+05 -2.950202e-02 7.407e-04 1.667e+02
GE13 -1.639284e+05 8.972636e-03 8.118e-04 1.666e+02
GE14 -1.639285e+05 -1.773380e-02 4.452e-04 1.667e+02
GE15 -1.639284e+05 7.928516e-03 7.306e-04 1.672e+02
GE16 -1.639285e+05 -1.785576e-02 3.037e-04 1.669e+02
GE17 -1.639285e+05 -2.880893e-03 1.031e-04 1.665e+02
GE18 -1.639285e+05 -3.737139e-04 5.887e-05 1.667e+02
GE19 -1.639285e+05 -1.059564e-04 2.553e-05 1.664e+02
GE20 -1.639285e+05 -2.129738e-05 1.545e-05 1.665e+02
GE21 -1.639285e+05 -7.530288e-06 1.370e-05 1.665e+02
GE22 -1.639285e+05 -5.803601e-06 3.086e-06 1.667e+02
GE23 -1.639285e+05 5.566955e-07 5.313e-06 1.663e+02
GE24 -1.639285e+05 -1.096116e-06 1.022e-06 1.664e+02
GE25 -1.639285e+05 -3.274242e-08 1.269e-06 1.665e+02
GE26 -1.639285e+05 4.521573e-08 8.713e-07 1.668e+02
GE27 -1.639285e+05 -7.758693e-08 6.493e-07 1.664e+02
GE28 -1.639285e+05 -1.116163e-08 2.068e-07 1.665e+02
GE29 -1.639285e+05 2.029387e-09 2.055e-07 1.669e+02
GE30 -1.639285e+05 5.989166e-09 8.294e-08 2.711e+02

START Time : Tue Oct 31 12:41:43 2023
FINISH Time : Tue Oct 31 14:07:28 2023
TOTAL Time : 5145
SEE INFORMATION IN : OUT.ABACUS/

[hongriTianqi]

This case is OK with the command of OMP_NUM_THREADS=16 mpirun -np 1 abacus:
GE17 -1.639285e+05 -2.880899e-03 1.031e-04 6.460e+01
GE18 -1.639285e+05 -3.737170e-04 5.887e-05 6.495e+01
GE19 -1.639285e+05 -1.059542e-04 2.553e-05 6.468e+01
GE20 -1.639285e+05 -2.129832e-05 1.545e-05 6.459e+01
GE21 -1.639285e+05 -7.527837e-06 1.370e-05 6.459e+01
GE22 -1.639285e+05 -5.803205e-06 3.086e-06 6.465e+01
GE23 -1.639285e+05 5.520180e-07 5.313e-06 6.484e+01
GE24 -1.639285e+05 -1.096834e-06 1.022e-06 6.489e+01
GE25 -1.639285e+05 -3.348488e-08 1.269e-06 6.491e+01
GE26 -1.639285e+05 4.462176e-08 8.713e-07 6.488e+01
GE27 -1.639285e+05 -7.974006e-08 6.493e-07 6.483e+01
GE28 -1.639285e+05 7.993804e-09 2.068e-07 6.481e+01
GE29 -1.639285e+05 -1.017168e-08 2.055e-07 6.525e+01
GE30 -1.639285e+05 1.056766e-08 8.294e-08 1.408e+02
START Time : Tue Oct 31 15:44:31 2023
FINISH Time : Tue Oct 31 16:18:22 2023

[caic99]

Hi @hongriTianqi ,
No memory leak at the end means all allocated is freed finally. However, this does not mean that the memory required for a single scf step is freed after this step. If that happenes, all memory is freed by the end, rather than at the end of each scf step.
I would suggest monitoring if the memory growth as the scf step accumulates. Before that, please make sure this signal is sent by OOM killer.

[hongriTianqi]

@caic99 Thx, you are right.

[hongriTianqi]

The memory exceeded 124 G at the 16 scf step as I tested on an intel machine with the command OMP_NUM_THREADS=1 mpirun -np 16 abacus. 124 G is the maximum memory on that machine as indicated by the htop command.

[hongriTianqi]

This is not a HSE example, it is a PBE example.

[caic99]

Here is the latest address sanitizer report on memory leak.