Memory leak in ELPA diagonalization

[LiuXiaohui123321]

Describe the bug

When using ABACUS to do structure relaxation calculation, I find that the amount of the memory used will gradually increase with the calculation steps. I use the top command to dynamically view the amount of memory used, for example,
Moment 1:

Moment 2:

I also use Grafana to give a graph of the results of the amount of memory used over time,

To determine whether the memory leak is coming from scf or relaxation, I do another somewhat "extreme" test,

scf_thr 1e-20
scf_nmax 10000

and also use the top command and Grafana
Moment 1:

Moment 2:

It appears that the memory leak is at least in the scf process. And the relaxation process needs more tests.

Expected behavior

No response

To Reproduce

memLeak.tar.gz

1. ABACUS: version 3.2.2
2. Build: intel-2019.update5, intelmpi-2019.update5, mkl-2019.update5, gcc-9.2.0; cmake
3. Run: mpirun -n 40 abacus

Environment

• gcc-9.2.0
• intel-2019.update5
• intelmpi-2019.update5
• mkl-2019.update5

Additional Context

No response

Task list for Issue attackers

• Verify the issue is not a duplicate.
• Describe the bug.
• Steps to reproduce.
• Expected behavior.
• Error message.
• Environment details.
• Additional context.
• Assign a priority level (low, medium, high, urgent).
• Assign the issue to a team member.
• Label the issue with relevant tags.
• Identify possible related issues.
• Create a unit test or automated test to reproduce the bug (if applicable).
• Fix the bug.
• Test the fix.
• Update documentation (if necessary).
• Close the issue and inform the reporter (if applicable).

[hongriTianqi]

@LiuXiaohui123321 We have an issue of possible scf process memory leak in #2935 . Would please help test that one?

[LiuXiaohui123321]

@LiuXiaohui123321 We have an issue of possible scf process memory leak in #2935 . Would please help test that one?

@hongriTianqi I have tested the example in issue #2935, and have attached the results below that issue.

[hongriTianqi]

This issue has been solved thanks to insights from @dyzheng. When we feel nowhere to solve the issue, @dyzheng realized it might be caused be MKL inconsistency between libraries.

[LiuXiaohui123321]

I noticed that the memory leak bug has been fixed by PR #3472 in the release version 3.5.1. Then I re-tested the example here, but it looks like the bug is not completely resolved and the memory leak is still there.

The test environment used is as follows:

1. elpa-2021.05.002/2021.11.002

• module purge
• module load intel/2019.update5 intelmpi/2019.update5 mkl/2019.update5 gcc/9.2.0
• FC=mpiifort CC=mpiicc ./configure --prefix=... SCALAPACK_LDFLAGS="-L$MKLROOT/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm -Wl,-rpath,$MKLROOT/lib/intel64" SCALAPACK_FCFLAGS="-L$MKLROOT/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm -I$MKLROOT/include/intel64/lp64"

2. cereal-1.3.0

• tar zxvf cereal-1.3.0.tar.gz -C ./

3. abacus-develop-3.5.1

• module purge
• module load cmake/3.19.0 intel/2019.update5 intelmpi/2019.update5 mkl/2019.update5 gcc/9.2.0
• CC=icc CXX=icpc cmake -B build -DCereal_INCLUDE_DIR=.../cereal-1.3.0/include -DELPA_LIBRARY=.../lib/libelpa.so -DELPA_INCLUDE_DIR=.../include/elpa-2021.xx.002

[dyzheng]

Thanks for your update, we will retest the case with v3.5.1.

[LiuXiaohui123321]

I noticed that the memory leak bug has been fixed by PR #3472 in the release version 3.5.1. Then I re-tested the example here, but it looks like the bug is not completely resolved and the memory leak is still there.

The test environment used is as follows:

1. elpa-2021.05.002/2021.11.002

• module purge
• module load intel/2019.update5 intelmpi/2019.update5 mkl/2019.update5 gcc/9.2.0
• FC=mpiifort CC=mpiicc ./configure --prefix=... SCALAPACK_LDFLAGS="-L$MKLROOT/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm -Wl,-rpath,$MKLROOT/lib/intel64" SCALAPACK_FCFLAGS="-L$MKLROOT/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm -I$MKLROOT/include/intel64/lp64"

2. cereal-1.3.0

• tar zxvf cereal-1.3.0.tar.gz -C ./

3. abacus-develop-3.5.1

• module purge
• module load cmake/3.19.0 intel/2019.update5 intelmpi/2019.update5 mkl/2019.update5 gcc/9.2.0
• CC=icc CXX=icpc cmake -B build -DCereal_INCLUDE_DIR=.../cereal-1.3.0/include -DELPA_LIBRARY=.../lib/libelpa.so -DELPA_INCLUDE_DIR=.../include/elpa-2021.xx.002

Here are the input files,
2960_input-files.tar.gz

The function DiagoElpa::diag() in the iteration is used to detect the memory usage,

==> DiagoElpa::diag 152.761 GB 24.008 s
==> DiagoElpa::diag 148.648582458 GB 52.2605142593 s
==> DiagoElpa::diag 148.210006714 GB 78.6172110289 s
==> DiagoElpa::diag 147.836853027 GB 104.926947951 s
==> DiagoElpa::diag 147.556583405 GB 131.186710715 s
==> DiagoElpa::diag 147.277137756 GB 157.465421528 s
==> DiagoElpa::diag 146.908161163 GB 183.724356905 s
==> DiagoElpa::diag 146.597167969 GB 210.000731617 s
==> DiagoElpa::diag 146.396087646 GB 236.282636881 s
==> DiagoElpa::diag 146.320915222 GB 262.553348139 s
==> DiagoElpa::diag 146.269115448 GB 288.850581214 s
==> DiagoElpa::diag 146.199905396 GB 315.123554945 s
==> DiagoElpa::diag 146.132167816 GB 341.371098965 s
==> DiagoElpa::diag 146.083370209 GB 367.664950594 s
==> DiagoElpa::diag 146.016197205 GB 393.920230776 s
...
==> DiagoElpa::diag 118.571239471 GB 11613.5364821 s
==> DiagoElpa::diag 118.489513397 GB 11639.7941203 s
==> DiagoElpa::diag 118.408836365 GB 11666.0673717 s
==> DiagoElpa::diag 118.355426788 GB 11692.3510467 s
==> DiagoElpa::diag 118.298965454 GB 11718.6152955 s
==> DiagoElpa::diag 118.220375061 GB 11744.8890342 s
==> DiagoElpa::diag 118.143379211 GB 11771.1769129 s
==> DiagoElpa::diag 118.080581665 GB 11797.4658343 s
==> DiagoElpa::diag 118.068721771 GB 11823.7530197 s
==> DiagoElpa::diag 118.022708893 GB 11850.0350929 s
==> DiagoElpa::diag 117.957267761 GB 11876.3323782 s
==> DiagoElpa::diag 117.887569427 GB 11902.6097547 s
==> DiagoElpa::diag 117.837528229 GB 11928.8735333 s
==> DiagoElpa::diag 117.771236420 GB 11955.1298046 s
==> DiagoElpa::diag 117.696727753 GB 11981.4391402 s
==> DiagoElpa::diag 117.612930298 GB 12007.7014688 s

And the output of top command, where the red boxes indicate that memory has been increasing for these processes

[LiuXiaohui123321]

This bug was probably introduced in release version 2.3.5, corresponding to a commit record of #1213.

Currently it is traced to the following code, and more tests needed,

abacus-develop/source/module_hsolver/genelpa/elpa_new_real.cpp

Line 16 in 3c38596

int ELPA_Solver::eigenvector(double* A, double* EigenValue, double* EigenVector)

[WHUweiqingzhou]

@LiuXiaohui123321,
Is the memory leak still there for ABACUS v3.5.2?