Bug for magnetic system calculation compiled from ABACUS by icx-toolchain

[QuantumMisaka]

Describe the bug

When running relax job using ABACUS by icx-mkl toolchain, the gradient calculation part seemed to have problem, leading to the calculated force up to a strange value 6.0e+3, and the calculated stress was also blast

# in ABACUS by icx-mkl toolchain
><><><><><><><><><><><><><><><><><><><><><><
TOTAL-STRESS (KBAR):
><><><><><><><><><><><><><><><><><><><><><><
3.689e+05      -6.754e-01     2.270e-01      
-6.754e-01     3.689e+05      2.184e-01      
2.270e-01      2.184e-01      3.689e+05      
TOTAL-PRESSURE: 3.689e+05 KBAR
ETOT DIFF (eV)       : 0.000e+00
LARGEST GRAD (eV/A)  : 6.483e+03
BFGS TRUST (Bohr)    : 5.000e-01
-------------------------------------------
STEP OF ION RELAXATION : 2
-------------------------------------------

After change toolchain to icc-mkl, the force calculation result was normal and relax can converged

# in ABACUS by `icc-mkl` toolchain
 ><><><><><><><><><><><><><><><><><><><><><><
 TOTAL-STRESS (KBAR):
 ><><><><><><><><><><><><><><><><><><><><><><
 -3.510e+02     -6.058e-01     5.005e-01      
 -6.058e-01     -3.467e+02     1.103e+00      
 5.005e-01      1.103e+00      -3.293e+02     
 TOTAL-PRESSURE: -3.423e+02 KBAR
 ETOT DIFF (eV)       : 0.000e+00
 LARGEST GRAD (eV/A)  : 1.894e+00
 BFGS TRUST (Bohr)    : 5.000e-01
 -------------------------------------------
 STEP OF ION RELAXATION : 2
 -------------------------------------------

toolchain method is the same as #2857

calculation example:

• for icx-mkl toolchain
  bfgs_icx_16_omp16.tar.gz
• for icc-mkl toolchain
  bfgs_oneapi_16_omp16.tar.gz

Expected behavior

• icx-mkl toolchain can run relax job properly
• Since oneapi will update icc to icx, ABACUS compilation by icx should be better than icc, more compilation optimization job should be done in ABACUS itself or in toolchain

To Reproduce

1. Build abacus by icc-mkl or icx-mkl (better by toolchain method)
2. Run the job in attached example

Environment

• Compiler: OneAPI (icc or icx), Intel MKL, Intel MPI
• OS. Rockylinux 8
• Hardware: Intel Platinum 8358
• Parallel Running method: MPI, OMP or MPI-OMP hybrid.

This Problem can also be observed in Bohrium c16-m32 machine by ABACUS compiled by icx toolchain

Additional Context

No response

[QuantumMisaka]

Tried icx-mkl toolchain with MgO-LCAO and Si-LCAO example, the GRAD value is normal.
The problem may occur in magnetic system by using nspin=2
Since GRAD is gradient from energy, the problem may lies in scf calculation

in icx-mkl calculation example:

(base) [2201110432@wm2-login01 bfgs_icx_16_omp16]$ grep E_KS OUT.bfgs/running_relax.log 
   E_KS(sigma->0)                +63347.8019274                +861891.061988
   E_KS(sigma->0)                +63347.8019275                +861891.061990
   E_KS(sigma->0)           +62359.431213083022          +848443.588536981260
   E_KS(sigma->0)           +60832.922948267500          +827674.378091397230
   E_KS(sigma->0)           +60777.874079201276          +826925.399803640670
   E_KS(sigma->0)           +59329.832721135492          +807223.786394287716
   E_KS(sigma->0)           +58702.075110047597          +798682.705920624430
   E_KS(sigma->0)           +57168.874028287093          +777822.435028917622
   E_KS(sigma->0)           +55760.314344934181          +758657.997362242313
   E_KS(sigma->0)           +55709.606076708587          +757968.075978661887
   E_KS(sigma->0)           +54763.658611046325          +745097.800436995807
   E_KS(sigma->0)           +54763.658611010796          +745097.800436512334
   E_KS(sigma->0)           +54280.460895070151          +738523.558239134145
   E_KS(sigma->0)           +54280.460894947370          +738523.558237463702
   E_KS(sigma->0)           +54243.939856542522          +738026.664018280921
   E_KS(sigma->0)           +53419.124227723587          +726804.471666890313

in icc-mkl calculation example:

(base) [2201110432@wm2-login01 bfgs_oneapi_16_omp16]$ grep E_KS OUT.bfgs/running_relax.log 
   E_KS(sigma->0)                -19390.7549737                -263824.756164
   E_KS(sigma->0)                -19390.7549739                -263824.756166
   E_KS(sigma->0)           -19390.266322550593          -263818.107724193949
   E_KS(sigma->0)           -19390.756552289229          -263824.777641968452
   E_KS(sigma->0)           -19390.758582225277          -263824.805260665307
   E_KS(sigma->0)           -19390.762130160532          -263824.853532800917
   E_KS(sigma->0)           -19390.763274225130          -263824.869098598312
   E_KS(sigma->0)           -19390.764814087008          -263824.890049493988
   E_KS(sigma->0)           -19390.765900185215          -263824.904826618207
   E_KS(sigma->0)           -19390.767387304972          -263824.925059920468
   E_KS(sigma->0)           -19390.767824806459          -263824.931012433604
   E_KS(sigma->0)           -19390.768468793583          -263824.939774327911
   E_KS(sigma->0)           -19390.768609947296          -263824.941694822686
   E_KS(sigma->0)           -19390.768885415480          -263824.945442759607
(base) [2201110432@wm2-login01 bfgs_oneapi_16_omp16]$ ls

So the problem seems to be in the electronic structure calculation in nspin=2 system

[QuantumMisaka]

Test with Cu-graphene magnetic system with 0.5 AMAG, calculation can do properly with a faster speed,
The problem is interesting, and can be repeated in different server environment

using icpc-toolchain to compile dependences and the use icpx to compile ABACUS will also lead to this bug.

[hongriTianqi]

• Verify the issue is not a duplicate.
• Describe the bug.
• Steps to reproduce.
• Expected behavior.
• Error message.
• Environment details.
• Additional context.
• Assign a priority level (low, medium, high, urgent).
• Assign the issue to a team member.
• Label the issue with relevant tags.
• Identify possible related issues.
• Create a unit test or automated test to reproduce the bug (if applicable).
• Fix the bug.
• Test the fix.
• Update documentation (if necessary).
• Close the issue and inform the reporter (if applicable).

[QuantumMisaka]

Tried abacus-latest:intel Bohrium image for daily test, result below
https://labs.dp.tech/projects/abacustest/?request=GET%3A%2Fapplications%2Fabacustest%2Fjobs%2Fjob-fe-example-ompintel-abacustest-v0.3.27-530f46
The problem still existed

[QuantumMisaka]

As #2927, after applied -fp-model=strict to compilation, the energy calculation result is still wrong

[QuantumMisaka]

By using ABACUS v3.4.0, This problem seems to be solved.

                              ABACUS v3.4.0

               Atomic-orbital Based Ab-initio Computation at UStc                    

                     Website: http://abacus.ustc.edu.cn/                             
               Documentation: https://abacus.deepmodeling.com/                       
                  Repository: https://github.com/abacusmodeling/abacus-develop       
                              https://github.com/deepmodeling/abacus-develop         
                      Commit: 923b69739 (Wed Oct 4 08:03:06 2023 +0800)

 Mon Oct  9 18:47:20 2023
 MAKE THE DIR         : OUT.bfgs/

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 Warning: the number of valence electrons in pseudopotential > 8 for Fe: [Ar] 3d6 4s2
 Pseudopotentials with additional electrons can yield (more) accurate outcomes, but may be less efficient.
 If you're confident that your chosen pseudopotential is appropriate, you can safely ignore this warning.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

 UNIFORM GRID DIM        : 125 * 160 * 240
 UNIFORM GRID DIM(BIG)   : 25 * 32 * 80
 DONE(0.109067   SEC) : SETUP UNITCELL
 DONE(0.141456   SEC) : SYMMETRY
 DONE(0.219378   SEC) : INIT K-POINTS
 ---------------------------------------------------------
 Ion relaxation calculations
 ---------------------------------------------------------
 SPIN    KPOINTS         PROCESSORS  NBASE       
 2       8               16          2664        
 ---------------------------------------------------------
 Use Systematically Improvable Atomic bases
 ---------------------------------------------------------
 ELEMENT ORBITALS        NBASE       NATOM       XC          
 H       2s1p-6au        5           2           
 C       2s2p1d-7au      13          37          
 O       2s2p1d-7au      13          1           
 Fe      4s2p2d1f-8au    27          80          
 ---------------------------------------------------------
 Initial plane wave basis and FFT box
 ---------------------------------------------------------
 DONE(0.252691   SEC) : INIT PLANEWAVE
 -------------------------------------------
 STEP OF ION RELAXATION : 1
 -------------------------------------------
 START CHARGE      : atomic
 DONE(2.6286     SEC) : INIT SCF
 ITER   TMAG      AMAG      ETOT(eV)       EDIFF(eV)      DRHO       TIME(s)    
 GE1    1.30e+02  1.36e+02  -2.636373e+05  0.000000e+00   8.669e-02  1.996e+01  
 GE2    8.03e+01  8.55e+01  -2.636549e+05  -1.761770e+01  1.552e-01  1.742e+01  
 GE3    1.31e+02  1.39e+02  -2.637876e+05  -1.327641e+02  6.050e-02  1.749e+01  
 GE4    1.39e+02  1.47e+02  -2.638028e+05  -1.518177e+01  5.103e-02  1.740e+01  
 GE5    1.39e+02  1.48e+02  -2.638058e+05  -3.011335e+00  3.696e-02  1.743e+01  
 GE6    1.39e+02  1.48e+02  -2.638040e+05  1.794449e+00   3.384e-02  1.739e+01  
 GE7    1.41e+02  1.50e+02  -2.638065e+05  -2.440206e+00  2.423e-02  1.737e+01  
 GE8    1.43e+02  1.53e+02  -2.638074e+05  -9.270501e-01  1.923e-02  1.739e+01  
 GE9    1.43e+02  1.53e+02  -2.638079e+05  -4.920444e-01  1.698e-02  1.736e+01  
 GE10   1.43e+02  1.53e+02  -2.638076e+05  2.842442e-01   1.704e-02  1.735e+01  
 GE11   1.44e+02  1.55e+02  -2.638080e+05  -3.599622e-01  1.517e-02  1.736e+01  
 GE12   1.44e+02  1.55e+02  -2.638081e+05  -1.185213e-01  1.484e-02  1.731e+01  
 GE13   1.44e+02  1.55e+02  -2.638086e+05  -5.031615e-01  1.170e-02  1.736e+01  
 GE14   1.45e+02  1.57e+02  -2.638088e+05  -1.602977e-01  8.759e-03  1.732e+01  
 GE15   1.45e+02  1.57e+02  -2.638088e+05  -8.408801e-02  7.838e-03  1.737e+01  
 GE16   1.45e+02  1.58e+02  -2.638089e+05  -4.257580e-02  6.828e-03  1.735e+01  
 GE17   1.45e+02  1.58e+02  -2.638089e+05  -2.479346e-02  6.255e-03  1.734e+01  
 GE18   1.45e+02  1.58e+02  -2.638089e+05  -2.366145e-02  5.812e-03  1.732e+01  
 GE19   1.45e+02  1.58e+02  -2.638090e+05  -1.738967e-02  5.561e-03  1.735e+01  
 GE20   1.45e+02  1.58e+02  -2.638090e+05  -3.612013e-02  5.325e-03  1.735e+01  

Interesting, waiting for reason
may be correlated with something of LCAO calculation part

[hongriTianqi]

May we close this issue if it has disappeared? @QuantumMisaka

[QuantumMisaka]

May we close this issue if it has disappeared? @QuantumMisaka

Surely can be closed, but I’m curious about the reason