Skip to content

Fix GPU segfaults in PW USPP calculations#7647

Merged
mohanchen merged 9 commits into
deepmodeling:developfrom
chengleizheng:fix/uspp-gpu-bug
Jul 21, 2026
Merged

Fix GPU segfaults in PW USPP calculations#7647
mohanchen merged 9 commits into
deepmodeling:developfrom
chengleizheng:fix/uspp-gpu-bug

Conversation

@chengleizheng

@chengleizheng chengleizheng commented Jul 16, 2026

Copy link
Copy Markdown

Summary

Fix GPU segfaults in PW USPP calculations caused by CPU code accessing GPU
memory pointers. Added CPU intermediate buffers and D2H/H2D sync patterns
in three USPP-specific code paths.

Changes

  • hamilt_pw: sPsi reads qq_nt from CPU backup (qq_nt.ptr) instead of
    GPU pointer, fills a CPU buffer, then syncs to GPU for cuBLAS.

  • elecstate_pw (cal_becsum): becp computed on GPU via gemm then synced
    to CPU; auxk1/auxk2 use CPU vectors; aux_gk computed on GPU then synced
    to CPU; becsum allocated on CPU (consumed by CPU-only force/stress code).

  • elecstate_pw (add_usrho): Added D2H/H2D wrappers around real2recip
    and recip2real FFT calls, which read/write via CPU loops.

  • elecstate_pw (addusdens_g): Rewritten to use pure CPU computation
    (CPU BLAS zgemm_, CPU radial_fft_q); ylmk0 computed on GPU with H2D'd
    gcar then D2H'd back.

Test plan

  • Run PW USPP test cases (003, 008, 009) on GPU, verify no segfault
  • Run compute-sanitizer to confirm 0 GPU memory errors

chengleizheng and others added 2 commits July 16, 2026 20:00
@chengleizheng chengleizheng reopened this Jul 16, 2026
@mohanchen mohanchen added Bugs Bugs that only solvable with sufficient knowledge of DFT Refactor Refactor ABACUS codes labels Jul 17, 2026
@chengleizheng

chengleizheng commented Jul 20, 2026

Copy link
Copy Markdown
Author

Benchmark: Fe atom single-point — device=gpu

Hardware: i9-14900HX + RTX 4060 (x1), 32 threads
System: 1 Fe atom, USPP, 9×9×9 grid, collinear spin, 8 k-points

SCF convergence

Iter TMAG AMAG ETOT/eV EDIFF/eV DRHO TIME/s
CG1 1.25e+00 1.25e+00 -2860.20876 0.00e+00 4.22e+01 1.38
CG2 7.50e-01 7.50e-01 -2715.34625 1.45e+02 1.42e+01 0.87
CG3 9.52e-01 9.52e-01 -2736.26617 -2.09e+01 3.15e-01 0.94
CG4 8.94e-01 8.94e-01 -2736.93124 -6.65e-01 2.63e-01 0.78
CG5 7.51e-01 7.51e-01 -2737.65342 -7.22e-01 5.83e-02 0.82
CG6 7.50e-01 7.50e-01 -2738.69933 -1.05e+00 3.02e-02 0.79
CG7 7.50e-01 7.50e-01 -2738.89917 -2.00e-01 1.72e-02 0.95
CG8 7.50e-01 7.50e-01 -2739.30581 -4.07e-01 1.11e-02 0.91
CG9 7.49e-01 7.49e-01 -2739.33261 -2.68e-02 9.13e-04 1.04
CG10 7.48e-01 7.48e-01 -2739.33515 -2.54e-03 8.27e-05 1.01
CG11 7.46e-01 7.46e-01 -2739.33520 -5.26e-05 8.06e-05 0.95
CG12 7.47e-01 7.47e-01 -2739.33531 -1.02e-04 1.11e-06 1.07
CG13 7.47e-01 7.47e-01 -2739.33531 -4.71e-06 4.53e-07 1.07
CG14 7.46e-01 7.46e-01 -2739.33531 -1.28e-06 4.57e-08 1.12
CG15 7.46e-01 7.46e-01 -2739.33531 -3.13e-07 1.75e-09 1.16
CG16 7.46e-01 7.46e-01 -2739.33531 -1.20e-07 4.32e-11 1.33

Final ETOT: -2739.33531 eV

Stress (kbar)

     -92851.610       -0.0073       -0.0056
         -0.0073   -92851.612       -0.0011
         -0.0056       -0.0011   -92851.609

TOTAL-PRESSURE: -92851.610 kbar

Timing (total 17.22s)

Hotspot Time/s Pct
diag_once (CG) 10.91 63.4%
pp_cell_vnl::getvnl 1.95 11.3%
diag_subspace 2.49 14.5%
Operator::hPsi 1.85 10.8%
diag_hegvd 1.32 7.7%
Operator::veff_pw 1.38 8.0%
ElecStatePW::psiToRho 0.99 5.7%
PW_Basis_K::recip_to_real gpu 0.96 5.6%

Benchmark: Fe atom single-point — device=cpu

Hardware: i9-14900HX, 32 threads
System: 1 Fe atom, USPP, 9×9×9 grid, collinear spin, 8 k-points

SCF convergence

Iter TMAG AMAG ETOT/eV EDIFF/eV DRHO TIME/s
CG1 1.25e+00 1.25e+00 -2860.20876 0.00e+00 4.22e+01 0.52
CG2 7.50e-01 7.50e-01 -2715.34625 1.45e+02 1.42e+01 0.76
CG3 9.52e-01 9.52e-01 -2736.26617 -2.09e+01 3.15e-01 1.51
CG4 8.94e-01 8.94e-01 -2736.93124 -6.65e-01 2.63e-01 1.17
CG5 7.51e-01 7.51e-01 -2737.65342 -7.22e-01 5.83e-02 0.76
CG6 7.50e-01 7.50e-01 -2738.69933 -1.05e+00 3.02e-02 0.65
CG7 7.50e-01 7.50e-01 -2738.89917 -2.00e-01 1.72e-02 0.89
CG8 7.50e-01 7.50e-01 -2739.30581 -4.07e-01 1.11e-02 0.47
CG9 7.49e-01 7.49e-01 -2739.33261 -2.68e-02 9.13e-04 0.82
CG10 7.48e-01 7.48e-01 -2739.33515 -2.54e-03 8.27e-05 0.41
CG11 7.46e-01 7.46e-01 -2739.33520 -5.25e-05 8.06e-05 1.16
CG12 7.47e-01 7.47e-01 -2739.33531 -1.02e-04 1.11e-06 0.39
CG13 7.47e-01 7.47e-01 -2739.33531 -4.71e-06 4.53e-07 0.72
CG14 7.46e-01 7.46e-01 -2739.33531 -1.28e-06 4.57e-08 0.64
CG15 7.46e-01 7.46e-01 -2739.33531 -3.15e-07 1.75e-09 0.83
CG16 7.46e-01 7.46e-01 -2739.33531 -1.20e-07 4.32e-11 0.97

Final ETOT: -2739.33531 eV

Stress (kbar)

     -92851.610       -0.0073       -0.0056
         -0.0073   -92851.612       -0.0011
         -0.0056       -0.0011   -92851.609

TOTAL-PRESSURE: -92851.610 kbar

Timing (total 14.01s)

Hotspot Time/s Pct
diag_once (CG) 9.19 65.6%
Operator::veff_pw 4.03 28.8%
PW_Basis_K::recip2real 2.92 20.9%
diag_subspace 2.69 19.2%
diag_hegvd 1.93 13.8%
PW_Basis_K::real2recip 1.18 8.4%
PSIPrepare::initialize_psi 0.94 6.7%
pp_cell_vnl::getvnl 0.72 5.1%
Nonlocal::getvnl 0.69 4.9%
ElecStatePW::psiToRho 0.50 3.6%

@mohanchen

Copy link
Copy Markdown
Collaborator

Great job figuring out this GPU bug with your efforts, and thanks a lot for your contribution.

@mohanchen
mohanchen merged commit 5a6a6f5 into deepmodeling:develop Jul 21, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bugs Bugs that only solvable with sufficient knowledge of DFT Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants