Libxc: 7.0.0 -> 7.1.2 (toolchain); fix external parameter handling

[Growl1234]

This update might need careful checks from the ABACUS side, especially regarding module_xc path. I detected several "fatal deviation" when running integration tests:

01_PW
[ERROR     ] Fatal deviation details:
204_PW_SY: etotref cal=-1689.42889973 ref=-1674.11071764 deviation=15.31818209
204_PW_SY: totalstressref cal=298.73181900 ref=290.50831200 deviation=-8.22350700

07_OFDFT
[ERROR     ] Fatal deviation details:
26_OF_MD_LibxcPBE: totalstressref cal=46.96707800 ref=28.58611600 deviation=-18.38096200

CC: @susilehtola, I hope you could have a look if convenient.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[susilehtola]

Changes in properties that depend on higher derivatives are expected, since Libxc 7.1.0 fixes many finite precision issues. In specific, exchange functionals are no longer evaluated in the problematic $(r_s, \zeta)$ parametrization that gets tricky for $\zeta \to \pm 1$; instead, they're evaluated in the physical $(n_\alpha, n_\beta)$

[Growl1234]

Changes in properties that depend on higher derivatives are expected, since Libxc 7.1.0 fixes many finite precision issues. In specific, exchange functionals are no longer evaluated in the problematic $(r_s, \zeta)$ parametrization that gets tricky for $\zeta \to \pm 1$; instead, they're evaluated in the physical $(n_\alpha, n_\beta)$

Thank you for pointing out! I'd be definitely happy if such deviation is due to issue fixes.

That said, one of the concerns is that adapting to the changes will definitely break the testing workflow, as they worked "well" and consistent from libxc v5.1.7 to v7.0.0, and CI currently uses the system-installed libxc from Ubuntu 22.04. Maybe we will have to raise the minimum version requirement to v7.1.0 and then build libxc from the toolchain in CI; fortunately, thanks to the redesign of maple2c into sympy2c, the installation burden for libxc has been significantly reduced.

[Growl1234]

@susilehtola I tested with ABACUS's internal PBE functional instead in 204_PW_SY, and the energy is still -1.67411089e+03, not around -1689. Therefore, I am still doubtful if libxc has some changes that introduced numerical issues or affected interface ABACUS is using.

Or, anyone familiar with ABACUS's libxc interface can take a look?

[Growl1234]

Moreover the libxc unittest fails.

MODULE_HAMILT_XCTest_PBE

128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:913: Failure
128: The difference between e_gga[i] and e_gga_ref[i] is 0.0034341727201370187, which exceeds 1.0e-8, where
128: e_gga[i] evaluates to -1.6305915205558414,
128: e_gga_ref[i] evaluates to -1.6271573478357044, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:914: Failure
128: The difference between v1_gga[i] and v1_gga_ref[i] is 0.0025431755939138156, which exceeds 1.0e-8, where
128: v1_gga[i] evaluates to -1.2566000636491668,
128: v1_gga_ref[i] evaluates to -1.2591432392430806, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:915: Failure
128: The difference between v2_gga[i] and v2_gga_ref[i] is 0.0039114217380277587, which exceeds 1.0e-8, where
128: v2_gga[i] evaluates to -0.0041500393380277589,
128: v2_gga_ref[i] evaluates to -0.00023861760000000001, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:913: Failure
128: The difference between e_gga[i] and e_gga_ref[i] is 0.048805004428798648, which exceeds 1.0e-8, where
128: e_gga[i] evaluates to -1.4617011932922337,
128: e_gga_ref[i] evaluates to -1.412896188863435, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:914: Failure
128: The difference between v1_gga[i] and v1_gga_ref[i] is 0.011652296334958789, which exceeds 1.0e-8, where
128: v1_gga[i] evaluates to -1.1492652553263285,
128: v1_gga_ref[i] evaluates to -1.1609175516612873, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:915: Failure
128: The difference between v2_gga[i] and v2_gga_ref[i] is 0.0015898088372032059, which exceeds 1.0e-8, where
128: v2_gga[i] evaluates to -0.004174065037203206,
128: v2_gga_ref[i] evaluates to -0.0025842562000000001, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:913: Failure
128: The difference between e_gga[i] and e_gga_ref[i] is 0.0034717780828392789, which exceeds 1.0e-8, where
128: e_gga[i] evaluates to -0.040212536197536924,
128: e_gga_ref[i] evaluates to -0.036740758114697646, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:914: Failure
128: The difference between v1_gga[i] and v1_gga_ref[i] is 0.010753409146911419, which exceeds 1.0e-8, where
128: v1_gga[i] evaluates to -0.45458682522524479,
128: v1_gga_ref[i] evaluates to -0.44383341607833338, and
128: 1.0e-8 evaluates to 1e-08.
128: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc.cpp:915: Failure
128: The difference between v2_gga[i] and v2_gga_ref[i] is 0.027103131262640495, which exceeds 1.0e-8, where
128: v2_gga[i] evaluates to -0.10961965181817482,
128: v2_gga_ref[i] evaluates to -0.082516520555534323, and
128: 1.0e-8 evaluates to 1e-08.

MODULE_HAMILT_XCTest_PZ_SPN

 130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:480: Failure
130: The difference between e_gga[i] and e_gga_ref[i] is 0.019725756092576541, which exceeds 1.0e-8, where
130: e_gga[i] evaluates to -1.6637487720925765,
130: e_gga_ref[i] evaluates to -1.644023016, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:481: Failure
130: The difference between v1_gga[i] and v1_gga_ref[i] is 0.0085821335810192245, which exceeds 1.0e-8, where
130: v1_gga[i] evaluates to -1.3183552994189809,
130: v1_gga_ref[i] evaluates to -1.3269374330000001, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:482: Failure
130: The difference between v2_gga[i] and v2_gga_ref[i] is 0.0071636817589035839, which exceeds 1.0e-8, where
130: v2_gga[i] evaluates to -1.1679279132410965,
130: v2_gga_ref[i] evaluates to -1.175091595, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:480: Failure
130: The difference between e_gga[i] and e_gga_ref[i] is 0.10225190686264352, which exceeds 1.0e-8, where
130: e_gga[i] evaluates to -2.3512216048626433,
130: e_gga_ref[i] evaluates to -2.2489696979999998, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:481: Failure
130: The difference between v1_gga[i] and v1_gga_ref[i] is 0.062312171312131959, which exceeds 1.0e-8, where
130: v1_gga[i] evaluates to -1.8732481263121319,
130: v1_gga_ref[i] evaluates to -1.8109359549999999, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:482: Failure
130: The difference between v2_gga[i] and v2_gga_ref[i] is 0.11615966378396836, which exceeds 1.0e-8, where
130: v2_gga[i] evaluates to -1.7038814007839684,
130: v2_gga_ref[i] evaluates to -1.5877217370000001, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:480: Failure
130: The difference between e_gga[i] and e_gga_ref[i] is 0.0017489954175754496, which exceeds 1.0e-8, where
130: e_gga[i] evaluates to -0.038746598557575451,
130: e_gga_ref[i] evaluates to -0.036997603140000002, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:481: Failure
130: The difference between v1_gga[i] and v1_gga_ref[i] is 0.058269331296355253, which exceeds 1.0e-8, where
130: v1_gga[i] evaluates to -0.7442576481963552,
130: v1_gga_ref[i] evaluates to -0.68598831689999995, and
130: 1.0e-8 evaluates to 1e-08.
130: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc2.cpp:482: Failure
130: The difference between v2_gga[i] and v2_gga_ref[i] is 0.082461397731545538, which exceeds 1.0e-8, where
130: v2_gga[i] evaluates to -0.55740697333154554,
130: v2_gga_ref[i] evaluates to -0.4749455756, and
130: 1.0e-8 evaluates to 1e-08.

MODULE_HAMILT_XCTest_VXC

133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:205: Failure
133: The difference between et1 and -22.58754423 is 0.15964571548737538, which exceeds 1.0e-8, where
133: et1 evaluates to -22.747189945487374,
133: -22.58754423 evaluates to -22.587544229999999, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:206: Failure
133: The difference between vt1 and -29.58543393 is 0.44835305338823161, which exceeds 1.0e-8, where
133: vt1 evaluates to -29.137080876611769,
133: -29.58543393 evaluates to -29.585433930000001, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:208: Failure
133: The difference between v1(0,1) and -2.104367948 is 0.056455815092989692, which exceeds 1.0e-8, where
133: v1(0,1) evaluates to -2.0479121329070105,
133: -2.104367948 evaluates to -2.1043679480000002, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:209: Failure
133: The difference between v1(0,2) and -2.635712365 is 0.048373339515350278, which exceeds 1.0e-8, where
133: v1(0,2) evaluates to -2.5873390254846496,
133: -2.635712365 evaluates to -2.6357123649999998, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:210: Failure
133: The difference between v1(0,3) and -3.005350979 is 0.043895403449478287, which exceeds 1.0e-8, where
133: v1(0,3) evaluates to -2.9614555755505219,
133: -3.005350979 evaluates to -3.0053509790000001, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:211: Failure
133: The difference between v1(0,4) and -3.298397079 is 0.040866087479026092, which exceeds 1.0e-8, where
133: v1(0,4) evaluates to -3.2575309915209738,
133: -3.298397079 evaluates to -3.2983970789999999, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:213: Failure
133: The difference between et2 and -28.97838189 is 0.15071256641907027, which exceeds 1.0e-8, where
133: et2 evaluates to -29.129094456419072,
133: -28.97838189 evaluates to -28.978381890000001, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:214: Failure
133: The difference between vt2 and -38.1541987 is 0.41196498069309939, which exceeds 1.0e-8, where
133: vt2 evaluates to -37.742233719306903,
133: -38.1541987 evaluates to -38.154198700000002, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:216: Failure
133: The difference between v2(0,1) and -2.560885532 is 0.048039854270099269, which exceeds 1.0e-8, where
133: v2(0,1) evaluates to -2.5128456777299006,
133: -2.560885532 evaluates to -2.5608855319999999, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:217: Failure
133: The difference between v2(0,2) and -3.219339294 is 0.041492111667155385, which exceeds 1.0e-8, where
133: v2(0,2) evaluates to -3.1778471823328447,
133: -3.219339294 evaluates to -3.2193392940000001, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:218: Failure
133: The difference between v2(0,3) and -3.678773042 is 0.037804950001257165, which exceeds 1.0e-8, where
133: v2(0,3) evaluates to -3.6409680919987428,
133: -3.678773042 evaluates to -3.678773042, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:219: Failure
133: The difference between v2(0,4) and -4.043604335 is 0.035290175869275409, which exceeds 1.0e-8, where
133: v2(0,4) evaluates to -4.008314159130725,
133: -4.043604335 evaluates to -4.0436043350000004, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:221: Failure
133: The difference between v2(1,1) and -1.394276473 is 0.03088095107095401, which exceeds 1.0e-8, where
133: v2(1,1) evaluates to -1.3633955219290459,
133: -1.394276473 evaluates to -1.3942764729999999, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:222: Failure
133: The difference between v2(1,2) and -1.739027899 is 0.027978447324239264, which exceeds 1.0e-8, where
133: v2(1,2) evaluates to -1.7110494516757608,
133: -1.739027899 evaluates to -1.7390278990000001, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:223: Failure
133: The difference between v2(1,3) and -1.975058937 is 0.026059589066348199, which exceeds 1.0e-8, where
133: v2(1,3) evaluates to -1.9489993479336518,
133: -1.975058937 evaluates to -1.975058937, and
133: 1.0e-8 evaluates to 1e-08.
133: /__w/abacus-develop/abacus-develop/source/source_hamilt/module_xc/test/test_xc5.cpp:224: Failure
133: The difference between v2(1,4) and -2.160368003 is 0.024659209289907391, which exceeds 1.0e-8, where
133: v2(1,4) evaluates to -2.1357087937100925,
133: -2.160368003 evaluates to -2.1603680029999999, and
133: 1.0e-8 evaluates to 1e-08.

[AsTonyshment]

Thanks you for preparing this update and running the additional tests. It is great to see Libxc 7.1.0 finally released, congratulations to @susilehtola and the Libxc team on the long-awaited numerical and implementation improvements.

That said, given the substantial changes in this release and the currently unexplained deviations in ABACUS, switching the default Toolchain and CI to 7.1.0 may be premature. I suggest moving the investigation to a dedicated issue and collecting the relevant test results and discussion there.

Until these differences are understood, I would prefer to keep Libxc 7.0.0 as the default and not merge this PR yet.

[Growl1234]

That said, given the substantial changes in this release and the currently unexplained deviations in ABACUS, switching the default Toolchain and CI to 7.1.0 may be premature. I suggest moving the investigation to a dedicated issue and collecting the relevant test results and discussion there.

I saw 7.1.1 is coming; I will do checks against this branch locally. I agree with filing an issue on the ABACUS side.

Edit: The deviation is still present.

[Growl1234]

I found the issue. Libxc 7.1 added a fourth external parameter, tscale, to GGA_C_PBE, while ABACUS still passes only the three parameters used by libxc 7.0. This causes an out-of-bounds read and effectively sets tscale to an invalid value, leading to the large numerical deviations. It needs to be fixed on the ABACUS side.

[AsTonyshment]

I found the issue. Libxc 7.1 added a fourth external parameter, tscale, to GGA_C_PBE, while ABACUS still passes only the three parameters used by libxc 7.0.

So is your fix compatible with both versions?

[Growl1234]

So is your fix compatible with both versions?

In principle it shoud be backward-compatible with previous versions (even the currently minimun-required v5.1.7).

[Growl1234]

The three deviations in 08_EXX looks acceptable to me (and I am not able to reproduce deviation in 06_KP_MD_HSE), but its performance (slow) might be an issue related to libxc itself (same with cp2k/cp2k#5607 (comment)):

[ RUN      ] 06_KP_MD_HSE
[----------] Molecular dynamics using HSE, multi-k points 
[WARNING   ]  etotref cal=-467.26767979 ref=-467.26767851 deviation=0.00000128
[WARNING   ]  etotperatomref cal=-155.75589326 ref=-155.75589284 deviation=0.00000042

[ RUN      ] 12_KP_OXC
[----------] test for output XC matrix with multipy k-points
[WARNING   ]  etotref cal=-192.45878094 ref=-192.45878106 deviation=-0.00000012

[ RUN      ] 13_NO_KP_CAMPBEH
[----------] CAM-PBEh calculation on Si, multiple k-points
[WARNING   ]  etotref cal=-184.26910481 ref=-184.26910470 deviation=0.00000011

[susilehtola]

I found the issue. Libxc 7.1 added a fourth external parameter, tscale, to GGA_C_PBE, while ABACUS still passes only the three parameters used by libxc 7.0. This causes an out-of-bounds read and effectively sets tscale to an invalid value, leading to the large numerical deviations. It needs to be fixed on the ABACUS side.

Sounds like you are not checking that your parameter arrays are of the right size; please add a proper check to guard from memory corruption.

[jeanwsr]

I think it should also quit when requested_ext_params is shorter than nref. It's dangerous to do "Missing trailing parameters retain Libxc's own defaults".

[Growl1234]

I think it should also quit when requested_ext_params is shorter than nref. It's dangerous to do "Missing trailing parameters retain Libxc's own defaults".

Agreed. However, with the current interface, a strict size check would make the Libxc 7.1 tests fail immediately because ABACUS still supplies the old three-parameter PBE array.

I think the interface should be redesigned so that functionals specified only by name do not call xc_func_set_ext_params() at all. Libxc would then retain its own version-specific defaults, while xc_func_set_ext_params() would be used only when parameters are explicitly overridden. This should also provide the best compatibility across Libxc versions for the time being.

[susilehtola]

The number should be an exact match, since otherwise you don't know what you're going to get...

[jeanwsr]

I think the interface should be redesigned so that functionals specified only by name do not call xc_func_set_ext_params() at all. Libxc would then retain its own version-specific defaults, while xc_func_set_ext_params() would be used only when parameters are explicitly overridden. This should also provide the best compatibility across Libxc versions.

Yes, it's quite surprising to see xc_func_set_ext_params called when input is just PBE.

[AsTonyshment]

It seems that there is a change in your XC interface @kirk0830 🤔 Would you mind taking a review at your spare time? Thanks!

[kirk0830]

@AsTonyshment noted.
I remember I was totally puzzled by the xc autoset behavior in legacy versions and the place where the ext_exch_param and ext_corr_param take their effects was the best solution to the demand of customizing the xc at that time.
So if this PR can pass all the tests and makes ABACUS run more robustly, I think it's a good news to ABACUS!

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Growl1234]

Why is the AI review triggered with every push?