Segmentation fault with non squared blacs grid when calling chase_init_blockcyclic

[sangallidavide]

I'm testing a 2x4 blacs grid

Error message during the chase_init_blockcyclic

[giumello:199318] *** Process received signal ***
[giumello:199318] Signal: Segmentation fault (11)
[giumello:199318] Signal code: Address not mapped (1)
[giumello:199318] Failing at address: (nil)
[giumello:199318] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x7d6a9a18edf0]
[giumello:199318] [ 1] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0x21e8cae]
[giumello:199318] [ 2] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0x21e90a6]
[giumello:199318] [ 3] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0x21ea336]
[giumello:199318] [ 4] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0x21b0006]
[giumello:199318] [ 5] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0x21b0309]
[giumello:199318] [ 6] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xe4aeba]
[giumello:199318] [ 7] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xb927a9]
[giumello:199318] [ 8] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xb679e3]
[giumello:199318] [ 9] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xb3a14b]
[giumello:199318] [10] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xb13fa2]
[giumello:199318] [11] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xb11d59]
[giumello:199318] [12] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xf0b2e1]
[giumello:199318] [13] /home/sangalli/data/lumen/gpl-repo/tech/chase/bin/yambo() [0xb0ff52]
[giumello:199318] [14] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7d6a9a178ca8]

 call pcchase_init_blockcyclic(N, neig, nex, nbr, nbc, Mat%blc(:,:,1), nbr, Vtemp, Etemp,&
 &                              nprow, npcol, grid_major, 0, 0, comm, ierr)

with

N=2048
neig=nex=525
nbr=1024
nbc=512
nprow=2
mpcol=4

Blocks are correctly split, and allocated, with sizes

rows 1:1024 - 1025:2048;
cols 1:512 - 513:1024 - 1025:1536 - 1537 : 2048

• Reducing neig=100, some cores overcome the init step. However, I get the same error message.
• Switching to blacs grid 4x2 (and switching all dimensions accordingly) code runs but results are not correct

[brunowu]

Hi Davide, for performance reason, ChASE prefers to have squared MPI grid. If this is not squared, it asks for grid 4x2, rather than 2x4.

Can u paste the logs of residuals when you try to run with 4x2 grid, and neig=100?
If possible, for same eigenproblem, can u also run with 2x2 grid, and paste the logs of residuals here?

[sangallidavide]

Ok, I can stick to squared blacs grids then.

For the comparison, I did 4x4, 4x2, and 2x2
4x4 and 2x2 give identical spectra and eigenvalues. However the eigenvectors are somehow different (I guess due to gauge freedom or similar)
4x2 also gives different eigenvalues and spectra. Also it takes many more iterations

I just report the first 10 eigenvalues here:

4x2 run. E=  -0.374345362     -0.314299732     -0.302408814     -0.285650820     -0.277123630     -0.262408674     -0.247975573     -0.224508613     -0.210091233     -0.206744581
2x2 run. E=   0.464615673      0.464619577      0.464622438      0.494314224      0.494318336      0.494323075      0.497132182      0.497138232      0.505785227      0.505785644
4x4 run.  E=   0.464615673      0.464619577      0.464622438      0.494314224      0.494318336      0.494323075      0.497132182      0.497138232      0.505785227      0.505785644

Thanks for the clarification.

[brunowu]

Well, the 4x2 looks not good at all.

Since you use the Block-Block distribution in fact, even it is created with Block-Cyclic manner. Maybe you can try with more mature implementation of ChASE with Block-Block distribution using following API:

call pcchase_init(N, neig, nex, nrows, ncols, Mat%blc(:,:,1), nbr, Vtemp, Etemp, nprow, npcol, grid_major, comm, ierr)

rather than:

 call pcchase_init_blockcyclic(N, neig, nex, nbr, nbc, Mat%blc(:,:,1), nbr, Vtemp, Etemp,&
 &                              nprow, npcol, grid_major, 0, 0, comm, ierr)

Vtemp should be still the same as nrows x (neig+nex).

Can you try this with 2x2, 4x2, and 4x4 grid?

[sangallidavide]

With pcchase_init it works fine also with 4x2.

Also performances are not so bad

Chase solver. ser --> 108.4 s
Chase solver. 2x2 --> 25.7 s
Chase solver. 4x2 --> 14.3 s

Thanks as always.

Can I forget about pcchase_init_blockcyclic and just use pcchase_init ?

Also, could it be better:

call pcchase_init(N, neig, nex, nrows, ncols, Mat%blc(:,cols(1):cols(2),1), nbr, Vtemp, Etemp, nprow, npcol, grid_major, comm, ierr)

[brunowu]

Great, seems there is a bug in the block-cyclic version. I'd like to leave this issue open until the bug identified and fixed.

As long as you use de facto block-block distribution, then you can forget pcchase_init_blockcyclic . pcchase_init should be slightly better in terms of performance.

Btw, I just realized that the Lumen code is independent from Yambo. Will the changes related to ChASE eventually be merged into Yambo, or should we report this issue there ourselves?

[brunowu]

For Mat%blc(:,cols(1):cols(2),1) vs Mat%blc(:,:,1), I believe it should not matter. As long as the leading dimension is set correctly and matrix is in column major order, ChASE will internally use a pointer to reference the data.

[sangallidavide]

Btw, I just realized that the Lumen code is independent from Yambo. Will the changes related to ChASE eventually be merged into Yambo, or should we report this issue there ourselves?

Good question.

Short answer: on my side, I'm both involved in yambo and in the lumen fork, and I'd be glad to integrate any change from lumen into yambo and viceversa. My philosophy is to try avoiding work duplication as much as possible.

Long answer: so far I proposed to integrate many changes done in lumen into the main yambo version (in different occasions) but I never got a feedback. In December I also opened a PR (I was preparing lumen 2.0 release), but I did not get a feedback on that neither. yambo-code/yambo#219

Recently I wrote to both Andrea and Daniele informing them about my upgrades onto the yambo - chase interface. Mostly that, in my version, the BSE matrix is fully distributed in memory (although the eigenvectors are not). In my todo list, I'd like to (i) compare chase, elpa, and slepc (https://arxiv.org/html/2504.10096v1) and (ii) prepare a tutorial using this solver at a forthcoming TIMES school in Cineca in November on GPUs.

As soon as Petru (PhD student and first author of the arxiv) will have some time, we will tackle the CPU case, now that the code is ready. I'll be glad to share a the outcome. Then, I'll try the GPU porting in view of the Cineca school.