Explicitly specify nodes in srun calls

[xylar]

We want to explicitly specify how many nodes to use in srun calls for slurm steps because otherwise we may end up with a job running on more nodes than needed and an incompatible PIO task count.

Checklist

• Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

closes #629

[xylar]

@matthewhoffman, it would be good to make sure the full_integration suite works fine on Perlmutter with these changes. It might be worth testing on 3 nodes even if you don't have any tests that big, because that seemed to be the point at which the problem emerged.

[xylar]

Testing

The pr test suite on Perlmutter with 3 nodes passes with this branch and is BFB with a baseline using main (using 2 nodes). Without this fix, the pr suite hangs on 3 nodes as reported in #629.

[xylar]

With further testing, this problem persists even with the fix in this branch. I'm putting it in draft mode. @matthewhoffman, please don't review or test just yet.

[xylar]

Okay, things are now working for me in #631, so I think this is ready for you to test when you can, @matthewhoffman.

[xylar]

@matthewhoffman, I ran the full_integration suite on Chrysalis and everything passed so I think we're good. I'm going to go ahead with this one since it's the first in a chain that's getting pretty long.

[matthewhoffman]

@xylar , sorry to not have noticed this until now. Glad you were able to take care of it without me holding you up.