Switch to 64 vertical levels for 3 meshes

[xylar]

For the EC30to60, SO12to60 and WC14 meshes, switch from 60 to 64 vertical levels. The surface and deep-ocean resolutions are the same as before (10 and 250 m, respecitively) as is the bottom depth (5500 m).

For the ARRM10to60 mesh, we stick with 80 vertical levels. These have thicknesses between 2 and 150 m, and a bottom depth of 5500 m. These values are quite similar to but not identical to the E3SM v1 80-layer mesh, see:

https://github.com/MPAS-Dev/compass/blob/main/compass/ocean/vertical/80layerE3SMv1.json

This merge also updates time steps for the EC30to60, SO12to60 and WC14. The time steps for SSH adjustment (in ice-shelf cavities) and performance are made smaller because waves have not yet been dissipated in these runs. The time step in the final (simulation) step of dynamic adjustment has been changed for each mesh to have both config_dt and config_btr_dt to match E3SM.

Checklist

• User's Guide has been updated
• Developer's Guide has been updated
• Documentation has been built locally and changes look as expected
• Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

[xylar]

@vanroekel and @philipwjones, I'm flagging you both for a review mostly to make sure you're aware of this work and the numbers look okay to you. I'll post a list of layer interface locations shortly once I run a test.

[xylar]

@milenaveneziani, I wanted to check with you if you think switching ARRM from the hard-coded 80-layer grid from E3SM v1 to the algorithmic (tanh) approach here makes sense to you. I think the results will be quite similar to before, but this would give you flexibility to adjust the number of layers or their thicknesses in the future with relative ease. I'll post the layer thicknesses and an example initial condition as soon as I have one.

[philipwjones]

Looks ok to me. Thanks for moving to 64. I think it will help across all machines.

[xylar]

Testing 64 level meshes

The new vertical coordinate is:

 refBottomDepth = 10, 20.6111199929213, 31.8706979008467, 43.8183414214191, 
    56.4960625407976, 69.9484191832074, 84.2226641877549, 99.3689017631865, 
    115.440251518545, 132.493020099531, 150.586880373602, 169.785057997696, 
    190.154525066562, 211.766200371858, 234.69515559662, 259.020826519511, 
    284.827228000811, 312.203171158658, 341.24248071003, 372.044209935736, 
    404.712850121239, 439.358530613562, 476.097204807588, 515.05081642179, 
    556.347439335178, 600.121383028474, 646.513254303263, 695.669964451043, 
    747.744669429051, 802.896628906484, 861.290968328502, 923.098326488037, 
    988.49436961063, 1057.65915179795, 1130.7763010369, 1208.03201010594, 
    1289.61381288627, 1375.70912913791, 1466.50356507707, 1562.17896342966, 
    1662.91120532489, 1768.86777761061, 1880.20513291226, 1997.06588574952, 
    2119.57590565302, 2247.84138646432, 2381.94598840928, 2521.94816427816, 
    2667.87879105084, 2819.73923147404, 2977.49994462592, 3141.09974923443, 
    3310.4458182866, 3485.41444934181, 3665.85261430375, 3851.58024868864, 
    4042.39319783425, 4238.06670031903, 4438.35926085112, 4643.0167486602, 
    4851.77655408787, 5064.3716451189, 5280.53438510701, 5500 ;

EC30to60 initial condition available at:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230329/ec_64_levels/ocean/global_ocean/EC30to60/PHC/init/initial_state/initial_state.nc

The mesh, init and performance tests passed successfully.

dz vs. z

dz vs. z zoom

dz vs. index

dz vs. index zoom

[xylar]

Testing 80 layer ARRM

The new vertical coordinate is:

 refBottomDepth = 2, 4.17591948138654, 6.54323169495554, 9.11877002796724, 
    11.9208472002551, 14.9693847789926, 18.2860538645209, 21.8944278573358, 
    25.8201482738971, 30.0911046348016, 34.7376295009418, 39.7927097786755, 
    45.2922154499221, 51.2751469023766, 57.7839020319848, 64.8645642556923, 
    72.5672124958673, 80.9462540640204, 90.0607811616185, 99.9749514058703, 
    110.758392347862, 122.486629340995, 135.241535291592, 149.111799721775, 
    164.193413125534, 180.590160716107, 198.414117245263, 217.786131507033, 
    238.836285292847, 261.704306809841, 286.539912785645, 313.503046567341, 
    342.763971449518, 374.503169321746, 408.910984781181, 446.186944665891, 
    486.538673483564, 530.180317925041, 577.330390702246, 628.208948195547, 
    683.034031357984, 742.017328901528, 805.359069632915, 873.242219253316, 
    945.826145522801, 1023.24001941219, 1105.57632767941, 1192.88496667023, 
    1285.16844551204, 1382.37872585559, 1484.41614729749, 1591.13072822275, 
    1702.32590548837, 1817.7645168002, 1937.17658327994, 2060.26826370281, 
    2186.73126039587, 2316.25197139787, 2448.51979127509, 2583.23413268379, 
    2720.10993365231, 2858.8815964739, 2999.30544883039, 3141.16091534572, 
    3284.25063869824, 3428.39980167285, 3573.45488632258, 3719.28207491533, 
    3865.76545879997, 4012.8051823225, 4160.31561348702, 4308.22360317474, 
    4456.46687095921, 4604.99253756736, 4753.75581106267, 4902.7188249393, 
    5051.84962059929, 5201.12126332871, 5350.51107922219, 5500 ;

EC30to60 initial condition available at:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230329/arrm_new_80_levels/ocean/global_ocean/ARRM10to60/PHC/init/initial_state/initial_state.nc

The mesh, init and performance tests passed successfully.

dz vs. z

dz vs. z zoom

[xylar]

@vanroekel and @milenaveneziani, please let me know what you think.

[xylar]

@vanroekel, if you are concerned about that linearly increasing dz at the surface in the new coordinate vs. the unchanging values in PHC, we could add another type of vertical coordinate that is more similar to PHC. My hunch is that it won't matter (i.e. the values of dz matter at the surface but not their derivative as long as it's not too large).

[vanroekel]

@xylar my hunch is the same as yours. I definitely expect the near surface dz to be the dominant control here. However, it is possible that the resolution change near the bottom of the boundary layer will have consequences for simulated entrainment. Not sure what that change would be though. Either way I think this layer distribution is worth a test.

[milenaveneziani]

@xylar, @vanroekel: the changes to the 80-level discretization seem really small to me. But about the 60-64 level change: I am concerned by the fact that we loose 5 layers in the upper 150 m (we go from 15 to 10).

[xylar]

@milenaveneziani, do you have an alternative suggestion for how we should do this? Higher res at the surface? A different functional form?

[milenaveneziani]

Let us wait until @vanroekel weighs in, but yes, I wonder whether we could have more layers in the upper 200 meters, maybe borrowing them from the 1000-2000 m range, where the 64-levels seem to have many more than the 60-levels case. Not sure if you would have to change the functional form to achieve that.

[xylar]

I wonder whether we could have more layers in the upper 200 meters, maybe borrowing them from the 1000-2000 m range, where the 64-levels seem to have many more than the 60-levels case. Not sure if you would have to change the functional form to achieve that.

I will need to add another degree of freedom to our current tanh function or a different functional form to allow a more constant resolution near the surface. I'll think about it. Thanks for the feedback.

[vanroekel]

I would certainly support some more resolution near the top, but it's not immediately clear to me what the impact on the simulations would be. My gut feeling is the important aspect will be resolution near the boundary layer base. So it's possible that if we move closer to the PHC distribution we may have very coarse resolution in this region for very deep boundary layers and this could impact simulated entrainment. But without testing it's hard to say.

I'd say we try add a few more layers near the top if possible and run some tests to see the impact of these coordinate choices.

[xylar]

Updated 64 level meshes

This time, I did a tanh profile in layer index space instead of depth. I added a parameter to control the width of the tanh transition in layers (a transition over 28 layers gave a similar behavior to PHC's 60 level grid). The location of the transition in index layer space is determined such that last layer interface is exactly at bottom_depth.

The new vertical coordinate is:

 refBottomDepth = 10, 20.0036437001027, 30.0118475933041, 40.0257586472759, 
    50.0468122350918, 60.076804611333, 70.1179835767493, 80.1731618821008, 
    90.2458590524341, 100.340478720083, 110.462530303171, 120.618906035169, 
    130.818227034832, 141.071275417043, 151.391533514772, 161.795856259857, 
    172.305308819346, 182.946208874501, 193.751421619175, 204.761965757815, 
    216.029000518499, 227.616276795571, 239.603149508266, 252.088262078378, 
    265.194025712543, 279.072022744735, 293.909459565319, 309.936772871928, 
    327.436441825944, 346.752962612024, 368.303780880869, 392.590728539169, 
    420.211152528998, 451.867443459277, 488.373087334637, 530.652742118757, 
    579.733330779371, 636.722991713593, 702.775267431858, 779.03747180521, 
    866.584903791704, 966.346205462258, 1079.02884823729, 1205.05614463018, 
    1344.52699655311, 1497.20621507928, 1662.54736892408, 1839.74358666387, 
    2027.79671767077, 2225.59318650171, 2431.97588741952, 2645.80457797532, 
    2866.00103354373, 3091.57855173283, 3321.65764658961, 3555.47087990393, 
    3792.35999022977, 4031.76814551265, 4273.22956519518, 4516.35813857221, 
    4760.83611997796, 5006.40355136814, 5252.84875138729, 5500 ;

ECwISC30to60 initial condition available at:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230402/ecwisc_64_levels/ocean/global_ocean/ECwISC30to60/PHC/init/initial_state

The mesh and init tests passed successfully. I didn't run performance this time.

dz vs. z

dz vs. z zoom

dz vs. index

dz vs. index zoom

[xylar]

@milenaveneziani and @vanroekel, I think you'll likely be happier with these new results. If so, I will also use them for the other 2 meshes with 64 vertical levels.

[xylar]

In my testing of the SOwISC12to60 mesh with this new coordinate, it seems that the Haney-number vertical coordinate didn't converge. This needs to be addressed but this PR is not the place. Perhaps as part of #582.

The ECwISC30to60 mesh worked fine with the time step adjustments included here. The WCwISC14 tests are still running but seem to be on track (in SSH adjustment). The Haney-number coordinate converged fine. SSH adjustment also appears to be proceeding as expected (changes in SSH are diminishing with each iteration).

[vanroekel]

@xylar I really like the new version of the 64 layer coordinate! I'll be very interested to see this one in use. Thanks!

[milenaveneziani]

Yes, I agree, the new version is really nice! Thanks @xylar.

[xylar]

Thanks @vanroekel and @milenaveneziani.