Add script support for ARRM10to60E2r1 ocn/ice grid by darincomeau · Pull Request #5178 · E3SM-Project/E3SM

darincomeau · 2022-09-07T22:19:17Z

Brings in support for the ARRM10to60E2r1 ocn/ice mesh. MPAS-Dev/compass#414 has details on the mesh generation, which is an updated version of the oARRM60to10 mesh. This PR includes all necessary mapping, runoff, and domain files for the following configurations:

ne30pg2_ARRM10to60E2r1
arcticx4v1pg2_ARRM10to60E2r1
TL319_ARRM10to60E2r1 (JRA-forced)
T62_ARRM10to60E2r1 (CORE-forced)

Testing

The following 5 day smoke tests have been run on Chrysalis:
WCYCL1950.ne30pg2_ARRM10to60E2r1
WCYCL1850.arcticx4v1pg2_ARRM10to60E2r1
GMPAS-JRA1p4.TL319_ARRM10to60E2r1
GMPAS-IAF.T62_ARRM10to60E2r1

A WCYCL1950.ne30pg2_ARRM10to60E2r1 B-case is currently running on Cori as the final check for this mesh, and is in year 26. Here is the MPAS-Analysis for years 11-20

[BFB] for all currently tested configurations.

…ce timestep

milenaveneziani

This looks good @darincomeau.
Let me add that I also tested this on cori: two simulations are running successfully with this MPAS mesh and with ne30 and the ArcticRRM for the atmosphere/land mesh.

…in sea ice timestep" This reverts commit 2e4100b.

darincomeau · 2022-09-08T19:22:21Z

I've added the Do not merge tag until I can generate land ICs for these configurations.

xylar

It seems like at least 2 changes are needed here.

In addition, I would request that the diagnostics files get copied into place from:

/lcrc/group/e3sm/ac.xylar/compass_1.2/anvil/test_20220730/ec_arrm_for_e3sm/ocean/global_ocean/ARRM10to60/PHC/files_for_e3sm/assembled_files/diagnostics

to

/lcrc/group/e3sm/public_html/diagnostics

Please make sure they are all in the E3SM group and have group read/write and world read permissions.

These files being available will significantly speed up MPAS-Analysis runs. I will sync these files over to other E3SM supported machines as part of the E3SM-Unified emergency release (1.7.1) that I'm working on.

xylar · 2022-09-13T16:35:53Z

components/mpas-ocean/bld/namelist_files/namelist_defaults_mpaso.xml

-<config_Redi_min_layers_diag_terms>6</config_Redi_min_layers_diag_terms>
+<config_Redi_min_layers_diag_terms>15</config_Redi_min_layers_diag_terms>


If I undrestand correctly, this is changing Redi for all configurations that have GM on. This needs to only be for ARRM10to60E2r1. I believe this is correct:

<config_Redi_min_layers_diag_terms>6</config_Redi_min_layers_diag_terms> <config_Redi_min_layers_diag_terms ocn_grid="ARRM10to60E2r1">15</config_Redi_min_layers_diag_terms>

Oops, that was definitely my mistake commit, I certainly should have included the grid specification. Thanks for the catch!
@milenaveneziani I just want to confirm that we do indeed want this separate setting for this mesh, before I make the change (or just revert the commit)?

yes, we want it at 15 for this mesh.

xylar · 2022-09-13T16:41:10Z

components/mpas-ocean/cime_config/buildnml

+        decomp_date = '220730'
+        decomp_prefix = 'mpas-o.graph.info.'
+        restoring_file = 'sss.PHC3.0_monthlyClimatology.ARRM10to60E2r1.220802.nc'
+        analysis_mask_file = 'ARRM10to60E2r1_mocBasinsAndTransects220730.nc'


I don't think this file should have been given the date it was created. The date stamp on the mocBasinsAndTransects files is a date stamp for the regions used in the masks, rather than the masks themselves. This won't matter in E3SM but it will be a problem if the file to be added todiagnostics was modified in this way. In general, I think it would be better to leave the file with the date stamp created in compass:

ARRM10to60E2r1_mocBasinsAndTransects20210623.nc

For a little more background, this date is controlled and updated in geometric_features:
https://github.com/MPAS-Dev/geometric_features/blob/master/geometric_features/aggregation/__init__.py#L56-L58

Thanks for pointing this out. I just diffed the file with the 220730 datestamp and the 20210623 one and they are the same, so I must have mved this to match the other datestamps? I shouldn't have done that - I'll fix it.

we did go through a couple of mesh iterations between June and July though..

we did go through a couple of mesh iterations between June and July though..

@milenaveneziani, that date stamp is from June 2021, not 2022. It is the date that we added the separate Atlantic and Atlantic+Mediterranean MOC basins to geometric features and has nothing to do with when the file or the mesh itself was created.

darincomeau · 2022-09-13T18:14:57Z

In addition, I would request that the diagnostics files get copied into place from:

/lcrc/group/e3sm/ac.xylar/compass_1.2/anvil/test_20220730/ec_arrm_for_e3sm/ocean/global_ocean/ARRM10to60/PHC/files_for_e3sm/assembled_files/diagnostics

@xylar Done -thanks for pointing out the missed step!

xylar · 2022-09-13T19:29:50Z

@darincomeau, I just synced the files over from Anvil to Cori. Thanks for putting them in place!

I'm trying to run MPAS-Analysis (with --polar_regions this time to get some Arctic WOA analysis and to test the cached mapping and region mask files) but I'm getting permission denied. Can you give me access to the files in:

/global/cscratch1/sd/dcomeau/e3sm_scratch/cori-knl/20220803.WCYCL1950.ne30pg2_ARRM10to60E2r1.baseline.cori-knl/run/

?

darincomeau · 2022-09-13T19:56:21Z

Opened - thanks for testing @xylar !

…aving 6 as general default

…insAndTransects20210623.nc

darincomeau · 2022-09-13T20:32:15Z

@xylar the last two commits addressed your changes. As part of d81b754 I also renamed the analysis_mask _file from the 220730 datestamp to the correct 20210623 one in the inputdata directory to avoid future confusion. (rm is 'rename', right? :D)

xylar · 2022-09-13T21:12:39Z

Thanks @darincomeau. As soon as I have permission, I'll rerun the analysis and then (hopefully) approve.

darincomeau · 2022-09-13T21:17:53Z

@xylar that Cori run should be opened now. Note that the streams.ocean file for that run points to the old ARRM10to60E2r1_mocBasinsAndTransects220730.nc file in the inputdata directory at NERSC though, if that's relevant for your testing.

xylar · 2022-09-13T22:57:33Z

@darincomeau, ah, thanks. I was expecting you to change group rather than make world readable so didn't notice.

No, the ARRM10to60E2r1_mocBasinsAndTransects220730.nc file in inputdata is not the one used by analysis. It has its own redundant copy that is stored elsewhere and does not use anything from streams.ocean to find it. So we'll be fine.

xylar

@darincomeau and @milenaveneziani, I think this mesh is working okay and the necessary files are in place. With @darincomeau's recent changes, I think the code changes look reasonable.

I reran MPAS-Analysis here:
https://portal.nersc.gov/project/e3sm/xylar/analysis/arrm/20220803.WCYCL1950.ne30pg2_ARRM10to60E2r1.baseline.cori-knl/ts_0011-0020_climo_0011-0020/
The mapping and mask files were taken from the diagnostics cache, as expected.

There are some fairly serious temperature and especially salinity biases in the WOA18 plots. I'm merely pointing those out as something that HiLat will presumably need to investigate. It isn't clear to me whether changes in the mesh (e.g. resolution or passages) would have a mitigating effect.

darincomeau · 2022-09-14T00:58:51Z

Thanks @xylar !

There are some fairly serious temperature and especially salinity biases in the WOA18 plots. I'm merely pointing those out as something that HiLat will presumably need to investigate. It isn't clear to me whether changes in the mesh (e.g. resolution or passages) would have a mitigating effect.

Yeah that's been a pretty persistant problem, actually looks a little bit better with the new mesh. Here's an run with an older mesh for comparison:
https://portal.nersc.gov/project/m1199/milena/20210914.WCYCL1950.ne30pg2_oARRM60to10.hybrid/Years17-21/ocean/index.html#arctic_salinity_woa

xylar · 2022-09-14T01:59:54Z

Okay, thanks @darincomeau. Nothing new, then. That's presumably good news, and we need to find a solution in the physics and parameterizations, not the mesh.

milenaveneziani · 2022-09-14T15:01:18Z

@xylar: this is great. I believe the new mesh is so much better than the old one, and the new bathymetry is so much smoother. The B-cases with the low-res atmosphere look quite awful at the moment, but that has nothing to do with the mesh (imho). The coupled cases with the atmRRM mesh should look much better.
Thanks for all the help on this new Arctic mesh!

darincomeau · 2022-09-21T20:38:50Z

I've added a default land finidat (from a 50 year I-case spinup) for the WCYCL1950.ne30pg2_ARRM10to60E2r1 configuration only, and ran a quick smoke test to make sure it was picked up in lnd_in and used.

darincomeau · 2022-09-21T20:39:40Z

Removed the Do not merge flag - @jonbob I think this is ready whenever's a good time.

Add script support for ARRM10to60E2r1 ocn/ice grid Brings in support for the ARRM10to60E2r1 ocn/ice mesh. MPAS-Dev/compass#414 has details on the mesh generation, which is an updated version of the oARRM60to10 mesh. This PR includes all necessary mapping, runoff, and domain files for the following configurations: * ne30pg2_ARRM10to60E2r1 * arcticx4v1pg2_ARRM10to60E2r1 * TL319_ARRM10to60E2r1 (JRA-forced) * T62_ARRM10to60E2r1 (CORE-forced) [BFB] for all currently tested configurations

jonbob · 2022-10-03T20:59:58Z

test merge passes:

SMS_Ld1.ne30pg2_EC30to60E2r2.WCYCL1850.chrysalis_intel.allactive-wcprod
ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel

verified that all new files are available on the inputdata repo

merged to next

jonbob · 2022-10-04T19:58:42Z

merged to master

darincomeau added 8 commits August 2, 2022 18:30

Adding support for ARRM10to60E2r1 ocn/ice mesh to config_grids

8a6c28b

Adding ARRM10to60E2r1 ocn/ice mesh to list of valid meshes for ELM

33baa35

Adding ARRM10to60E2r1 ocn/ice mesh to E3SM namelist/streams generation

fa1cabf

Setting default namelist options for ARRM10to60E2r1 ocn/ice mesh

761b1d3

Updating spunup ICs for ARRM10to60E2r1 ocn/ice mesh

29a1207

Updating sea ice graph partition files to v2

41f7b1d

Changing default Redi min layers to 15

b480ae8

Changing default coupling frequencies for G-cases so uses 15min sea i…

2e4100b

…ce timestep

darincomeau added MPAS-ocean Concerning the MPAS-ocean model coupled to E3SM. MPAS-seaice Concerning the MPAS-seaice model Scripts BFB PR leaves answers BFB RRM Regionally refined model labels Sep 7, 2022

darincomeau requested review from milenaveneziani and xylar September 7, 2022 22:19

darincomeau assigned jonbob Sep 7, 2022

darincomeau mentioned this pull request Sep 7, 2022

New Mesh: Add ARRM10to60E2r1 mesh MPAS-Dev/compass#414

Merged

milenaveneziani approved these changes Sep 7, 2022

View reviewed changes

Revert "Changing default coupling frequencies for G-cases so uses 15m…

c0c860a

…in sea ice timestep" This reverts commit 2e4100b.

darincomeau added the Do not merge label Sep 8, 2022

xylar requested changes Sep 13, 2022

View reviewed changes

darincomeau added 2 commits September 13, 2022 15:22

Making change to Redi_min_layers_diag_terms 15 for this mesh only, le…

debc6b0

…aving 6 as general default

Changing name of analysis_mask_file to intended ARRM10to60E2r1_mocBas…

d81b754

…insAndTransects20210623.nc

xylar approved these changes Sep 13, 2022

View reviewed changes

Adding land initial condition for WCYCL1950 compset with ne30pg2 atm

b87a2bf

darincomeau removed the Do not merge label Sep 21, 2022

jonbob merged commit 4aad3b2 into master Oct 4, 2022

jonbob deleted the darincomeau/grids/add-ARRM10to60E2r1 branch October 4, 2022 19:58

darincomeau mentioned this pull request Oct 5, 2022

Updates to ARRM10to60E2r1 mesh darincomeau/E3SM#26

Merged

xylar mentioned this pull request Nov 30, 2022

Ocean fails stand-alone decomp test, intel optimized #5219

Closed

		<config_Redi_min_layers_diag_terms>6</config_Redi_min_layers_diag_terms>
		<config_Redi_min_layers_diag_terms>15</config_Redi_min_layers_diag_terms>

Conversation

darincomeau commented Sep 7, 2022

Uh oh!

milenaveneziani left a comment

Choose a reason for hiding this comment

Uh oh!

darincomeau commented Sep 8, 2022

Uh oh!

xylar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xylar Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

darincomeau Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

milenaveneziani Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

xylar Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

xylar Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

darincomeau Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

milenaveneziani Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

xylar Sep 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darincomeau commented Sep 13, 2022

Uh oh!

xylar commented Sep 13, 2022

Uh oh!

darincomeau commented Sep 13, 2022

Uh oh!

darincomeau commented Sep 13, 2022

Uh oh!

xylar commented Sep 13, 2022

Uh oh!

darincomeau commented Sep 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Sep 13, 2022

Uh oh!

xylar left a comment

Choose a reason for hiding this comment

Uh oh!

darincomeau commented Sep 14, 2022

Uh oh!

xylar commented Sep 14, 2022

Uh oh!

milenaveneziani commented Sep 14, 2022

Uh oh!

darincomeau commented Sep 21, 2022

Uh oh!

darincomeau commented Sep 21, 2022

Uh oh!

jonbob commented Oct 3, 2022

Uh oh!

jonbob commented Oct 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xylar left a comment •

edited

Loading

xylar Sep 13, 2022 •

edited

Loading

darincomeau commented Sep 13, 2022 •

edited

Loading