Hi Angus.
The quest for a debuggable executable goes back to these tests on a small test domain.
I could run a small test case at two resolutions and multiple decompositions, but when I ran with a 10x10 decomposition at 0.05 degree resolution, it generated this error.
FATAL from PE 86: ERROR: ESMF mesh and MOM6 domain masks are inconsistent! - MOM n, maskMesh(n), mask(n) = 306
1 0
FATAL from PE 82: ERROR: ESMF mesh and MOM6 domain masks are inconsistent! - MOM n, maskMesh(n), mask(n) = 81
1 0
I did some tests on the masks here:
I’ve just run some tests to ensure the bathymetry exists for every grid point outside of the land (i.e. no NaNs in the ocean).
bathymetry = xr.open_dataset(f'{input_dir}/bathymetry.nc')
bathymetry_grid = xr.DataArray(bathymetry.depth.values.reshape(249,140),
dims=['ny','nx'],
coords={'ny':ocean_mask.ny,
'nx':ocean_mask.nx})
(bathymetry_grid.isnull() & ESMF_mask.astype(bool)).sum()
The sum of this quantity is zero. The bathymetry is consistent with the mask.
If I set
bathymetry_grid[100,101]=np.nan the sum is non-zero.
Anyway, I still generate that inconsistent mask error using an older executable. But if I run the same configuration with the debug executable, I generate the error stated above, i.e.
forrtl: error (65): floating invalid
Image PC Routine Line Source
libpthread-2.28.s 0000149D19D07990 Unknown Unknown Unknown
access-om3-MOM6-C 00000000010E58EF limit_topography 422 MOM_shared_initialization.F90
So the plan was create an OM3 executable with debug flags to try and understand the interaction between MOM6 and NUOPCY better.
If this isn’t possible, we may have to revert to good ‘ol print statements and ASCII output files.