Mom6 fails after exactly 1 day, U_TRUNC / V_TRUNC output files missing

COSIMA
,
1

@pearseb has set up a new regional mom6 domain, and it inexplicably stops running very suddenly with heaps of truncations. However, we can’t seem to get the U_TRUNC and V_TRUNC files to print out.

I’ve even tried putting in an absolute path rather than relative, so my MOM_parameter_docs says

U_TRUNC_FILE = "/g/data/gb02/ab8992/rom3-rundirs/pearse/work/U_velocity_truncations"

but no files are ever produced.

Firstly

Does anyone know why MOM6 would throw a fatal saying “too many truncations” , ocean.stats says “4000 truncations”, but MOM6 doesn’t print to its U_TRUNC_FILE?

Secondly, I’m a bit stuck with debugging @pearseb 's domain. I’ve tried outputting hourly 3d velocities, temperature, salinity, and nothing looks suspicious (that I’ve found so far) on the hour before the model dies, so I do find the 4000 truncations very surprising. Has anyone seen this kind of behaviour before? Perhaps the “max truncations” error is a red herring?

Initial/boundary conditions generated by regional mom6 look fine. The model runs for exactly a day. I’ve got the mediator saving all coupling fields, and the last one that gets saved (which looks fine) is one coupling timestep before midnight. Perhaps something weird going on with surface or boundary forcing right at 2010-01-02, or some fairytale thing happening at midnight

To replicate, run the regional-mom6 demo with the following settings:

    longitude_extent = [110, 160],
    latitude_extent = [-60, -39.25],
    date_range = ["2012-01-01", "2012-02-01"], 
    layer_thickness_ratio = 250, 
    minimum_depth = 3

and all other settings left as defaults. As per the regional template, we’re running access-om3/2026.05.004. The domain in question contains the demo since it’s the SOTS region in the Southern Ocean - it’s bizarre that this domain, with otherwise identical settings fails. Next step is to just fiddle slightly with the domain boundaries and see if it’s just some quirk. I’d like to get to the bottom of the U_TRUNC thing though as this does not seem to be the intended behaviour!!

Things we’ve ruled out:

  • Obviously dodgy IC/BCs produced by rmom6
  • Timestep issues (have reduced to BC/THERM=10/40s)
2

Related: MOM6 fails to print truncation files even when it dies due to 4000 truncations · Issue #104 · ACCESS-NRI/MOM6 · GitHub

3

We might be best waiting for the truncation files before diagnosing the issue. Truncation errors frequently occur due to bathymetric features and can appear and grow very suddenly. We would need to be printing out every timestep in the lead-up to the issue to diagnose this via the normal outputs (and even then could miss it!). There are a couple of places in the domain which could cause bathymetric related truncation errors - but I don’t want to point fingers without evidence …
@pearseb has a figure of the domain here

4

Thanks @helen ! I think @pearseb and I aren’t particularly attached to that domain - I think you’re right in that just fiddling with domain location & resolution we’d fix some silly bathy issue and find a stable run

But I would like to learn how to fix the truncations-not-printing issue

5

Okay, there’s a funny quirk around CFL_TRUNCATE and CFL_REPORT here. The run has CFL_TRUNCATE = 0.5 (as expected). The default reporting CFL matches this. However, the run also has CFL_TRUNCATE_RAMP_TIME = 7200.0 with the default CFL_TRUNCATE_START = 0.0. This means that during the ramp-up time, the truncation CFL is below the reporting CFL.

We can’t just set the CFL_REPORTto 0.0, because then it would report any velocity as a truncation (even if it’s not truncated)! I’ll write up a PR to allow the reporting CFL to track the truncation CFL.

I have a suspicion the root of the issue is the minimum depth being reduced to 3m instead of the default of 10m? That could uncover some odd bathymetric features…

6

Thanks so much @angus-g !!

Yes I agree I need to dig around more with the changing of minimum depth. For what it’s worth Pearse had originally tried 25m minimum depth and got the same error, then I changed it to 3m for my attempt. I’ve also successfully run a similar bit smaller SOTS domain with no issues. It’s likely some boundary or bathy quirk that might not be worth anyone’s time chasing down.

But that’s really great progress if we can ensure that the model-crashing truncations are actually printed out