Issues with AM3-n640: NaNs in BICGstab

Atmosphere
1

Hi all,

We are running the AM3 at N640 resolution, but the model consistently crashes with NaNs. The initial condition was reconfigured from an N320 run. The N640 run can integrate for about 32 days before crashing.

  • Setup: The initial date is 1982-03-01. The model can integrate normally for one month (to 1982-04-01), but it cannot continue beyond that.
  • Tests performed:
  1. Restarting from 1982-04-01 → crash after 2 days.
  2. Running continuously from 1982-03-01 without restart → integrates for 32–33 days before crash (so the error is not due to the restart dump).
  3. Redating the 1982-04-01 restart back to 1982-03-01 (with and without recon) → crash after 1–2 days (so the error is not due to external ancillary files from April).
  4. Adding damping and a polar filter extends the integration by ~10 days, but the model still crashes with NaNs.
[namelist:run_diffusion]
pofil_opt=1
l_pofil_new=.true.
top_diff=0.3
hdiffopt=2

[namelist:run_dyn]
eg_vert_damp_profile=2
eg_vert_damp_coeff=0.10
damp_height=85000.0   !! also tested 70000.0, but still crashed
polar_cap_rows=7

I suspect that the overall quality of the initial conditions is poor, because the output log shows abnormally large omega values before crash, sometimes near the pole and sometimes over the tropics. I’m a little bit surprised that even after one month of integration the model has not stabilized. I have placed the log from test 1 in /scratch/public/sl5165.

Question: does anyone have suggestions on how to generally overcome this issue?

2

AM3 is out of scope for support since it hasn’t been fully released yet.

For pointers, I would look closer at the last output for the month that runs and see if anything jumps out as anomalous. You might want/need to redo the run and turn on outputs at each timestep (but that will be a huge amount of data at the resolution) or at least remove any time averaging in the outputs (i.e. if you output daily data, it should be instantaneous values and not averaged).