ACCESS-rAM3 'Flagship' Experiments

[Paul.Gregory]

A thread to document large rAM3 experiments used as ‘flagships’ for the 21stCenturyWeather Centre-of-Excellence.

Here is the domain for the first experiment. We are using BARRA-R2 to initialise the land-surface, which constrains the outer nest extents to lie within the BARRA-R2 domain.

I had to make the following changes to the rAM3 defaults to cycle for two days.

1. Change the rose-suite metadata to allow rose edit to view four resolutions: python2 setup_metadata 4 4 (note: this will erase your existing rose-suit.conf file)
2. Doubled NPROC to 36,32 for the 5 km resolution
3. Increased NPROC to 54,48 for 1 km resolution
4. Double WALL_CLOCK_LIMIT from 1800 (30 mins) to 3600 (60 mins) to run the 6 hour forecast tasks for 1 km resolution (they take about 40 minutes using 58x48 processors).

Current suite takes about 5.5 hours to run 24 hours. The forecast tasks themselves take just over 4 hours.

It takes about 15 kSU to run 24 hours of forecasts, which consumes about 750 Gb of disk space on /scratch.

Here are the details of this configuration (open in new tab to enlarge):

I repeated this configuration with rg01_rs0[1-3]_ioproc=48 which showed no difference.

Further work:

1. Try OpenMP threads. What is the UM namelist control to activate this? And will this require recompilation of the UM?
2. Incorporate @m.lipson 's world-cover surface data for the 1km nest.
3. Run some scaling tests for the 5 km and 1 km domains.

I’ve encountered some NPROC restrictions with the outer resolution. I generated this message: ? Error message: Too many processors in the North-South direction.The maximum permitted is 32 ,when NPROC 36,32 was applied to the 12km nest. What is causing this error? Is this a restriction with the GAL9 model? (Because I can run the RAL3P2 model with more CPUs).

[Scott (Scott Wales)]

OpenMP doesn’t require a recompile, just set environment variable $OMP_NUM_THREADS (normally done in the Cylc task configuration) and make sure the PBS number of CPUs is (MPI ranks) * (OMP threads)

NPROC restriction is because there’s a minimum grid size for each process, e.g. halo of cells going to the north MPI rank can’t overlap with the halo of cells going to the south MPI rank.

[Scott (Scott Wales)]

I think the number of threads are required to be > 1 if you are using io servers, you may already be using them

[Aidan (Aidan Heerdegen, ACCESS-NRI Release Team Lead)]

Thanks for sharing what you’ve been doing @Paul.Gregory.