Updating UM source to handle very large high-resolution domains (> 4096 rows)

Atmosphere Unified Model
1

Hi all.

@bethanwhite attempted to run a very large 1 km domain and hit the following hard-coded limits:

???????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 20
?  Error from routine: NLSIZES_NAMELIST_MOD:CHECK_NLSIZES
?  Error message: 1 errors have been found in the nlsizes namelist
?        Please see output for details and then correct the namelist
?  Error from processor: 0
?  Error number: 0
????????????????????????????????????????????????????????????????????????????????

and

Defensive checks on global_row_length (4222) against [2:4096] .... FAIL
Defensive checks on global_rows (888) against [1:3200] .... pass
Defensive checks on model_levels (90) against [1:250] .... pass

Bethan identified the following hard-coded limits in the existing source at /src/control/misc/atmos_max_sizes.F90:MODULE atmos_max_sizes

MODULE atmos_max_sizes

IMPLICIT NONE

! Maximum sector size for I/O

#if defined(UTILIO)
! Small execs needs a larger value to cope with some of the wet
! model types
INTEGER, PARAMETER :: row_length_max   = 8000 ! maximum row length
INTEGER, PARAMETER :: rows_max         = 6000 ! max no of rows
#else
INTEGER, PARAMETER :: row_length_max   = 4096 ! maximum row length
INTEGER, PARAMETER :: rows_max         = 3200 ! max no of rows
#endif

! maximum permitted size of a halo
INTEGER, PARAMETER :: Max_Halo_Size    = 20
INTEGER, PARAMETER :: model_levels_max = 250 ! max no of total levels


INTEGER, PARAMETER :: horiz_dim_max=MAX(row_length_max,rows_max)

@Bethan contacted the UKMO and there are change sets that specifically address these limits.

https://code.metoffice.gov.uk/trac/um/changeset?reponame=&new=117100%40main%2Fbranches%2Fdev%2Fclaudiosanchez%2Fvn13.0_CTC_Kscale&old=116572%40main%2Fbranches%2Fdev%2Fclaudiosanchez%2Fvn13.0_CTC_Kscale

https://code.metoffice.gov.uk/trac/um/changeset?reponame=&new=125263%40main%2Fbranches%2Fdev%2Fclaudiosanchez%2Fvn13.5_CTC_Kscale&old=125141%40main%2Fbranches%2Fdev%2Fclaudiosanchez%2Fvn13.5_CTC_Kscale

These are change sets for UMv13.5. Some of these are specific to their ‘tropical channel’ configuration, others will be required to run rAM3 over large km-scale (and lower) domains.

Bethan is attempting to compile a version locally now to run her large 1km domain.

Question - Do we want to create a UMv13.5_kscale branch in the ACCESS-NRI/UM repo that contains all the changes required to run rAM3 for a very large number of grid points?

2

I wouldn’t call it a k-scale branch if we did this, it is too confusing with the Met Office k-scale project – I would give it a different branch name. But actually I would suggest bringing in the changes to main rAM3 release.

They are minimal and only refer to the limits and a related IO change.

The limits themselves are only used in a check, and the model fails purely because of the defensive checks.

3

I certainly know of at least one other person in 21st Century Weather who is going to run up against this limit. I’ve told him to hang on before constructing a large domain!

I will report back on how the local build goes & whether it successfully gets the model past this (nx, ny) limit.

4

@Paul.Gregory and @bethanwhite, please be careful with putting code up on ACCESS-Hive forum. You might want to check the licensing issues, etc.

Not a problem about making another branch of ACCESS-rAM3. I guess a further question is if you want to stick with UMv13.5.

Let’s talk offline.

5

Thanks Chermelle.

This is going into the Flagship branch for now – i.e. already a separate branch.

We could also create e.g. a rAM3_hugedomain branch. I just wonder whether it would be worth considering putting back into the main rAM3 release.

The changes are minimal (5 lines total, across 3 modules) so might be easiest to put it into 13.8 separately once that release is ready.

I’ll report back here or offline once I know that it works.

6

@Paul.Gregory and @bethanwhite, one sticking point is that the unless users are using Sapphire Rapids there is no point having the model able to run with a very large grid (because it would take an inordinate amount of time). Hence, the changes would need to be accompanied by the optimisations from the Flagship to make it worthwhile (not necessarily with ALL the non-optimisation changes) …

7

@cbengel why don’t I get a number on exactly how slow that is - I can put the same changes into a standard rAM3 build and run both out for 6 hours on that large domain.

8

Great, thanks @bethanwhite – only if you have time!!

9

I’ll get my main suite up and running first, then hopefully spinning up the same in a vanilla rAM3 should be pretty quick :crossed_fingers:

10

I can confirm that the UKMO k-scale suite changes that I have put into a local build allow the model to run with more than nx = 4096 and ny = 3200 :white_check_mark:

This local build is a Flagship suite running on the Sapphire Rapids.

We’ll point the Flagship to use these new executables from now on, and make note that anyone recompiling it needs to include these changes if they want to be able to run big domains.

Next week I’ll put the same changes into a local build of vanilla rAM3 and run this large domain out for 6 hours to get a number on how slow it is compared to Flagship on Sapphire Rapids.