Running ACCESS-ESM1.5 on Sapphire Rapids

General
,
1

Hiya - @manodeep mentioned at the ACCESS-NRI workshop that the ACCESS-ESM1.5 should run faster on the Sapphire Rapids processors than Cascade Lake (where I assume it goes by default(?)) I’ve found some mentions of this in a couple of Forum discussions but I think not enough to actually implement - I also tried to find info on the NCI website but I assume there might be a bit more to it than just submitting to the ‘normalsr’ queue.

So - any guidance here would be very much appreciated (: Also if I am indeed remembering this correctly and it is indeed faster to run the ESM1.5 on SR then maybe others will see this post and do so too.

1 Like
2

Hi @georgyfalster,

Here are the changes I made to get ESM 1.5 to run on the sapphirerapids queue:

diff --git a/atmosphere/um_env.yaml b/atmosphere/um_env.yaml
index d1cedf0..d2b4c5a 100644
--- a/atmosphere/um_env.yaml
+++ b/atmosphere/um_env.yaml
@@ -10,9 +10,9 @@ RPSEED: ''
 RUNID: PI-01
 TYPE: NRUN
 UM_ATM_NPROCX: '16'
-UM_ATM_NPROCY: '15'
+UM_ATM_NPROCY: '13'
 UM_NAM_MAX_SECONDS: '300'
-UM_NPES: '240'
+UM_NPES: '208'
 UM_SECTOR_SIZE: '2048'
 UM_STDOUT_FILE: atm.fort6.pe
 VN: '7.3'
diff --git a/config.yaml b/config.yaml
index b19a2a7..8dd9228 100644
--- a/config.yaml
+++ b/config.yaml
@@ -17,10 +17,14 @@
 laboratory: access-esm
 
 jobname: pre-industrial
-queue: normal
+queue: normalsr
 walltime: 2:30:00
 jobfs: 1500MB
 
+platform:
+  nodesize: 104
+  nodemem: 512
+
 # Modules for loading model executables
 modules:
   use:
@@ -34,7 +38,7 @@ model: access
 submodels:
     - name: atmosphere
       model: um
-      ncpus: 240
+      ncpus: 208
       exe: um_hg3.exe
       input:
         # Aerosols
@@ -67,7 +71,7 @@ submodels:
 
     - name: ocean
       model: mom
-      ncpus: 180
+      ncpus: 196
       exe: fms_ACCESS-CM.x
       input:
         # Biogeochemistry
diff --git a/ocean/input.nml b/ocean/input.nml
index cf8c8d2..bfd175c 100644
--- a/ocean/input.nml
+++ b/ocean/input.nml
  &ocean_model_nml
-      layout = 18,10
+      layout = 14,14
       time_tendency='twolevel'
       vertical_coordinate='zstar'
       dt_ocean = 3600

This changes the UM request to 208 cores with 16x13 layout (208 cores equals two sapphirerapids nodes), and MOM to 196 cores with a 14x14 layout.

Note that the results will not be bitwise identical due to the change in MOM total number of CPU cores (180 → 196) and layout (18x10 → 14x14). If that’s important, then you should not change any of the MOM settings above.

Please let me know if this works for you.

Cheers,
Manodeep

3 Likes
3

Many thanks @manodeep! I’ve marked this as the solution for now and will give it a crack when I get a moment

1 Like