Running ACCESS-OM2 on Leonardo supercomputer

Technical
, ,
21

Hi @Aidan,

The time frame is not urgent. I’m currently working on trying
to compile ACCESS OM2 without spack using system wide modules available (openmpi, etc.). Whom should I ask for the files?

Thanks
Natalia

1 Like
22

Thanks for the hint!
@ChrisC28, @john_reilly, guys :smiley: Do you have a kind of a summary of running with slurm?

23

Hi @Aidan, hi @harshula

Am I getting right that generally I can follow these instructions to
convert PBS to SLURM scheduling?

https://www.msi.umn.edu/slurm/pbs-conversion

Thanks,
Nataila

24

I am looking into it. Will get back to you when we have them somewhere you can access.

25

In theory there is minimal support for slurm, all you should need to do to is add

scheduler: slurm

to your config.yaml

However we have not tested it ourselves, and so far payu has been developed and supported only a single HPC system, so there will probably be changes required.

The branch in this PR shows some of the examples of changes that were required to use on a different HPC (Pawsey):

26

Thank you for the reply @Aidan

I’ve been through this branch as well as through payu repo where I found
slurm.py script, but still …
Given the fact that payu was specifically made for Gadi probably I would
need some advices on how to port all these to Leonardo.
I wrote a letter to @ChrisC28 too. Waiting for the reply.

Natalia

27

Hi @Natalia, can you use globus to download the input files?

@Aidan we have a globus endpoint at /g/data/ik11/globus we could copy the files to.

1 Like
28

ACCESS-OM2 has been coupled to an active atmospheric model (UKMO’s UM) in the ACCESS-CM2 climate model, so it seems like it should be possible (but probably not easy) to couple ACCESS-OM2 to your SPEEDY atmosphere instead of YATM.

There are differences between the ACCESS-OM2 and ACCESS-CM2 versions of CICE to handle different ways to calculate surface fluxes between the model components.

29

I already provided the inputs (uploaded them to the ARDC object store). But good to know about the globus endpoint. Cool!

1 Like
30

I believe those differences are largely dictated by the UM atmosphere wanting to do things in a specific way. If SPEEDY doesn’t want to do it that way then perhaps ACCESS-OM2 is a better choice?

31

Hi @Natalia. Have you managed to compile ACCESS-OM2? Have you made any progress with getting Payu running on your system?

32

Back in March:

33

Apologies, I was refering to this from a little later:

34

Hi Dougie,

IIRC, Apparently there were difficulties with purging environment settings and that was abandoned. That led to attempting to build on a laptop with gfortran. That is still in-progress.

2 Likes
35

Hi Andrew @aekiss, it is good to know regarding the UKMO’s model, will definitely use it as an example.

36

Hi Dougie @dougiesquire! Almost, I did compiled YATM, libaccessom2 and MOM, at this stage using local (particular src folders for each) compilation scripts and modification of environments and also fixing some of the compiler specific flags (and some other changes, that I will document with git diff once it is done). At the moment working on CICE and finally will do the whole install.sh (hopefully). It just takes time to fix all the environmental issues.

1 Like
37

Will check it now, have never seen globus before. Thank you!

38

Dear all, @dougiesquire, @Aidan, @aekiss, @harshula

Some updates: ACCESS-OM2 built and compiled on Leonardo!

I did it four ways :smile:

  1. On Leonardo cluster from ACCESS-NRI repo together with @harshula during online debugging session and completely with spack (that builds all necessary sofware and puts it in an users’s environmental module). Thank you, @harshula!

  2. On my mac M1 laptop from COSIMA repo (just compilation to get to know the structure of repo and the whole project), surprisingly GCC+last netcdf+openmpi worked out with several non trivial issues that I solved with the help of google and stack overflow (sed → gsed, ranlib applied to all *.a librariy files and many others)

  3. On Leonardo from COSIMA repo using dependencies built with spack before (module access-om2 with openmpi, intel compiler and netcdf and other utils), main issues raised at the linking stage as the systemm was picking standard ld linker that instead of the linker built with spack.

  4. From scratch on Leonardo using their modules (least favourable option, gives a lot of troubles).

So for now I’ll be learning how to run it with SLURM and will keep this topic updated if I manage to do so.

|-- accessom.yml
|-- bin
|   |-- cice_auscom_1440x1080_480p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe
|   |-- cice_auscom_18x15.3600x2700_1682p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe
|   |-- cice_auscom_3600x2700_722p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe
|   |-- cice_auscom_360x300_24p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe
|   |-- fms_ACCESS-OM_730f0bf-modified_libaccessom2_d750b4b-modified.x
|   |-- mppnccombine
|   `-- yatm_d750b4b-modified.exe

Natalia

1 Like
39

Hi @Natalia Natalia, thanks for the update, that’s great news!

By the way, -modified in the exe names indicates your repos have uncommitted changes. If you want the hashes in the exe names to indicate the exact sources they were built from, you should commit your changes before building.

1 Like
40

Hi @aekiss. Thank you for the advice, done :white_check_mark:

I have also installed and configured globus,
I guess now my globus account can be added to the uses/groups
that can access endpoint on Gadi?

Thanks
Natalia