# Running ACCESS-OM2 on Leonardo supercomputer **URL:** \ **Category:** Technical\ **Tags:** help, access-om2\ **Created:** [11 February 2024 19:26 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804 "2024-02-11T19:26:36Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [18 March 2024 10:29 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/21 "2024-03-18T10:29:44Z")
Hi @Aidan, The time frame is not urgent. I’m currently working on trying to compile ACCESS OM2 without spack using system wide modules available (openmpi, etc.). Whom should I ask for the files? Thanks Natalia ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [18 March 2024 10:30 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/22 "2024-03-18T10:30:58Z")
Thanks for the hint! @ChrisC28, @john_reilly, guys 😃 Do you have a kind of a summary of running with slurm? ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [19 March 2024 12:08 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/23 "2024-03-19T12:08:04Z")
Hi @Aidan, hi @harshula Am I getting right that generally I can follow these instructions to convert PBS to SLURM scheduling? [https://www.msi.umn.edu/slurm/pbs-conversion](https://www.msi.umn.edu/slurm/pbs-conversion) Thanks, Nataila ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [20 March 2024 11:16 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/24 "2024-03-20T11:16:10Z")
> [@Natalia](#): > > Whom should I ask for the files? I am looking into it. Will get back to you when we have them somewhere you can access. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [20 March 2024 11:28 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/25 "2024-03-20T11:28:32Z")
> [@Natalia](#): > > Am I getting right that generally I can follow these instructions to > convert PBS to SLURM scheduling? In theory there is minimal support for `slurm`, all you should need to do to is add ```yaml scheduler: slurm ``` to your `config.yaml` However we have not tested it ourselves, and so far `payu` has been developed and supported only a single HPC system, so there will probably be changes required. The branch in this PR shows some of the examples of changes that were required to use on a different HPC (Pawsey): > > > Wrap ldd in try/except as executables on pawsey seem to be statically linked. > …Also for the same reason don't assume LD\_LIBRARY\_PATH is set. > > Commented out call to load\_modules. Pawsey has a lot of default modules > that it relies on, so can't reliably monkey with that. > > Removed a couple of the bespoke flags Marshall added to the slurm > scheduler, and also explicitly pass through the PAYU environment > variables. Also set in the current environment, but that didn't seem > to make it through to the submitted job. > > Closes #323 ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [20 March 2024 12:15 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/26 "2024-03-20T12:15:34Z")
Thank you for the reply @Aidan I’ve been through this branch as well as through payu repo where I found slurm.py script, but still … Given the fact that payu was specifically made for Gadi probably I would need some advices on how to port all these to Leonardo. I wrote a letter to @ChrisC28 too. Waiting for the reply. Natalia ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [30 April 2024 23:41 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/27 "2024-04-30T23:41:16Z")
Hi @Natalia, can you use [globus](https://www.globus.org/) to download the input files? @Aidan we have a globus endpoint at `/g/data/ik11/globus` we could copy the files to. ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [30 April 2024 23:56 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/28 "2024-04-30T23:56:12Z")
ACCESS-OM2 has been coupled to an active atmospheric model (UKMO’s UM) in the ACCESS-CM2 climate model, so it seems like it should be possible (but probably not easy) to couple ACCESS-OM2 to your SPEEDY atmosphere instead of YATM. There are differences between the ACCESS-OM2 and ACCESS-CM2 versions of CICE to handle different ways to calculate surface fluxes between the model components. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [1 May 2024 01:30 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/29 "2024-05-01T01:30:31Z")
> [@aekiss](#): > > Hi @Natalia, can you use [globus](https://www.globus.org/) to download the input files? I already provided the inputs (uploaded them to the ARDC object store). But good to know about the globus endpoint. Cool! ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [1 May 2024 01:33 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/30 "2024-05-01T01:33:29Z")
> [@aekiss](#): > > There are differences between the ACCESS-OM2 and ACCESS-CM2 versions of CICE to handle different ways to calculate surface fluxes between the model components. I believe those differences are largely dictated by the UM atmosphere wanting to do things in a specific way. If SPEEDY doesn’t want to do it that way then perhaps ACCESS-OM2 is a better choice? ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [1 May 2024 04:58 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/31 "2024-05-01T04:58:35Z")
Hi @Natalia. Have you managed to compile ACCESS-OM2? Have you made any progress with getting Payu running on your system? ---
**Author:** ![harshula](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/harshula/32/256_2.png) [@harshula](https://forum.access-hive.org.au/u/harshula)\ **Post date:** [1 May 2024 16:27 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/32 "2024-05-01T16:27:45Z")
> [@dougiesquire](#): > > Hi @Natalia. Have you managed to compile ACCESS-OM2? Back in March: > [@harshula](#): > > ACCESS-OM2 was built successfully with openmpi 4.1.4 via Spack. ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [1 May 2024 20:54 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/33 "2024-05-01T20:54:41Z")
Apologies, I was refering to this from a little later: > [@Natalia](#): > > I’m currently working on trying > to compile ACCESS OM2 without spack using system wide modules available (openmpi, etc.) ---
**Author:** ![harshula](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/harshula/32/256_2.png) [@harshula](https://forum.access-hive.org.au/u/harshula)\ **Post date:** [1 May 2024 23:30 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/34 "2024-05-01T23:30:13Z")
Hi Dougie, > [@Natalia](#): > > I’m currently working on trying > to compile ACCESS OM2 without spack using system wide modules available (openmpi, etc.) IIRC, Apparently there were difficulties with purging environment settings and that was abandoned. That led to attempting to build on a laptop with gfortran. That is still in-progress. ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [2 May 2024 08:51 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/35 "2024-05-02T08:51:57Z")
Hi Andrew @aekiss, it is good to know regarding the UKMO’s model, will definitely use it as an example. ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [2 May 2024 08:58 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/36 "2024-05-02T08:58:38Z")
Hi Dougie @dougiesquire! Almost, I did compiled YATM, libaccessom2 and MOM, at this stage using local (particular src folders for each) compilation scripts and modification of environments and also fixing some of the compiler specific flags (and some other changes, that I will document with git diff once it is done). At the moment working on CICE and finally will do the whole install.sh (hopefully). It just takes time to fix all the environmental issues. ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [2 May 2024 09:18 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/37 "2024-05-02T09:18:56Z")
Will check it now, have never seen globus before. Thank you! ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [1 June 2024 10:57 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/38 "2024-06-01T10:57:45Z")
Dear all, @dougiesquire, @Aidan, @aekiss, @harshula Some updates: ACCESS-OM2 built and compiled on Leonardo! I did it four ways 😄 1. On Leonardo cluster from ACCESS-NRI repo together with @harshula during online debugging session and completely with spack (that builds all necessary sofware and puts it in an users’s environmental module). Thank you, @harshula! 2. On my mac M1 laptop from COSIMA repo (just compilation to get to know the structure of repo and the whole project), surprisingly GCC+last netcdf+openmpi worked out with several non trivial issues that I solved with the help of google and stack overflow (sed → gsed, ranlib applied to all \*.a librariy files and many others) 3. On Leonardo from COSIMA repo using dependencies built with spack before (module **access-om2** with openmpi, intel compiler and netcdf and other utils), main issues raised at the linking stage as the systemm was picking standard **ld** linker that instead of the linker built with spack. 4. From scratch on Leonardo using their modules (least favourable option, gives a lot of troubles). So for now I’ll be learning how to run it with SLURM and will keep this topic updated if I manage to do so. ```auto |-- accessom.yml |-- bin | |-- cice_auscom_1440x1080_480p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe | |-- cice_auscom_18x15.3600x2700_1682p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe | |-- cice_auscom_3600x2700_722p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe | |-- cice_auscom_360x300_24p_edcfa6f-modified_libaccessom2_d750b4b-modified.exe | |-- fms_ACCESS-OM_730f0bf-modified_libaccessom2_d750b4b-modified.x | |-- mppnccombine | `-- yatm_d750b4b-modified.exe ``` Natalia ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [2 June 2024 23:16 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/39 "2024-06-02T23:16:35Z")
Hi @Natalia Natalia, thanks for the update, that’s great news! By the way, `-modified` in the exe names indicates your repos have uncommitted changes. If you want the hashes in the exe names to indicate the exact sources they were built from, you should commit your changes before building. ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [5 June 2024 11:53 UTC](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804/40 "2024-06-05T11:53:19Z")
Hi @aekiss. Thank you for the advice, done ✅ I have also installed and configured globus, I guess now my globus account can be added to the uses/groups that can access endpoint on Gadi? Thanks Natalia [Previous page](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804.md?page=1) [Next page](https://forum.access-hive.org.au/t/running-access-om2-on-leonardo-supercomputer/1804.md?page=3)