# Installing ACCESS-OM2 on NeSI (New Zealand supercomputer) **URL:** \ **Category:** COSIMA\ **Tags:** help, access-om2, technical, community-help\ **Created:** [18 November 2025 03:51 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566 "2025-11-18T03:51:15Z")\ **Posts on this page:** 18\ **Page:** 3
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [11 February 2026 23:29 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/41 "2026-02-11T23:29:05Z")
Thanks Aidan. I am trying one more time then… FYI the edit to the spack.yaml is this: ```auto netcdf-c: require: - '@4.9.2~blosc' ``` For what it’s worth, one thing I’ve found useful when running the spack install is to do this: ```auto spack -d -v install > install_log.txt 2> error_log.txt ``` Because that gives me some information about what is going on with the cmake and compiler steps. It’s all a bit too hidden otherwise. ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [12 February 2026 02:59 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/42 "2026-02-12T02:59:36Z")
Update on this. Unfortunately my attempted spack build crashed again later. Something went wrong in the `access-generic-tracers` part of it, and like I said, I’m finding it too difficult to figure out what’s going wrong with my Spack attempts, so I’m leaving that for now. I have managed to build the ACCESS-OM2 on NeSI using the old COSIMA setup, so that’s a relief! I’m cautious about declaring victory too soon, but the model is now running for ~3 months without crashing yet. ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [12 February 2026 04:10 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/43 "2026-02-12T04:10:16Z")
Ok, the model seems to run fine for one year, even outputs diagnostic and restart files. At the end of the run, CICE is crashing due to some kind of MPI problem at the end of the run: ```auto MPI traceback error handler called by rank: 65 Image PC Routine Line Source cice_auscom_360x3 000000000097A917 Unknown Unknown Unknown cice_auscom_360x3 000000000081FEB2 Unknown Unknown Unknown libmpi.so.12.0.0 0000152877DF4E9B MPIR_Err_return_c Unknown Unknown libmpi.so.12.0.0 000015287826B7EA PMPI_Send Unknown Unknown libmpifort.so.12. 00001528773720E3 PMPI_SEND Unknown Unknown cice_auscom_360x3 0000000000792A1B accessom2_mod_mp_ 833 accessom2.F90 cice_auscom_360x3 0000000000411BFB cice_finalmod_mp_ 85 CICE_FinalMod.f90 cice_auscom_360x3 000000000041189E MAIN__ 76 CICE.f90 cice_auscom_360x3 0000000000411822 Unknown Unknown Unknown libc.so.6 0000152876E29590 Unknown Unknown Unknown libc.so.6 0000152876E29640 __libc_start_main Unknown Unknown cice_auscom_360x3 0000000000411725 Unknown Unknown Unknown ``` I will fight that battle another day… ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [12 February 2026 17:59 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/44 "2026-02-12T17:59:34Z")
Hi David, How does the submission line from payu looks? I have a modified version of payu for SLURM job submission that works on Leonardo (under SLURM) > **[GitHub - VanuatuN/payu at leonardo](https://github.com/VanuatuN/payu/tree/leonardo)** > > leonardo Natalia ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [12 February 2026 18:04 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/45 "2026-02-12T18:04:11Z")
It took me a while to find out that in my case (idk why) the MPI ranks were not properly communicating through InfiniBand until these systemwide variables were set before the job submission: ```auto export OMPI_MCA_btl_tcp_if_include=ib0 export MCA_IO=ompio export MCA_IO_OMPIO_NUM_AGGREGATORS=1 ``` ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [12 February 2026 22:19 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/46 "2026-02-12T22:19:43Z")
Hi Natalia! Thanks for your suggestion. The fun part here is that I’m not using OpenMPI. I’m using IntelMPI. So the OMPI flags are not directly applicable here. I don’t know for sure that it’s an mpirun problem yet, or something I did wrong with the compilation. I will probably try re-running my compile of CICE5 just to make sure I didn’t accidentally mess something up. For what it’s worth, my actual mpirun submission from within payu looks like this: ```auto mpirun -wdir /home/david.hutchin6926/00_nesi_projects/vuw04597_nobackup/david.hutchin6926/access-om2/work/1deg_COS_1m-c9062aa4/atmosphere -np 1 /home/david.hutchin6926/00_nesi_projects/vuw04597_nobackup/david.hutchin6926/access-om2/work/1deg_COS_1m-c9062aa4/atmosphere/yatm.exe : -wdir /home/david.hutchin6926/00_nesi_projects/vuw04597_nobackup/david.hutchin6926/access-om2/work/1deg_COS_1m-c9062aa4/ocean -np 64 /home/david.hutchin6926/00_nesi_projects/vuw04597_nobackup/david.hutchin6926/access-om2/work/1deg_COS_1m-c9062aa4/ocean/fms_ACCESS-OM.x : -wdir /home/david.hutchin6926/00_nesi_projects/vuw04597_nobackup/david.hutchin6926/access-om2/work/1deg_COS_1m-c9062aa4/ice -np 24 /home/david.hutchin6926/00_nesi_projects/vuw04597_nobackup/david.hutchin6926/access-om2/work/1deg_COS_1m-c9062aa4/ice/cice_auscom_360x300_24p.exe ``` I.e. no special flags at all. I’ll probably have to revisit this next week. I will also be happy to share my modifications to payu (next week)… as yet I haven’t created a new github fork for it. ---
**Author:** ![anton](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/anton/32/1836_2.png) [@anton](https://forum.access-hive.org.au/u/anton)\ **Post date:** [12 February 2026 22:57 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/47 "2026-02-12T22:57:45Z")
> [@dkhutch](#): > > `cice_auscom_360x3 0000000000792A1B accessom2_mod_mp_ 833 accessom2` This line looks quite innocuous: > It’s trying to send cice model time to `matm` to confirm that the time in the same across the model components. It might be helpful to add a call to `MPI_err_string` and write the result to stderr something like ```Fortan integer :: ierr, errcode, resultlen character(len=MPI_MAX_ERROR_STRING) :: errstr ... else if (self%my_local_pe == 0) then buf(1) = checksum call MPI_Comm_set_errhandler(MPI_COMM_WORLD,MPI_ERRORS_RETURN) call MPI_send(buf, 1, MPI_INTEGER, self%atm_ic_root, tag, & MPI_COMM_WORLD, err) if (err != MPI_SUCCESS) then call MPI_Error_string(err, errstr, resultlen) write(stderr,*) 'MPI error: ', trim(errstr)) endif endif endif ``` But it might also be crashing before returning the error code ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [12 February 2026 23:00 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/48 "2026-02-12T23:00:49Z")
> [@dkhutch](#): > > I’ll probably have to revisit this next week. I will also be happy to share my modifications to payu (next week)… as yet I haven’t created a new github fork for it. We have an unfinished project to [add HPCpy to payu](https://github.com/payu-org/payu/issues/568), one of the drivers of which is `slurm` support and better cross platform support in general. The plan is for that to get revived in the next month or so. We’d welcome testers when we have something working if you’re interested. ---
**Author:** ![Natalia](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/natalia/32/1612_2.png) [@Natalia](https://forum.access-hive.org.au/u/Natalia)\ **Post date:** [16 February 2026 15:05 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/50 "2026-02-16T15:05:58Z")
Hi @dkhutch ! Do you allocate resources before launching _mpirun_ within _payu_? I would try to use _srun_, so that the scheduler distributes processes properly. I might be wrong, but could it be that _mpirun_ launches the whole thing on the login node? It is 126 cores per node on NeSI? [Hardware - Support Documentation](https://docs.nesi.org.nz/Batch_Computing/Hardware/) On Leonardo I have 112 cores per node and my submission line looks as (truncated): > **srun** _-A ICT25\_MHPC --time=00:01:00 --partition=dcgp\_usr\_prod --chdir_ ./ntilinin/access-om2/work/1deg\_jra55\_ryf-expt-b559b2a8/atmosphere \*\*–\*\*_nodes=1 --ntasks=1 --ntasks-per-node=1_/leonardo\_scratch/large/userexternal/ntilinin/access-om2/1deg\_jra55\_ryf/ntilinin/access-om2/work/1deg\_jra55\_ryf-expt-b559b2a8/atmosphere/yatm.exe : _–partition=dcgp\_usr\_prod --chdir_./ntilinin/access-om2/work/1deg\_jra55\_ryf-expt-b559b2a8/ocean\*–nodes=8 --ntasks=256 --ntasks-per-node=32\*/leonardo\_scratch/large/userexternal/ntilinin/access-om2/1deg\_jra55\_ryf/ntilinin/access-om2/work/1deg\_jra55\_ryf-expt-b559b2a8/ocean/fms\_ACCESS-OM.x : _–partition=dcgp\_usr\_prod --chdir_./ntilinin/access-om2/work/1deg\_jra55\_ryf-expt-b559b2a8/ice _–nodes=2 --ntasks=64 --ntasks-per-node=32_ /leonardo\_scratch/large/userexternal/ntilinin/access-om2/1deg\_jra55\_ryf/ntilinin/access-om2/work/1deg\_jra55\_ryf-expt-b559b2a8/ice/cice\_auscom\_360x300\_24x1\_24p.exe etc While in config.yaml: scheduler: slurm #queue: do not set for slurm walltime: 03:00:00 jobname: 1deg\_jra55\_ryf\_bench\_cice5 mem: 200G account: ICT25\_MHPC partition: dcgp\_usr\_prod #project: do not set for slurm, creates separate folder **ncpus: 672 nnodes: 6** ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [16 February 2026 21:54 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/51 "2026-02-16T21:54:40Z")
Hi Natalia, I’m submitting the job through `sbatch` and I can see it’s making it to the compute nodes ok. Then it does the `payu-run` script from the compute nodes which calls either `mpirun` or `mpiexec` (I’ve tried both… they seem to do the same thing). But, I am going to follow up with NeSI support just to make sure I’m not missing some important feature of the job submission. It’s curious that your method uses srun instead of mpirun. I don’t know if that would work on mine as it needs to find the correct `mpirun` wrapper for my compiler but I could check. ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [17 February 2026 04:10 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/52 "2026-02-17T04:10:12Z")
To update you on my payu hacks, here is my forked version of payu/1.1 that I am using on Nesi: > **[GitHub - dkhutch/payu at nesi](https://github.com/dkhutch/payu/tree/nesi)** > > nesi Currently it’s pretty messy. The reason I chose v1.1 is because that was the last one I could find that still had the `bin` folder with `payu-run` and `payu` shell commands. After that it I think goes into `singularity` to run the shell commands instead… which I couldn’t make sense of so I skipped it. ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [20 February 2026 01:16 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/53 "2026-02-20T01:16:39Z")
Ok… so I couldn’t figure out how to fix my deprecated build that was crashing at the end of CICE5. I did however go back and try again with the spack installation based on Harshula’s instructions: > [@How to build ACCESS-OM2 on NeSI HPC](https://forum.access-hive.org.au/t/how-to-build-access-om2-on-nesi-hpc/5623): > > How to build ACCESS-OM2 on NeSI HPC (v2) [18/12/2025: Add instructions on how to use the oneapi compiler] Ensure that the initial setup requirements have been completed: [Set up Spack for building ACCESS models on NeSI HPC](https://forum.access-hive.org.au/t/set-up-spack-for-building-access-models-on-nesi-hpc/5622) Login to mahuika cd to the parent directory of the Spack instance. Enable Spack: . spack-config/spack-enable.bash Choose the compiler: a) To use gcc, run: git clone https://github.com/ACCESS-NRI/ACCESS-OM2 --branch api-v2-nesi-gcc b) To use oneapi, run: spa… **And I got it to run successfully with no error**! I think I must have stuffed up the mpirun call earlier, and the PMI and SLURM issues are resolved by calling the correct path for mpirun. So, I may have just run around in a giant circle with my alternative build, but never mind. So, **thank you** to @harshula for providing this build recipe. In the end spack wins. I did however use a bunch of `spack develop` steps to download the code packages and edit the AVX2 vectorization flags. My spack.yaml is attached: [spack.yaml](https://forum.access-hive.org.au/uploads/short-url/3gwSmtADOKKZMuvG0VCLcmADr8Q.yaml) (1.9 KB) In each of my “develop” packages, I just did a recursive grep for -xCORE-AVX2 or -axCORE-AVX2 and then updated that flag to -mavx2 to avoid the Intel / AMD problem. Mostly was updating CMakeLists.txt. Strangely, some of my edits (e.g. in the CICE5 build) got ignored, and the -axCORE-AVX2 persisted in the build. This didn’t seem to matter (I don’t know why). It did matter for the MOM5 build, and my edits to CMakeLists.txt did change the compilation to avoid the -xCORE-AVX2 flag. Regards, David ---
**Author:** ![harshula](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/harshula/32/256_2.png) [@harshula](https://forum.access-hive.org.au/u/harshula)\ **Post date:** [20 February 2026 03:32 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/54 "2026-02-20T03:32:37Z")
Hi @dkhutch , That’s great news, well done! Can you please create PRs for ACCESS-NRI repositories that you had to modify so that we can merge your changes? e.g. ```auto develop: access-fms: spec: access-fms@git.mom5-2025.05.000=mom5 access-generic-tracers: spec: access-generic-tracers@=2025.09.000 access-mocsy: spec: access-mocsy@=2025.07.002 libaccessom2: spec: libaccessom2@git.2025.05.001=access-om2 mom5: spec: mom5@git.2025.08.000=access-om2 cice5: spec: cice5@git.2025.03.001=access-om2 oasis3-mct: spec: oasis3-mct@git.2025.03.001=2025.03.001 ``` Thanks! Alternatively, if you provide us the diffs, we can incorporate the changes. ---
**Author:** ![dkhutch](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dkhutch/32/510_2.png) [@dkhutch](https://forum.access-hive.org.au/u/dkhutch)\ **Post date:** [20 February 2026 05:23 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/55 "2026-02-20T05:23:16Z")
Ok, so I made a bunch of pull requests for the changes I implemented. Apologies if I did anything silly in my PRs… I am not so familiar with doing those. ---
**Author:** ![harshula](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/harshula/32/256_2.png) [@harshula](https://forum.access-hive.org.au/u/harshula)\ **Post date:** [22 February 2026 05:52 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/56 "2026-02-22T05:52:58Z")
Assigned @harshula to post on 22 February 2026 05:52 UTC placeholder ---
**Author:** ![harshula](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/harshula/32/256_2.png) [@harshula](https://forum.access-hive.org.au/u/harshula)\ **Post date:** [22 February 2026 05:53 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/57 "2026-02-22T05:53:19Z")
Unassigned @harshula from post on 22 February 2026 05:53 UTC placeholder ---
**Author:** ![harshula](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/harshula/32/256_2.png) [@harshula](https://forum.access-hive.org.au/u/harshula)\ **Post date:** [22 February 2026 05:54 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/58 "2026-02-22T05:54:04Z")
Assigned @harshula on 22 February 2026 05:54 UTC ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [2 March 2026 04:59 UTC](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566/59 "2026-03-02T04:59:56Z")
@dkhutch, just a note that the latest release of ACCESS-OM2 ([2026.02.001](https://github.com/ACCESS-NRI/ACCESS-OM2/releases/tag/2026.02.001)) includes fixes to MOM5 and libaccessom2 to allow compiling and running with GCC. Note that this version of ACCESS-OM2 **does not** reproduce answers from the previous version. The latent heat of vapourisation used by MOM was changed slightly to be consistent with what is used in CICE. [Previous page](https://forum.access-hive.org.au/t/installing-access-om2-on-nesi-new-zealand-supercomputer/5566.md?page=2)