How to build ACCESS-OM3 on Setonix

harshula · 14 November 2025 01:07

How to build ACCESS-OM3 on Setonix (v2)

[18/11/2025: Refer to How to set up access-spack-packages on Setonix to build ACCESS models]

Ensure that the initial setup requirements have been completed: How to set up access-spack-packages on Setonix to build ACCESS models
Login to Setonix
module load spack/0.23.1
cd /software/projects/$PAWSEY_PROJECT/$USER
Build ACCESS-OM3 (initialisation may take a long time without any output):

git clone https://github.com/ACCESS-NRI/ACCESS-OM3.git --branch api-v1-setonix
spack env activate -p ./ACCESS-OM3
spack concretize -f --fresh
spack install

cbull · 14 November 2025 01:32

Compiling OM3

So far…

cbull@setonix-01:~> cd /software/projects/$PAWSEY_PROJECT/$USER
cbull@setonix-01:/software/projects/pawsey0889/cbull> module load spack/0.23.1
Note: when this module is unloaded, the shell environment will NOT revert to its original state, and retain some Spack settings. If you need the original shell environment, start a new
shell session instead.
cbull@setonix-01:/software/projects/pawsey0889/cbull> git clone https://github.com/ACCESS-NRI/ACCESS-OM3.git --branch api-v1-setonix
Cloning into 'ACCESS-OM3'...
remote: Enumerating objects: 950, done.
remote: Counting objects: 100% (344/344), done.
remote: Compressing objects: 100% (128/128), done.
remote: Total 950 (delta 314), reused 220 (delta 216), pack-reused 606 (from 3)
Receiving objects: 100% (950/950), 274.73 KiB | 1.29 MiB/s, done.
Resolving deltas: 100% (587/587), done.
cbull@setonix-01:/software/projects/pawsey0889/cbull> git clone https://github.com/ACCESS-NRI/access-spack-packages.git --branch api-v1-setonix
Cloning into 'access-spack-packages'...
remote: Enumerating objects: 2586, done.
remote: Counting objects: 100% (239/239), done.
remote: Compressing objects: 100% (84/84), done.
remote: Total 2586 (delta 159), reused 199 (delta 133), pack-reused 2347 (from 2)
Receiving objects: 100% (2586/2586), 558.63 KiB | 2.16 MiB/s, done.
Resolving deltas: 100% (1371/1371), done.
cbull@setonix-01:/software/projects/pawsey0889/cbull> spack repo add access-spack-packages
==> Added repo with namespace 'access.nri'.
cbull@setonix-01:/software/projects/pawsey0889/cbull> spack env create om3 ACCESS-OM3/spack.yaml
==> Created environment om3 in: /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3
==> Activate with: spack env activate om3

==> Warning: upstream not found: /software/projects/pawsey0889/setonix/2025.08/software/.spack-db/index.json
cbull@setonix-01:/software/projects/pawsey0889/cbull> spack env activate -p om3
==> Warning: upstream not found: /software/projects/pawsey0889/setonix/2025.08/software/.spack-db/index.json
[om3] cbull@setonix-01:/software/projects/pawsey0889/cbull> spack concretize -f --fresh
==> Warning: upstream not found: /software/projects/pawsey0889/setonix/2025.08/software/.spack-db/index.json
==> Warning: upstream not found: /software/projects/pawsey0889/setonix/2025.08/software/.spack-db/index.json
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v1/blobs/sha256:29fda6eb0e8149d3fb662e9e3047fc49584c940c6450c56b36cadac0f59f0300
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v1/blobs/sha256:a572ab6db954f4a850d1292bb1ef6d6055916784a894d149d657996fa98d0367
==> Installing "clingo-bootstrap@=spack%gcc@=10.2.1~docs+ipo+optimized+python+static_libstdcpp build_system=cmake build_type=Release generator=make patches=bebb819,ec99431 arch=linux-centos7-x86_64" from a buildcache
==> Warning: using "curl@8.6.0" which is a deprecated version
==> Concretized 1 spec:
 -   nw2teqx  access-om3@git.2025.08.001=2025.08.001%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=bundle arch=linux-sles15-zen3
 -   iu4ibaa      ^access3@2025.08.000%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' ~ipo build_system=cmake build_type=Release configurations=MOM6,MOM6-CICE6,MOM6-CICE6-WW3 generator=make arch=linux-sles15-zen3
 -   65x24fz          ^access-cice@CICE6.6.1-0%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' +access3~ipo~openmp build_system=cmake build_type=Release driver=none generator=make io_type=PIO arch=linux-sles15-zen3
 -   ihwnawx              ^parallelio@2.6.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +fortran~ipo~logging+mpi~ncint~pnetcdf+shared~timing build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   dod3pr3          ^access-mom6@2025.07.000%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' +access3~asymmetric_mem~ipo~openmp build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   mrz4b6u              ^access-generic-tracers@2025.08.000%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' ~ipo~shared~use_access_fms build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   hlhyz4u                  ^access-mocsy@2025.07.002%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' ~ipo~shared build_system=cmake build_type=RelWithDebInfo generator=make precision=2 arch=linux-sles15-zen3
 -   vlcfotv              ^fms@2025.03%gcc@14.2.0 cflags=-O3 cppflags='-O3 -DMAXFIELDMETHODS_=600' cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~gfs_phys+internal_file_nml~ipo+large_file~openmp~pic~quad_precision~yaml build_system=cmake build_type=Release constants=GFDL generator=make precision=32,64 arch=linux-sles15-zen3
 -   6qigqni              ^netcdf-fortran@4.6.1%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~doc+pic+shared build_system=autotools arch=linux-sles15-zen3
 -   jvysvf2          ^access-ww3@2025.08.000%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +access3~ipo~openmp build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   7eiveac          ^access3-share@2025.08.000%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' ~ipo~openmp build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   6kwyyow              ^fortranxml@4.1.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  jygykst          ^cmake@3.30.5%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release patches=dbc3892 arch=linux-sles15-zen3
[e]  sl4qb2d              ^curl@8.6.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen3
[^]  ukwxigu              ^ncurses@6.5%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-sles15-zen3
[e]  xli5wec                  ^pkg-config@0.29.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +internal_glib build_system=autotools arch=linux-sles15-zen3
[^]  mqzqbcq              ^zlib-ng@2.2.1%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-sles15-zen3
[e]  a5jqbog          ^cray-mpich@8.1.32%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +wrappers build_system=generic arch=linux-sles15-zen3
 -   ppkeqdu          ^esmf@8.7.0%gcc@14.2.0 cflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' cppflags=-O3 cxxflags='-O3 -march=sapphirerapids -mtune=sapphirerapids -unroll' fflags='-O3 -fallow-argument-mismatch -march=sapphirerapids -mtune=sapphirerapids -unroll' ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf~python+shared~xerces build_system=makefile esmf_comm=auto esmf_os=auto esmf_pio=auto patches=f63d405 snapshot=none arch=linux-sles15-zen3
 -   di3viab              ^netcdf-c@4.9.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +blosc~byterange~dap~fsync+hdf4~ipo~jna~logging+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared+szip+zstd build_system=cmake build_type=RelWithDebInfo generator=make patches=0161eb8,3b09181 arch=linux-sles15-zen3
[^]  kfuf6os                  ^bzip2@1.0.8%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~debug~pic+shared build_system=generic arch=linux-sles15-zen3
[^]  ci4uve7                      ^diffutils@3.10%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  nktjd7s                  ^c-blosc@1.21.5%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +avx2~ipo build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
[^]  qovlzcb                      ^lz4@1.10.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +pic build_system=makefile libs=shared,static arch=linux-sles15-zen3
[^]  2z3bpqa                      ^snappy@1.2.1%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~ipo+pic+shared build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
[^]  ckxxi3s                  ^hdf@4.2.15%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +external-xdr~fortran~java~netcdf+pic+shared~szip build_system=autotools patches=49733dd,5434f29,ce75518,d54592d,f7d67e8 arch=linux-sles15-zen3
[^]  35hyzgc                      ^bison@3.8.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~color build_system=autotools arch=linux-sles15-zen3
[^]  3wlkb34                      ^flex@2.6.3%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +lex~nls build_system=autotools arch=linux-sles15-zen3
[^]  nahfbta                          ^findutils@4.9.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools patches=440b954 arch=linux-sles15-zen3
[^]  dazlbri                      ^libjpeg-turbo@3.0.3%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~ipo~jpeg8~partial_decoder+pic build_system=cmake build_type=Release generator=make libs=shared,static arch=linux-sles15-zen3
[^]  litr5ri                          ^nasm@2.16.03%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  34smd2y                      ^libtirpc@1.3.3%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  qpxoj2t                          ^krb5@1.21.3%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +shared build_system=autotools arch=linux-sles15-zen3
[e]  g2fhp7a                              ^openssl@3.1.4%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~docs+shared build_system=generic certs=mozilla arch=linux-sles15-zen3
[^]  yf5a6v5                  ^hdf5@1.14.5%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~cxx+fortran+hl~ipo~java~map+mpi+shared~subfiling+szip~threadsafe+tools api=v112 build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
[^]  gtrucs4                  ^libaec@1.0.6%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~ipo+shared build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
[^]  5fhjwl3                  ^m4@1.4.19%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-sles15-zen3
[^]  mt6plhr                      ^libsigsegv@2.14%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  32cfurx                  ^zstd@1.5.6%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +programs build_system=makefile compression=none libs=shared,static arch=linux-sles15-zen3
 -   df5eiqv              ^py-pyyaml@6.0.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +libyaml build_system=python_pip arch=linux-sles15-zen3
[e]  ma2mcxr                  ^libyaml@0.1.7%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  35suhcg                  ^py-cython@3.0.11%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=python_pip arch=linux-sles15-zen3
[^]  f36jkhv                  ^py-pip@23.1.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=generic arch=linux-sles15-zen3
[^]  z2oi3cj                  ^py-setuptools@59.4.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=generic arch=linux-sles15-zen3
[^]  vq74grw                  ^py-wheel@0.41.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=generic arch=linux-sles15-zen3
[^]  m2n7szy                  ^python-venv@1.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=generic arch=linux-sles15-zen3
[^]  qbfq4gi              ^python@3.11.6%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis+optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3~ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-sles15-zen3
[^]  jumrpbz                  ^expat@2.6.4%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +libbsd build_system=autotools arch=linux-sles15-zen3
[^]  pxijb7l                      ^libbsd@0.12.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  4zoe4mx                          ^libmd@1.0.4%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  3h74ofb                  ^gdbm@1.23%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  2whuzu2                  ^gettext@0.22.5%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-sles15-zen3
[^]  njnwstq                      ^libiconv@1.17%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools libs=shared,static arch=linux-sles15-zen3
[^]  kn23dmb                      ^libxml2@2.13.4%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +pic~python+shared build_system=autotools arch=linux-sles15-zen3
[^]  oo2eqqg                      ^tar@1.34%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools zip=pigz arch=linux-sles15-zen3
[^]  sjzyf2a                          ^pigz@2.8%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=makefile arch=linux-sles15-zen3
[^]  tffygrs                  ^libffi@3.4.6%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  d7up7xp                  ^libxcrypt@4.4.35%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~obsolete_api build_system=autotools patches=4885da3 arch=linux-sles15-zen3
[^]  7pxyvrk                      ^perl@5.40.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +cpanm+opcode+open+shared+threads build_system=generic arch=linux-sles15-zen3
[^]  mkfwww6                          ^berkeley-db@18.1.40%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-sles15-zen3
[^]  dmmw2we                  ^readline@8.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools patches=bbf97f1 arch=linux-sles15-zen3
[^]  ub6ftsc                  ^sqlite@3.46.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' +column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-sles15-zen3
[^]  vloo6pu                  ^util-linux-uuid@2.40.2%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3
[^]  rzuagzv                  ^xz@5.4.6%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~pic build_system=autotools libs=shared,static arch=linux-sles15-zen3
[^]  n4mzbrz          ^gmake@4.4.1%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch' ~guile build_system=generic arch=linux-sles15-zen3
[^]  lcqimel      ^gcc-runtime@14.2.0%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=generic arch=linux-sles15-zen3
[e]  kq4yvye      ^glibc@2.38%gcc@14.2.0 cflags=-O3 cppflags=-O3 cxxflags=-O3 fflags='-O3 -fallow-argument-mismatch'  build_system=autotools arch=linux-sles15-zen3

==> Updating view at /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view

Then the install step

[om3] cbull@setonix-01:/software/projects/pawsey0889/cbull> spack install
==> Warning: upstream not found: /software/projects/pawsey0889/setonix/2025.08/software/.spack-db/index.json
[+] /usr (external curl-8.6.0-sl4qb2dyuzgnrr5qvnfcxvuyibejdare)
[+] / (external glibc-2.38-kq4yvyeeueb43pw2gpecw4zbq7v6dtvu)
==> cray-mpich@8.1.32 : has external module in ['cray-mpich/8.1.32', 'craype-network-ofi']
[+] /opt/cray/pe/mpich/8.1.32/ofi/gnu/12.3 (external cray-mpich-8.1.32-a5jqbog3kruhjbqtmjzs6gwmvwgbuwxj)
[+] /usr (external openssl-3.1.4-g2fhp7anq73hyomxcdndqi5g2swxnsif)
[+] /usr (external pkg-config-0.29.2-xli5wecpnhekyoboorrn5kcuj2i2jluq)
[+] /usr (external libyaml-0.1.7-ma2mcxrhybmypbc2a32w2fs7svwv426a)
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/bzip2-1.0.8-kfuf6os5dk32gezabo775mjneroqgtmi
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/zstd-1.5.6-32cfurxs4cvlxh4327v4m5vbg3atr3yr
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libaec-1.0.6-gtrucs4ow5dylpkbrz5d4wy56yedmaiv
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libiconv-1.17-njnwstq3buh4oevyto5glzy3x7sqhj36
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/util-linux-uuid-2.40.2-vloo6pubnobsd5zjmock7brozrr3u3e7
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/zlib-ng-2.2.1-mqzqbcqdqvpnteot62mwtxcbduldaxp7
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libjpeg-turbo-3.0.3-dazlbri7rs46332wqaeomfskigp76cyv
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libxcrypt-4.4.35-d7up7xpk5vjef7ldyczbuc6ti6lmgokk
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/snappy-1.2.1-2z3bpqagig4o2b2raxs4f6rmcy2m5v6g
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/ncurses-6.5-ukwxiguvkceewckyyv4hw54o7l7v2n2o
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libsigsegv-2.14-mt6plhrbfpqqsqtpjp5rxhfzah6kezpk
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gmake-4.4.1-n4mzbrz4wgwbu3bsh5hysivvkjyc7orb
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/m4-1.4.19-5fhjwl3g5ijbrgeryvxcnl4acesh7f5v
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/pigz-2.8-sjzyf2aaumxyaptiyz4xl2vwgxehxcnq
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/cmake-3.30.5-jygyksts5elbfbkne45cr3quznf3cjxr
==> Installing fortranxml-4.1.2-6kwyyowujc3ajklcyv33n4ejyeb5dfch [23/60]
==> No binary for fortranxml-4.1.2-6kwyyowujc3ajklcyv33n4ejyeb5dfch found: installing from source
==> Fetching https://github.com/andreww/fox/tarball/4.1.2
==> No patches needed for fortranxml
==> fortranxml: Executing phase: 'autoreconf'
==> fortranxml: Executing phase: 'configure'
==> fortranxml: Executing phase: 'build'
==> fortranxml: Executing phase: 'install'
==> fortranxml: Successfully installed fortranxml-4.1.2-6kwyyowujc3ajklcyv33n4ejyeb5dfch
  Stage: 5.00s.  Autoreconf: 0.00s.  Configure: 14.49s.  Build: 26.22s.  Install: 3.44s.  Post-install: 0.28s.  Total: 50.81s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/fortranxml-4.1.2-6kwyyowujc3ajklcyv33n4ejyeb5dfch
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/readline-8.2-dmmw2wewlpfwzhfr27cxziouwyg36fja
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/lz4-1.10.0-qovlzcbgmyzfg5mrfpvq3daumktcphxa
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libmd-1.0.4-4zoe4mxrcgs6qqp33v3or2cov46rjmjh
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libffi-3.4.6-tffygrsy53vvf22tb2izuu6ozfgnquv7
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/xz-5.4.6-rzuagzvd4mrgzn4gmgwda72nuiqxjwgg
==> Installing access-mocsy-2025.07.002-hlhyz4uoqsqhap4c42hpzjpsy6hkqqfg [29/60]
==> No binary for access-mocsy-2025.07.002-hlhyz4uoqsqhap4c42hpzjpsy6hkqqfg found: installing from source
==> No patches needed for access-mocsy
==> access-mocsy: Executing phase: 'cmake'
==> access-mocsy: Executing phase: 'build'
==> access-mocsy: Executing phase: 'install'
==> access-mocsy: Successfully installed access-mocsy-2025.07.002-hlhyz4uoqsqhap4c42hpzjpsy6hkqqfg
  Stage: 5.41s.  Cmake: 2.75s.  Build: 10.85s.  Install: 0.54s.  Post-install: 0.20s.  Total: 20.02s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access-mocsy-2025.07.002-hlhyz4uoqsqhap4c42hpzjpsy6hkqqfg
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/hdf5-1.14.5-yf5a6v5ydtegwuxmzf775deq52235a63
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gdbm-1.23-3h74ofbjfhkhm6cdpoztizefixui4xue
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/sqlite-3.46.0-ub6ftscnmofndwymtjwhlt3tknjrdzzm
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/c-blosc-1.21.5-nktjd7sm24fmit6jep4yl3mwdkdwqogo
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libbsd-0.12.2-pxijb7l2q55qcnh6v6i7eqcmodpgy65y
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libxml2-2.13.4-kn23dmbzlj3kakprw3c45dftt4pg52il
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/tar-1.34-oo2eqqgmdtktvlofny5rdocqwoxlg565
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/expat-2.6.4-jumrpbzexd5mvupy7gtiovlb6h2yrdrr
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gettext-0.22.5-2whuzu2omsfzbbd5lk4v7hfnxdexi3ju
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/python-3.11.6-qbfq4gia36gtjz7mhw7moqn2xuxsgdkv
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/krb5-1.21.3-qpxoj2tn7vjc3s3ru5nz7hh2c5w4wpul
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/python-venv-1.0-m2n7szydtgb7bpi6wgox2eybyfnm653r
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libtirpc-1.3.3-34smd2ydzuojiq3lklsnp4ib6vzmynhq
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/py-setuptools-59.4.0-z2oi3cjomim5kh6luxt6nod7hicqoc2v
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/py-pip-23.1.2-f36jkhvqk3v24dayeabio3picqugcmvw
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/hdf-4.2.15-ckxxi3syyundk6oqliejy53hady43svd
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/py-wheel-0.41.2-vq74grwh6mneus5jubwbto4xf4mijv7u
==> Installing netcdf-c-4.9.2-di3viabtlfd5q4pimzfnjnq3v26sciju [47/60]
==> No binary for netcdf-c-4.9.2-di3viabtlfd5q4pimzfnjnq3v26sciju found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/bc/bc104d101278c68b303359b3dc4192f81592ae8640f1aee486921138f7f88cb7.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/01/0161eb870fdfaf61be9d70132c9447a537320342366362e76b8460c823bf95ca
==> Applied patch /software/setonix/2025.08/spack/var/spack/repos/pawsey/packages/netcdf-c/netcdfc_correct_and_export_link_interface.patch
==> Applied patch https://github.com/Unidata/netcdf-c/commit/f8904d5a1d89420dde0f9d2c0e051ba08d08e086.patch?full_index=1
==> netcdf-c: Executing phase: 'cmake'
==> netcdf-c: Executing phase: 'build'
==> netcdf-c: Executing phase: 'install'
==> netcdf-c: Successfully installed netcdf-c-4.9.2-di3viabtlfd5q4pimzfnjnq3v26sciju
  Stage: 11.47s.  Cmake: 1m 58.27s.  Build: 42.46s.  Install: 4.45s.  Post-install: 0.44s.  Total: 2m 58.28s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/netcdf-c-4.9.2-di3viabtlfd5q4pimzfnjnq3v26sciju
[+] /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/py-cython-3.0.11-35suhcge2l474lb65g2ydfgnq5k3fahd
==> Installing netcdf-fortran-4.6.1-6qigqniy4t5a5pvjnfhy5yrwimyhuhjy [49/60]
==> No binary for netcdf-fortran-4.6.1-6qigqniy4t5a5pvjnfhy5yrwimyhuhjy found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/b5/b50b0c72b8b16b140201a020936aa8aeda5c79cf265c55160986cd637807a37a.tar.gz
==> No patches needed for netcdf-fortran
==> netcdf-fortran: Executing phase: 'autoreconf'
==> netcdf-fortran: Executing phase: 'configure'
==> netcdf-fortran: Executing phase: 'build'
==> netcdf-fortran: Executing phase: 'install'
==> netcdf-fortran: Successfully installed netcdf-fortran-4.6.1-6qigqniy4t5a5pvjnfhy5yrwimyhuhjy
  Stage: 1.70s.  Autoreconf: 0.00s.  Configure: 56.52s.  Build: 1m 14.19s.  Install: 1.43s.  Post-install: 0.39s.  Total: 2m 15.27s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/netcdf-fortran-4.6.1-6qigqniy4t5a5pvjnfhy5yrwimyhuhjy
==> Installing py-pyyaml-6.0.2-df5eiqvebj5owzno7v26xt7v6q3efrm6 [50/60]
==> No binary for py-pyyaml-6.0.2-df5eiqvebj5owzno7v26xt7v6q3efrm6 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/d5/d584d9ec91ad65861cc08d42e834324ef890a082e591037abe114850ff7bbc3e.tar.gz
==> No patches needed for py-pyyaml
==> py-pyyaml: Executing phase: 'install'
==> py-pyyaml: Successfully installed py-pyyaml-6.0.2-df5eiqvebj5owzno7v26xt7v6q3efrm6
  Stage: 3.72s.  Install: 19.27s.  Post-install: 0.44s.  Total: 24.02s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/py-pyyaml-6.0.2-df5eiqvebj5owzno7v26xt7v6q3efrm6
==> Installing parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv [51/60]
==> No binary for parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/c3/c318894f0230197458917e932ec66301b4407a744df481e9c6a6d9d85f7e5ab1.tar.gz
==> Moving resource stage
        source: /scratch/pawsey0889/cbull/setonix/2025.08/software/cbull/build_stage/resource-genf90-ihwnawxlyn2bzxk33dmaxxs2lug57vlv/spack-src/
        destination: /scratch/pawsey0889/cbull/setonix/2025.08/software/cbull/build_stage/spack-stage-parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv/spack-src/genf90
==> Moving resource stage
        source: /scratch/pawsey0889/cbull/setonix/2025.08/software/cbull/build_stage/resource-CMake_Fortran_utils-ihwnawxlyn2bzxk33dmaxxs2lug57vlv/spack-src/
        destination: /scratch/pawsey0889/cbull/setonix/2025.08/software/cbull/build_stage/spack-stage-parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv/spack-src/CMake_Fortran_utils
==> No patches needed for parallelio
==> parallelio: Executing phase: 'cmake'
==> parallelio: Executing phase: 'build'
==> parallelio: Executing phase: 'install'
==> parallelio: Successfully installed parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv
  Stage: 8.49s.  Cmake: 27.83s.  Build: 7.71s.  Install: 0.28s.  Post-install: 0.40s.  Total: 47.73s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv
==> Installing fms-2025.03-vlcfotvluhrsdxite7gzm2gjka3syg5z [52/60]
==> No binary for fms-2025.03-vlcfotvluhrsdxite7gzm2gjka3syg5z found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/17/17bdbac86bd3b1e5c1c3a5496c1495e8badcd5f671c659b17921532c990f014c.tar.gz
==> No patches needed for fms
==> fms: Executing phase: 'cmake'
==> fms: Executing phase: 'build'
==> fms: Executing phase: 'install'
==> fms: Successfully installed fms-2025.03-vlcfotvluhrsdxite7gzm2gjka3syg5z
  Stage: 2.89s.  Cmake: 13.18s.  Build: 4m 24.61s.  Install: 4.17s.  Post-install: 0.69s.  Total: 4m 47.04s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/fms-2025.03-vlcfotvluhrsdxite7gzm2gjka3syg5z
==> Installing esmf-8.7.0-ppkeqdu6fovqwydnyqwehiyxecx7po5s [53/60]
==> No binary for esmf-8.7.0-ppkeqdu6fovqwydnyqwehiyxecx7po5s found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/d7/d7ab266e2af8c8b230721d4df59e61aa03c612a95cc39c07a2d5695746f21f56.tar.gz
==> Applied patch /software/setonix/2025.08/spack/var/spack/repos/builtin/packages/esmf/esmf_cpp_info.patch
==> esmf: Executing phase: 'edit'
==> esmf: Executing phase: 'build'
==> esmf: Executing phase: 'install'
==> esmf: Successfully installed esmf-8.7.0-ppkeqdu6fovqwydnyqwehiyxecx7po5s
  Stage: 21.37s.  Edit: 0.00s.  Build: 29m 23.61s.  Install: 45.50s.  Post-install: 0.95s.  Total: 30m 34.12s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/esmf-8.7.0-ppkeqdu6fovqwydnyqwehiyxecx7po5s
==> Installing access-generic-tracers-2025.08.000-mrz4b6ug5nzvksspz2afovryww7qpae5 [54/60]
==> No binary for access-generic-tracers-2025.08.000-mrz4b6ug5nzvksspz2afovryww7qpae5 found: installing from source
==> No patches needed for access-generic-tracers
==> access-generic-tracers: Executing phase: 'cmake'
==> access-generic-tracers: Executing phase: 'build'
==> access-generic-tracers: Executing phase: 'install'
==> access-generic-tracers: Successfully installed access-generic-tracers-2025.08.000-mrz4b6ug5nzvksspz2afovryww7qpae5
  Stage: 5.01s.  Cmake: 7.90s.  Build: 1m 4.66s.  Install: 0.38s.  Post-install: 0.41s.  Total: 1m 19.22s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access-generic-tracers-2025.08.000-mrz4b6ug5nzvksspz2afovryww7qpae5
==> Installing access3-share-2025.08.000-7eiveacn23d5oqob3uf4ihyqqhwlhony [55/60]
==> No binary for access3-share-2025.08.000-7eiveacn23d5oqob3uf4ihyqqhwlhony found: installing from source
==> No patches needed for access3-share
==> access3-share: Executing phase: 'cmake'
==> access3-share: Executing phase: 'build'
==> access3-share: Executing phase: 'install'
==> access3-share: Successfully installed access3-share-2025.08.000-7eiveacn23d5oqob3uf4ihyqqhwlhony
  Stage: 19.64s.  Cmake: 4.26s.  Build: 1m 23.87s.  Install: 1.04s.  Post-install: 0.66s.  Total: 1m 51.37s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access3-share-2025.08.000-7eiveacn23d5oqob3uf4ihyqqhwlhony
==> Installing access-cice-CICE6.6.1-0-65x24fzwognzrz4g2celi7in4jfeffnn [56/60]
==> No binary for access-cice-CICE6.6.1-0-65x24fzwognzrz4g2celi7in4jfeffnn found: installing from source
==> No patches needed for access-cice
==> access-cice: Executing phase: 'cmake'
==> access-cice: Executing phase: 'build'
==> access-cice: Executing phase: 'install'
==> access-cice: Successfully installed access-cice-CICE6.6.1-0-65x24fzwognzrz4g2celi7in4jfeffnn
  Stage: 17.49s.  Cmake: 11.92s.  Build: 7m 51.25s.  Install: 1.00s.  Post-install: 0.59s.  Total: 8m 24.47s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access-cice-CICE6.6.1-0-65x24fzwognzrz4g2celi7in4jfeffnn
==> Installing access-ww3-2025.08.000-jvysvf22rqptbhhcbtesl5jd7alusigm [57/60]
==> No binary for access-ww3-2025.08.000-jvysvf22rqptbhhcbtesl5jd7alusigm found: installing from source
==> No patches needed for access-ww3
==> access-ww3: Executing phase: 'cmake'
==> access-ww3: Executing phase: 'build'
==> access-ww3: Executing phase: 'install'
==> access-ww3: Successfully installed access-ww3-2025.08.000-jvysvf22rqptbhhcbtesl5jd7alusigm
  Stage: 1m 21.13s.  Cmake: 6.40s.  Build: 46.74s.  Install: 0.50s.  Post-install: 0.69s.  Total: 2m 16.65s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access-ww3-2025.08.000-jvysvf22rqptbhhcbtesl5jd7alusigm
==> Installing access-mom6-2025.07.000-dod3pr3ans4iy32kqnivpmbdtz2cn54h [58/60]
==> No binary for access-mom6-2025.07.000-dod3pr3ans4iy32kqnivpmbdtz2cn54h found: installing from source
==> No patches needed for access-mom6
==> access-mom6: Executing phase: 'cmake'
==> access-mom6: Executing phase: 'build'
==> access-mom6: Executing phase: 'install'
==> access-mom6: Successfully installed access-mom6-2025.07.000-dod3pr3ans4iy32kqnivpmbdtz2cn54h
  Stage: 21.20s.  Cmake: 7.13s.  Build: 2m 21.88s.  Install: 1.79s.  Post-install: 0.61s.  Total: 2m 54.26s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access-mom6-2025.07.000-dod3pr3ans4iy32kqnivpmbdtz2cn54h
==> Installing access3-2025.08.000-iu4ibaazibp3pt2mcvysw5oys2mwilqk [59/60]
==> No binary for access3-2025.08.000-iu4ibaazibp3pt2mcvysw5oys2mwilqk found: installing from source
==> Using cached archive: /scratch/pawsey0889/cbull/setonix/2025.08/software/cbull/source_cache/_source-cache/git//ACCESS-NRI/access3-share/f2f35ce5915e82a83899b69560d826deab53b668.tar.gz
==> Warning: Using download cache instead of version control
  The required sources are normally checked out from a version control system, but have been archived in download cache: file:///scratch/pawsey0889/cbull/setonix/2025.08/software/cbull/source_cache/_source-cache/git//ACCESS-NRI/access3-share/f2f35ce5915e82a83899b69560d826deab53b668.tar.gz. Spack lacks a tree hash to verify the integrity of this archive. Make sure your download cache is in a secure location.
==> No patches needed for access3
==> access3: Executing phase: 'cmake'
==> access3: Executing phase: 'build'
==> access3: Executing phase: 'install'
==> access3: Successfully installed access3-2025.08.000-iu4ibaazibp3pt2mcvysw5oys2mwilqk
  Stage: 6.26s.  Cmake: 6.67s.  Build: 11.05s.  Install: 0.68s.  Post-install: 0.69s.  Total: 27.57s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access3-2025.08.000-iu4ibaazibp3pt2mcvysw5oys2mwilqk
==> Installing access-om3-git.2025.08.001=2025.08.001-nw2teqxom7njw5emnxuhfvifkxbhrfpe [60/60]
==> No binary for access-om3-git.2025.08.001=2025.08.001-nw2teqxom7njw5emnxuhfvifkxbhrfpe found: installing from source
==> No patches needed for access-om3
==> access-om3: Executing phase: 'install'
==> access-om3: Successfully installed access-om3-git.2025.08.001=2025.08.001-nw2teqxom7njw5emnxuhfvifkxbhrfpe
  Stage: 0.38s.  Install: 0.00s.  Post-install: 0.56s.  Total: 3.62s
[+] /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/access-om3-git.2025.08.001_2025.08.001-nw2teqxom7njw5emnxuhfvifkxbhrfpe
==> Updating view at /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view

Success!

Note spack environment is installed in:
cd /software/projects/$PAWSEY_PROJECT/$USER/setonix/2025.08/environments/

cbull · 14 November 2025 03:14

Getting the forcing files

Next step is to try and run this:

github.com/ACCESS-NRI/access-om3-configs

config.yaml

dev-MC_100km_jra_iaf

# Payu configuration

# If submitting to a different project to your default, uncomment line below 
# and change project code as appropriate; also set shortpath below
# project: x77
# Force payu to always find, and save, files in this scratch project directory
# shortpath: /scratch/v45

# Using payu sync is recommend to copy data from ephemeral scratch space to 
# longer term storage
sync:
    enable: False # set path below and change to true
    path: null # Set to location on /g/data (e.g. /g/data/PROJECT/USER/EXPERIMENT)
    restart: True

#Model software version
modules:
    use:
        - /g/data/vk83/modules
    load:

This file has been truncated. show original

Grabbing the input files

#/home/561/cyb561/repos/access-om3-configs/config.yaml
#from
#    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-ESMFmesh.nc
#    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-nomask-ESMFmesh.nc
#    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-datm-ESMFmesh.nc
#    - /g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-drof-ESMFmesh.nc
#    - /g/data/vk83/configurations/inputs/access-om3/share/grids/global.1deg/2020.10.22/topog.nc
#    - /g/data/vk83/configurations/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
#    - /g/data/vk83/prerelease/configurations/inputs/access-om3/mom/grids/vertical/share/2025.03.12/ocean_vgrid.nc
#    - /g/data/vk83/configurations/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc
#    - /g/data/vk83/configurations/inputs/access-om3/mom/surface_salt_restoring/global.1deg/2020.05.30/salt_sfc_restore.nc
#    - /g/data/vk83/configurations/inputs/access-om3/cice/grids/global.1deg/2024.05.14/kmt.nc
#    - /g/data/vk83/configurations/inputs/access-om3/cice/initial_conditions/global.1deg/2023.07.28/iced.1900-01-01-10800.nc
#    - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0

set -x
#SYNCME=rsync -avz --progress cyb561@gadi.nci.org.au:

LCLDIR=/scratch/pawsey0889/cbull/om3forcing/

mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/share/grids/global.1deg/2020.10.22/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/
mkdir -p ${LCLDIR}vk83/prerelease/configurations/inputs/access-om3/mom/grids/vertical/share/2025.03.12/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/mom/surface_salt_restoring/global.1deg/2020.05.30/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/cice/grids/global.1deg/2024.05.14/
mkdir -p ${LCLDIR}vk83/configurations/inputs/access-om3/cice/initial_conditions/global.1deg/2023.07.28/
mkdir -p ${LCLDIR}qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0

rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-ESMFmesh.nc  ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-nomask-ESMFmesh.nc         ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-datm-ESMFmesh.nc                         ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-drof-ESMFmesh.nc                         ${LCLDIR}vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/share/grids/global.1deg/2020.10.22/topog.nc                                    ${LCLDIR}vk83/configurations/inputs/access-om3/share/grids/global.1deg/2020.10.22/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc                         ${LCLDIR}vk83/configurations/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/prerelease/configurations/inputs/access-om3/mom/grids/vertical/share/2025.03.12/ocean_vgrid.nc                  ${LCLDIR}vk83/prerelease/configurations/inputs/access-om3/mom/grids/vertical/share/2025.03.12/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc           ${LCLDIR}vk83/configurations/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/mom/surface_salt_restoring/global.1deg/2020.05.30/salt_sfc_restore.nc          ${LCLDIR}vk83/configurations/inputs/access-om3/mom/surface_salt_restoring/global.1deg/2020.05.30/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/cice/grids/global.1deg/2024.05.14/kmt.nc                                       ${LCLDIR}vk83/configurations/inputs/access-om3/cice/grids/global.1deg/2024.05.14/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/vk83/configurations/inputs/access-om3/cice/initial_conditions/global.1deg/2023.07.28/iced.1900-01-01-10800.nc        ${LCLDIR}vk83/configurations/inputs/access-om3/cice/initial_conditions/global.1deg/2023.07.28/
rsync -avz --progress cyb561@gadi.nci.org.au:/g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0                                                           ${LCLDIR}qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0

The above, whilst worked was very slow, so I tried this for the JRA forcing, now going from Gadi to setonix, i.e.:

[cyb561@gadi-login-04 cyb561]$ cat push_jra.sh
#!/bin/bash -l

#to submit this
#qsub -P tm70 -q copyq -l walltime=10:00:00,ncpus=1,mem=8GB,jobfs=100GB,storage=gdata/tm70+scratch/tm70+gdata/qv56 /g/data/tm70/cyb561/push_jra.sh

#CBull
#17/11/2025

cd  /g/data/tm70/cyb561/
rsync -avz --progress /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0 cbull@data-mover.pawsey.org.au:/scratch/pawsey0889/cbull/

echo "job finished: `date`"

[cyb561@gadi-login-04 cyb561]$ qsub -P tm70 -q copyq -l walltime=10:00:00,ncpus=1,mem=8GB,jobfs=100GB,storage=gdata/tm70+scratch/tm70+gdata/qv56 /g/data/tm70/cyb561/push_jra.sh
154527107.gadi-pbs
[cyb561@gadi-login-04 cyb561]$ pwd
/g/data/tm70/cyb561

(not currently working)

See this link.

… Something is weird with the way submitted jobs are different to log-in node jobs, @joshuatorrance and I decided to run on screen on Setonix.. (note the use of Gadi’s dm)

rsync -avz --progress cyb561@gadi-dm.nci.org.au:/g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0 /scratch/pawsey0889/cbull/joshtest

cbull · 14 November 2025 03:51

Attempting to run OM3

cd https://github.com/ACCESS-NRI/
git clone https://github.com/ACCESS-NRI/access-om3-configs.git
git checkout dev-MC_100km_jra_iaf

cbull@setonix-06:~> module unload access-om3/git.2025.08.001_2025.08.001-nw2teqx
cbull@setonix-06:~> ldd /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6
        linux-vdso.so.1 (0x00007ffc7a082000)
        /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/esmf-8.7.0-ppkeqdu6fovqwydnyqwehiyxecx7po5s/lib/libesmf.so (0x000014c290b03000)
        libpiof.so => /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv/lib/libpiof.so (0x000014c290a97000)
        libpioc.so => /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/parallelio-2.6.2-ihwnawxlyn2bzxk33dmaxxs2lug57vlv/lib/libpioc.so (0x000014c290a53000)
        libnetcdff.so.7 => /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/netcdf-fortran-4.6.1-6qigqniy4t5a5pvjnfhy5yrwimyhuhjy/lib64/libnetcdff.so.7 (0x000014c290957000)
        libnetcdf.so.19 => /software/projects/pawsey0889/cbull/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/netcdf-c-4.9.2-di3viabtlfd5q4pimzfnjnq3v26sciju/lib64/libnetcdf.so.19 (0x000014c290773000)
        libgfortran.so.5 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr/lib/libgfortran.so.5 (0x000014c29046d000)
        libm.so.6 => /lib64/libm.so.6 (0x000014c290369000)
        libmvec.so.1 => /lib64/libmvec.so.1 (0x000014c29026e000)
        libstdc++.so.6 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr/lib/libstdc++.so.6 (0x000014c290018000)
        libmpifort_gnu_123.so.12 => /opt/cray/pe/mpich/8.1.32/ofi/gnu/12.3/lib/libmpifort_gnu_123.so.12 (0x000014c28ff81000)
        libxpmem.so.0 => /opt/xpmem/lib64/libxpmem.so.0 (0x000014c28ff7e000)
        libmpi_gnu_123.so.12 => /opt/cray/pe/mpich/8.1.32/ofi/gnu/12.3/lib/libmpi_gnu_123.so.12 (0x000014c28d7a4000)
        libgomp.so.1 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr/lib/libgomp.so.1 (0x000014c28d74e000)
        libgcc_s.so.1 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr/lib/libgcc_s.so.1 (0x000014c28d71f000)
        libc.so.6 => /lib64/libc.so.6 (0x000014c28d400000)
        libhugetlbfs.so => /usr/lib64/libhugetlbfs.so (0x000014c28d6e9000)
        libmfhdf.so.0 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/hdf-4.2.15-ckxxi3syyundk6oqliejy53hady43svd/lib64/libmfhdf.so.0 (0x000014c28d6b8000)
        libdf.so.0 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/hdf-4.2.15-ckxxi3syyundk6oqliejy53hady43svd/lib64/libdf.so.0 (0x000014c28d343000)
        libhdf5_hl.so.310 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/hdf5-1.14.5-yf5a6v5ydtegwuxmzf775deq52235a63/lib/libhdf5_hl.so.310 (0x000014c28d693000)
        libhdf5.so.310 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/hdf5-1.14.5-yf5a6v5ydtegwuxmzf775deq52235a63/lib/libhdf5.so.310 (0x000014c28cea9000)
        libz.so.1 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/zlib-ng-2.2.1-mqzqbcqdqvpnteot62mwtxcbduldaxp7/lib/libz.so.1 (0x000014c28d662000)
        libzstd.so.1 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/zstd-1.5.6-32cfurxs4cvlxh4327v4m5vbg3atr3yr/lib/libzstd.so.1 (0x000014c28cdb7000)
        libbz2.so.1.0 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/bzip2-1.0.8-kfuf6os5dk32gezabo775mjneroqgtmi/lib/libbz2.so.1.0 (0x000014c28d648000)
        libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x000014c28ccda000)
        libxml2.so.2 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libxml2-2.13.4-kn23dmbzlj3kakprw3c45dftt4pg52il/lib64/libxml2.so.2 (0x000014c28cb5c000)
        /lib64/ld-linux-x86-64.so.2 (0x000014c292804000)
        libfabric.so.1 => /opt/cray/libfabric/1.22.0/lib64/libfabric.so.1 (0x000014c28ca24000)
        libatomic.so.1 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr/lib/libatomic.so.1 (0x000014c28d63c000)
        libpmi.so.0 => /opt/cray/pe/lib64/libpmi.so.0 (0x000014c28d619000)
        libpmi2.so.0 => /opt/cray/pe/lib64/libpmi2.so.0 (0x000014c28ca01000)
        libquadmath.so.0 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gcc-runtime-14.2.0-lcqimelwhbmkb62p2hq53j5wg25pxvdr/lib/libquadmath.so.0 (0x000014c28c9ba000)
        libdl.so.2 => /lib64/libdl.so.2 (0x000014c28c9b7000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x000014c28c9b4000)
        libjpeg.so.62 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libjpeg-turbo-3.0.3-dazlbri7rs46332wqaeomfskigp76cyv/lib64/libjpeg.so.62 (0x000014c28c8c1000)
        libtirpc.so.3 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libtirpc-1.3.3-34smd2ydzuojiq3lklsnp4ib6vzmynhq/lib64/libtirpc.so.3 (0x000014c28c891000)
        libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x000014c28c868000)
        libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x000014c28c600000)
        libssh.so.4 => /usr/lib64/libssh.so.4 (0x000014c28c591000)
        libpsl.so.5 => /usr/lib64/libpsl.so.5 (0x000014c28c200000)
        libssl.so.3 => /usr/lib64/libssl.so.3 (0x000014c28c4db000)
        libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x000014c28bc00000)
        libgssapi_krb5.so.2 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/krb5-1.21.3-qpxoj2tn7vjc3s3ru5nz7hh2c5w4wpul/lib64/libgssapi_krb5.so.2 (0x000014c28c485000)
        libldap_r-2.4.so.2 => /usr/lib64/libldap_r-2.4.so.2 (0x000014c28c430000)
        liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x000014c28c856000)
        libbrotlidec.so.1 => /usr/lib64/libbrotlidec.so.1 (0x000014c28c847000)
        liblzma.so.5 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/xz-5.4.6-rzuagzvd4mrgzn4gmgwda72nuiqxjwgg/lib64/liblzma.so.5 (0x000014c28c1cb000)
        libiconv.so.2 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/libiconv-1.17-njnwstq3buh4oevyto5glzy3x7sqhj36/lib64/libiconv.so.2 (0x000014c28baef000)
        libcxi.so.1 => /usr/lib64/libcxi.so.1 (0x000014c28c826000)
        libpals.so.0 => /opt/cray/pals/1.6/lib/libpals.so.0 (0x000014c28c81e000)
        libkrb5.so.3 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/krb5-1.21.3-qpxoj2tn7vjc3s3ru5nz7hh2c5w4wpul/lib64/libkrb5.so.3 (0x000014c28ba17000)
        libk5crypto.so.3 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/krb5-1.21.3-qpxoj2tn7vjc3s3ru5nz7hh2c5w4wpul/lib64/libk5crypto.so.3 (0x000014c28c19a000)
        libcom_err.so.3 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/krb5-1.21.3-qpxoj2tn7vjc3s3ru5nz7hh2c5w4wpul/lib64/libcom_err.so.3 (0x000014c28c429000)
        libunistring.so.2 => /usr/lib64/libunistring.so.2 (0x000014c28b600000)
        libjitterentropy.so.3 => /usr/lib64/libjitterentropy.so.3 (0x000014c28c41f000)
        libkrb5support.so.0 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/krb5-1.21.3-qpxoj2tn7vjc3s3ru5nz7hh2c5w4wpul/lib64/libkrb5support.so.0 (0x000014c28c18c000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x000014c28c17a000)
        libintl.so.8 => /software/setonix/2025.08/software/linux-sles15-zen3/gcc-14.2.0/gettext-0.22.5-2whuzu2omsfzbbd5lk4v7hfnxdexi3ju/lib64/libintl.so.8 (0x000014c28ba00000)
        libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x000014c28b9e2000)
        libbrotlicommon.so.1 => /usr/lib64/libbrotlicommon.so.1 (0x000014c28b9c0000)
        libnl-3.so.200 => /usr/lib64/libnl-3.so.200 (0x000014c28b99d000)

To find the executable (@helen the bit you asked about)

module load access-om3/git.2025.08.001_2025.08.001-nw2teqx
module load spack/0.23.1
spack env activate -p om3
which access-om3-MOM6-CICE6

then /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6

running tips

In particular, from first link:

examples of batch scripts designed to run parallel and distributed computations making use of MPI. In this scenario, the use of the srun command is critical for the creation of many tasks on multiple nodes.

submitting…

[356][cbull.setonix-02: cbull]$ pwd
/scratch/pawsey0889/cbull

[cbull.setonix-02: cbull]$ cat runom3.sh
#!/bin/bash --login

# SLURM directives
#
# Here we specify to SLURM we want 512 tasks
# distributed by 128 tasks per node (using all available cores on 4 nodes)
# a wall-clock time limit of 24 hours
#
# Replace [your-project] with the appropriate project name
# following --account (e.g., --account=pawsey00XX)

#SBATCH --account=pawsey0889
#SBATCH --partition=work
#SBATCH --ntasks=512
#SBATCH --ntasks-per-node=128
#SBATCH --exclusive
#SBATCH --time=00:01:00


# ---
# Load here the needed modules

# ---
# Set MPI related environment variables. (Not all need to be set)
# Main variables for multi-node jobs (activate for multinode jobs)
export MPICH_OFI_STARTUP_CONNECT=1
export MPICH_OFI_VERBOSE=1
#Ask MPI to provide useful runtime information (activate if debugging)
export MPICH_ENV_DISPLAY=1
export MPICH_MEMORY_REPORT=1


# https://pawsey.atlassian.net/wiki/spaces/US/pages/51927426/Example+Slurm+Batch+Scripts+for+Setonix+on+CPU+Compute+Nodes#ExampleSlurmBatchScriptsforSetonixonCPUComputeNodes-MPIjobs

# ---
# Run the desired code:
cd /scratch/pawsey0889/cbull/access-om3-configs
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6

It not working

[cbull.setonix-02: cbull]$ cat slurm-34894388.out
srun: error: nid001879: tasks 0-2,4-7,9,11-15,18-29,31-48,51-63,65-77,79,81-111,113-115,117,119,121-123,125-127: Illegal instruction
srun: Terminating StepId=34894388.0
srun: error: nid001879: tasks 3,8,10,16-17,30,49-50,64,78,80,112,116,118,120,124: Illegal instruction (core dumped)
srun: error: nid001923: tasks 128-130,144,149,167,188,191,204-205,226,232,238,240,248,255: Illegal instruction (core dumped)
srun: error: nid001923: tasks 131-143,145-148,150-166,168-187,189-190,192-203,206-225,227-231,233-237,239,241-247,249-254: Illegal instruction
srun: error: nid001925: tasks 384-390,392-394,396-404,406-409,411-413,415-427,429-436,438-441,444-456,458-491,495,497,499-503,505-511: Illegal instruction
srun: error: nid001925: tasks 391,395,405,410,414,428,437,442-443,457,492-494,496,498,504: Illegal instruction (core dumped)
srun: error: nid001924: tasks 256-383: Terminated
srun: Force Terminated StepId=34894388.0

Or interactively on 256 cores

[cbull.nid001681: cbull]$ srun -N 2 -n 256 /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6
srun: error: nid001681: tasks 0-1,10,15,27,41-42,46,58-59,91,106-107,122-123,126: Illegal instruction (core dumped)
srun: Terminating StepId=34894765.0
srun: error: nid001681: tasks 2-9,11-14,16-26,28-40,43-45,47-57,60-90,92-105,108-121,124-125,127: Illegal instruction
srun: error: nid001738: tasks 128-129,131-164,166-175,178-181,183,186-188,190-197,199,201-212,215-222,224-228,230-231,233-237,239-255: Illegal instruction
srun: error: nid001738: tasks 130,165,176-177,182,184-185,189,198,200,213-214,223,229,232,238: Illegal instruction (core dumped)

cbull · 17 November 2025 00:15

quality of life

Interactive jobs
salloc -p work -n 1 -N 1 -c 128 --exclusive -A pawsey0889

And now asking for 256 cores across two nodes:
salloc -p work -n 256 -N 2 --exclusive -A pawsey0889

Adding setonix to ssh config

Host setonix
    Hostname setonix.pawsey.org.au
    User cbull
    ForwardX11 true
    ForwardX11Trusted yes
    IdentityFile ~/.ssh/id_rsa
    AddKeysToAgent yes
    ForwardAgent yes

cbull · 17 November 2025 00:21

Installing and using Payu

Regarding payu:

I mentioned this to Angus Gibson last week, but our (Chris C and myself) Payu scripts on Setonix are quite hacked together to solve past issues - but I’m fairly sure if I grab the latest Payu there might only be 2 or 3 lines that need changing. I can have a go at this sometime this week and get back to you. There is a specific module set required though - as the default versions cause issues with Payu. This is what I’m currently loading:

module load pawseyenv/2024.05
module load gcc/12.2.0
module load libfabric/1.15.2.0
module load rclone/1.63.1
module load netcdf-c/4.9.2
module load netcdf-fortran/4.6.1
module load python/3.11.6 py-pip/23.1.2-py3.11.6

Regarding JRA
I’ve got the JRA forcing (padded at each end temporally for IAF runs) on ACACIA. This can be accessed using RCLONE once you’ve set up the keys; E.g.,

Rclone copy acacia:pawsey0410-jra-iaf-padded/<YEAR> /scratch/pawseyXXXX/<username>/

Installing Payu

This was installing @Aidan’s old branch which I didn’t get to work…

[cbull.setonix-02: /home/cbull]$ git clone https://github.com/payu-org/payu
Cloning into 'payu'...
remote: Enumerating objects: 6972, done.
remote: Counting objects: 100% (1822/1822), done.
remote: Compressing objects: 100% (540/540), done.
remote: Total 6972 (delta 1597), reused 1287 (delta 1282), pack-reused 5150 (from 3)
Receiving objects: 100% (6972/6972), 3.31 MiB | 8.88 MiB/s, done.
Resolving deltas: 100% (4942/4942), done.
[681][cbull.setonix-02: /home/cbull]$ cd payu
[682][cbull.setonix-02: /home/cbull/payu]$ git switch pawsey
branch 'pawsey' set up to track 'origin/pawsey'.
Switched to a new branch 'pawsey'
[683][cbull.setonix-02: /home/cbull/payu]$ pip install . --user
Processing /home/cbull/payu
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting f90nml>=0.16 (from payu==1.0.19)
  Obtaining dependency information for f90nml>=0.16 from https://files.pythonhosted.org/packages/21/f2/4454eefc15cc326b46530d230c58cc0bb91a1e9797f2842b2a1720cbb233/f90nml-1.5.0-py2.py3-none-any.whl.metadata
  Downloading f90nml-1.5.0-py2.py3-none-any.whl.metadata (6.4 kB)
Collecting yamanifest>=0.3.4 (from payu==1.0.19)
  Obtaining dependency information for yamanifest>=0.3.4 from https://files.pythonhosted.org/packages/2a/5a/91816ff3c7f589f027168218fe53bc79c5e6baa76086d6e90fefc0679681/yamanifest-0.3.14-py3-none-any.whl.metadata
  Downloading yamanifest-0.3.14-py3-none-any.whl.metadata (15 kB)
Collecting PyYAML (from payu==1.0.19)
  Obtaining dependency information for PyYAML from https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB)
Collecting requests[security] (from payu==1.0.19)
  Obtaining dependency information for requests[security] from https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl.metadata
  Downloading requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting python-dateutil (from payu==1.0.19)
  Obtaining dependency information for python-dateutil from https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting tenacity!=7.0.0 (from payu==1.0.19)
  Obtaining dependency information for tenacity!=7.0.0 from https://files.pythonhosted.org/packages/e5/30/643397144bfbfec6f6ef821f36f33e57d35946c44a2352d3c9f0ae847619/tenacity-9.1.2-py3-none-any.whl.metadata
  Downloading tenacity-9.1.2-py3-none-any.whl.metadata (1.2 kB)
Collecting six (from yamanifest>=0.3.4->payu==1.0.19)
  Obtaining dependency information for six from https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl.metadata
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting xxhash (from yamanifest>=0.3.4->payu==1.0.19)
  Obtaining dependency information for xxhash from https://files.pythonhosted.org/packages/11/4f/426f91b96701ec2f37bb2b8cec664eff4f658a11f3fa9d94f0a887ea6d2b/xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Downloading xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (13 kB)
Collecting charset_normalizer<4,>=2 (from requests[security]->payu==1.0.19)
  Obtaining dependency information for charset_normalizer<4,>=2 from https://files.pythonhosted.org/packages/c0/10/d20b513afe03acc89ec33948320a5544d31f21b05368436d580dec4e234d/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Downloading charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (37 kB)
Collecting idna<4,>=2.5 (from requests[security]->payu==1.0.19)
  Obtaining dependency information for idna<4,>=2.5 from https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl.metadata
  Downloading idna-3.11-py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3<3,>=1.21.1 (from requests[security]->payu==1.0.19)
  Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/a7/c2/fe1e52489ae3122415c51f387e221dd0773709bad6c6cdaa599e8a2c5185/urllib3-2.5.0-py3-none-any.whl.metadata
  Downloading urllib3-2.5.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests[security]->payu==1.0.19)
  Obtaining dependency information for certifi>=2017.4.17 from https://files.pythonhosted.org/packages/70/7d/9bc192684cea499815ff478dfcdc13835ddf401365057044fb721ec6bddb/certifi-2025.11.12-py3-none-any.whl.metadata
  Downloading certifi-2025.11.12-py3-none-any.whl.metadata (2.5 kB)
Downloading f90nml-1.5.0-py2.py3-none-any.whl (51 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.0/52.0 kB 12.8 MB/s eta 0:00:00
Downloading tenacity-9.1.2-py3-none-any.whl (28 kB)
Downloading yamanifest-0.3.14-py3-none-any.whl (20 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 41.1 MB/s eta 0:00:00
Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (807 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 84.9 MB/s eta 0:00:00
Downloading certifi-2025.11.12-py3-none-any.whl (159 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 159.4/159.4 kB 33.9 MB/s eta 0:00:00
Downloading charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (153 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.5/153.5 kB 31.4 MB/s eta 0:00:00
Downloading idna-3.11-py3-none-any.whl (71 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.0/71.0 kB 13.8 MB/s eta 0:00:00
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Downloading urllib3-2.5.0-py3-none-any.whl (129 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.8/129.8 kB 28.2 MB/s eta 0:00:00
Downloading requests-2.32.5-py3-none-any.whl (64 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.7/64.7 kB 15.6 MB/s eta 0:00:00
Downloading xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (193 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 193.9/193.9 kB 39.3 MB/s eta 0:00:00
Building wheels for collected packages: payu
  Building wheel for payu (pyproject.toml) ... done
  Created wheel for payu: filename=payu-1.0.19-py2.py3-none-any.whl size=80914 sha256=c32be0ca68cf4625298aa0d1d3bd0acabc516d4c186d47bd86dad26b407289e1
  Stored in directory: /tmp/pip-ephem-wheel-cache-2xn7ydcb/wheels/de/b6/3b/9e30c8c57505b9a124ae5dbce71f4febecf120aa3bdc831fcb
Successfully built payu
Installing collected packages: f90nml, xxhash, urllib3, tenacity, six, PyYAML, idna, charset_normalizer, certifi, yamanifest, requests, python-dateutil, payu
  WARNING: The script f90nml is installed in '/home/cbull/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script normalizer is installed in '/home/cbull/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script yamf is installed in '/home/cbull/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts payu, payu-collate, payu-profile and payu-run are installed in '/home/cbull/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed PyYAML-6.0.3 certifi-2025.11.12 charset_normalizer-3.4.4 f90nml-1.5.0 idna-3.11 payu-1.0.19 python-dateutil-2.9.0.post0 requests-2.32.5 six-1.17.0 tenacity-9.1.2 urllib3-2.5.0 xxhash-3.6.0 yamanifest-0.3.14

This was installing the latest version which did work, need to module load python/3.11.6

[cbull.setonix-05: cbull]$ git clone https://github.com/payu-org/payu
Cloning into 'payu'...
remote: Enumerating objects: 6972, done.
remote: Counting objects: 100% (1824/1824), done.
remote: Compressing objects: 100% (542/542), done.
remote: Total 6972 (delta 1598), reused 1287 (delta 1282), pack-reused 5148 (from 3)
Receiving objects: 100% (6972/6972), 3.31 MiB | 8.52 MiB/s, done.
Resolving deltas: 100% (4942/4942), done.
Updating files: 100% (133/133), done.
[703][cbull.setonix-05: cbull]$ cd payu
[704][cbull.setonix-05: payu]$ pip install . --user
Processing /scratch/pawsey0889/cbull/payu
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting f90nml>=0.16 (from payu==1.2.0)
  Obtaining dependency information for f90nml>=0.16 from https://files.pythonhosted.org/packages/21/f2/4454eefc15cc326b46530d230c58cc0bb91a1e9797f2842b2a1720cbb233/f90nml-1.5.0-py2.py3-none-any.whl.metadata
  Using cached f90nml-1.5.0-py2.py3-none-any.whl.metadata (6.4 kB)
Collecting yamanifest>=0.3.4 (from payu==1.2.0)
  Obtaining dependency information for yamanifest>=0.3.4 from https://files.pythonhosted.org/packages/2a/5a/91816ff3c7f589f027168218fe53bc79c5e6baa76086d6e90fefc0679681/yamanifest-0.3.14-py3-none-any.whl.metadata
  Using cached yamanifest-0.3.14-py3-none-any.whl.metadata (15 kB)
Collecting PyYAML (from payu==1.2.0)
  Obtaining dependency information for PyYAML from https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Using cached pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB)
Collecting requests (from payu==1.2.0)
  Obtaining dependency information for requests from https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl.metadata
  Using cached requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting python-dateutil (from payu==1.2.0)
  Obtaining dependency information for python-dateutil from https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting tenacity>=8.0.0 (from payu==1.2.0)
  Obtaining dependency information for tenacity>=8.0.0 from https://files.pythonhosted.org/packages/e5/30/643397144bfbfec6f6ef821f36f33e57d35946c44a2352d3c9f0ae847619/tenacity-9.1.2-py3-none-any.whl.metadata
  Using cached tenacity-9.1.2-py3-none-any.whl.metadata (1.2 kB)
Collecting cftime (from payu==1.2.0)
  Obtaining dependency information for cftime from https://files.pythonhosted.org/packages/d1/fd/a7266970312df65e68b5641b86e0540a739182f5e9c62eec6dbd29f18055/cftime-1.6.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
  Using cached cftime-1.6.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (8.7 kB)
Collecting GitPython>=3.1.40 (from payu==1.2.0)
  Obtaining dependency information for GitPython>=3.1.40 from https://files.pythonhosted.org/packages/01/61/d4b89fec821f72385526e1b9d9a3a0385dda4a72b206d28049e2c7cd39b8/gitpython-3.1.45-py3-none-any.whl.metadata
  Using cached gitpython-3.1.45-py3-none-any.whl.metadata (13 kB)
Collecting ruamel.yaml>=0.18.5 (from payu==1.2.0)
  Obtaining dependency information for ruamel.yaml>=0.18.5 from https://files.pythonhosted.org/packages/0f/73/bb1bc2529f852e7bf64a2dec885e89ff9f5cc7bbf6c9340eed30ff2c69c5/ruamel.yaml-0.18.16-py3-none-any.whl.metadata
  Using cached ruamel.yaml-0.18.16-py3-none-any.whl.metadata (25 kB)
Collecting packaging (from payu==1.2.0)
  Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl.metadata
  Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting netCDF4 (from payu==1.2.0)
  Obtaining dependency information for netCDF4 from https://files.pythonhosted.org/packages/97/1a/78b19893197ed7525edfa7f124a461626541e82aec694a468ba97755c24e/netcdf4-1.7.3-cp311-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata
  Using cached netcdf4-1.7.3-cp311-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.9 kB)
Collecting jsonschema (from payu==1.2.0)
  Obtaining dependency information for jsonschema from https://files.pythonhosted.org/packages/bf/9c/8c95d856233c1f82500c2450b8c68576b4cf1c871db3afac5c34ff84e6fd/jsonschema-4.25.1-py3-none-any.whl.metadata
  Using cached jsonschema-4.25.1-py3-none-any.whl.metadata (7.6 kB)
Collecting gitdb<5,>=4.0.1 (from GitPython>=3.1.40->payu==1.2.0)
  Obtaining dependency information for gitdb<5,>=4.0.1 from https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl.metadata
  Using cached gitdb-4.0.12-py3-none-any.whl.metadata (1.2 kB)
Collecting ruamel.yaml.clib>=0.2.7 (from ruamel.yaml>=0.18.5->payu==1.2.0)
  Obtaining dependency information for ruamel.yaml.clib>=0.2.7 from https://files.pythonhosted.org/packages/61/62/150c841f24cda9e30f588ef396ed83f64cfdc13b92d2f925bb96df337ba9/ruamel_yaml_clib-0.2.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Using cached ruamel_yaml_clib-0.2.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (3.5 kB)
Collecting six (from yamanifest>=0.3.4->payu==1.2.0)
  Obtaining dependency information for six from https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl.metadata
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting xxhash (from yamanifest>=0.3.4->payu==1.2.0)
  Obtaining dependency information for xxhash from https://files.pythonhosted.org/packages/11/4f/426f91b96701ec2f37bb2b8cec664eff4f658a11f3fa9d94f0a887ea6d2b/xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Using cached xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (13 kB)
Collecting numpy>=1.21.2 (from cftime->payu==1.2.0)
  Obtaining dependency information for numpy>=1.21.2 from https://files.pythonhosted.org/packages/b6/23/2a1b231b8ff672b4c450dac27164a8b2ca7d9b7144f9c02d2396518352eb/numpy-2.3.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata
  Using cached numpy-2.3.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)
Collecting attrs>=22.2.0 (from jsonschema->payu==1.2.0)
  Obtaining dependency information for attrs>=22.2.0 from https://files.pythonhosted.org/packages/3a/2a/7cc015f5b9f5db42b7d48157e23356022889fc354a2813c15934b7cb5c0e/attrs-25.4.0-py3-none-any.whl.metadata
  Using cached attrs-25.4.0-py3-none-any.whl.metadata (10 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema->payu==1.2.0)
  Obtaining dependency information for jsonschema-specifications>=2023.03.6 from https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl.metadata
  Using cached jsonschema_specifications-2025.9.1-py3-none-any.whl.metadata (2.9 kB)
Collecting referencing>=0.28.4 (from jsonschema->payu==1.2.0)
  Obtaining dependency information for referencing>=0.28.4 from https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl.metadata
  Using cached referencing-0.37.0-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema->payu==1.2.0)
  Obtaining dependency information for rpds-py>=0.7.1 from https://files.pythonhosted.org/packages/04/4c/f9b8a05faca3d9e0a6397c90d13acb9307c9792b2bff621430c58b1d6e76/rpds_py-0.29.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached rpds_py-0.29.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting certifi (from netCDF4->payu==1.2.0)
  Obtaining dependency information for certifi from https://files.pythonhosted.org/packages/70/7d/9bc192684cea499815ff478dfcdc13835ddf401365057044fb721ec6bddb/certifi-2025.11.12-py3-none-any.whl.metadata
  Using cached certifi-2025.11.12-py3-none-any.whl.metadata (2.5 kB)
Collecting charset_normalizer<4,>=2 (from requests->payu==1.2.0)
  Obtaining dependency information for charset_normalizer<4,>=2 from https://files.pythonhosted.org/packages/c0/10/d20b513afe03acc89ec33948320a5544d31f21b05368436d580dec4e234d/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata
  Using cached charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (37 kB)
Collecting idna<4,>=2.5 (from requests->payu==1.2.0)
  Obtaining dependency information for idna<4,>=2.5 from https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl.metadata
  Using cached idna-3.11-py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3<3,>=1.21.1 (from requests->payu==1.2.0)
  Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/a7/c2/fe1e52489ae3122415c51f387e221dd0773709bad6c6cdaa599e8a2c5185/urllib3-2.5.0-py3-none-any.whl.metadata
  Using cached urllib3-2.5.0-py3-none-any.whl.metadata (6.5 kB)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython>=3.1.40->payu==1.2.0)
  Obtaining dependency information for smmap<6,>=3.0.1 from https://files.pythonhosted.org/packages/04/be/d09147ad1ec7934636ad912901c5fd7667e1c858e19d355237db0d0cd5e4/smmap-5.0.2-py3-none-any.whl.metadata
  Using cached smmap-5.0.2-py3-none-any.whl.metadata (4.3 kB)
Collecting typing-extensions>=4.4.0 (from referencing>=0.28.4->jsonschema->payu==1.2.0)
  Obtaining dependency information for typing-extensions>=4.4.0 from https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl.metadata
  Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Using cached f90nml-1.5.0-py2.py3-none-any.whl (51 kB)
Using cached gitpython-3.1.45-py3-none-any.whl (208 kB)
Using cached ruamel.yaml-0.18.16-py3-none-any.whl (119 kB)
Using cached tenacity-9.1.2-py3-none-any.whl (28 kB)
Using cached yamanifest-0.3.14-py3-none-any.whl (20 kB)
Using cached cftime-1.6.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.6 MB)
Using cached jsonschema-4.25.1-py3-none-any.whl (90 kB)
Using cached netcdf4-1.7.3-cp311-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (9.5 MB)
Using cached packaging-25.0-py3-none-any.whl (66 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (807 kB)
Using cached requests-2.32.5-py3-none-any.whl (64 kB)
Using cached attrs-25.4.0-py3-none-any.whl (67 kB)
Using cached certifi-2025.11.12-py3-none-any.whl (159 kB)
Using cached charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (153 kB)
Using cached gitdb-4.0.12-py3-none-any.whl (62 kB)
Using cached idna-3.11-py3-none-any.whl (71 kB)
Using cached jsonschema_specifications-2025.9.1-py3-none-any.whl (18 kB)
Using cached numpy-2.3.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.6 MB)
Using cached referencing-0.37.0-py3-none-any.whl (26 kB)
Using cached rpds_py-0.29.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (395 kB)
Using cached ruamel_yaml_clib-0.2.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (788 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Using cached urllib3-2.5.0-py3-none-any.whl (129 kB)
Using cached xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (193 kB)
Using cached smmap-5.0.2-py3-none-any.whl (24 kB)
Using cached typing_extensions-4.15.0-py3-none-any.whl (44 kB)
Building wheels for collected packages: payu
  Building wheel for payu (pyproject.toml) ... done
  Created wheel for payu: filename=payu-1.2.0-py3-none-any.whl size=138743 sha256=01a7bd9f55d46b6c64d70fbf69d519e84207eb07b67739444afa84ef1bd30f85
  Stored in directory: /tmp/pip-ephem-wheel-cache-6knf0lww/wheels/dc/2b/0d/0a9e5a2ea73680c251c2c3fe52dd6bf042940ba47f4382be98
Successfully built payu
Installing collected packages: f90nml, xxhash, urllib3, typing-extensions, tenacity, smmap, six, ruamel.yaml.clib, rpds-py, PyYAML, packaging, numpy, idna, charset_normalizer, certifi, attrs, yamanifest, ruamel.yaml, requests, referencing, python-dateutil, gitdb, cftime, netCDF4, jsonschema-specifications, GitPython, jsonschema, payu
Successfully installed GitPython-3.1.45 PyYAML-6.0.3 attrs-25.4.0 certifi-2025.11.12 cftime-1.6.5 charset_normalizer-3.4.4 f90nml-1.5.0 gitdb-4.0.12 idna-3.11 jsonschema-4.25.1 jsonschema-specifications-2025.9.1 netCDF4-1.7.3 numpy-2.3.5 packaging-25.0 payu-1.2.0 python-dateutil-2.9.0.post0 referencing-0.37.0 requests-2.32.5 rpds-py-0.29.0 ruamel.yaml-0.18.16 ruamel.yaml.clib-0.2.15 six-1.17.0 smmap-5.0.2 tenacity-9.1.2 typing-extensions-4.15.0 urllib3-2.5.0 xxhash-3.6.0 yamanifest-0.3.14
[705][cbull.setonix-05: payu]$

Rsync tips

From @john_reilly

Rsync jobs timing out: Typically use the data-mover nodes for longer transfers (or any transfers really). E.g., from gadi:

* Rsync <gadi path> <username>@**data-mover**.pawsey.org.au:<pawsey path>

With discussion with @joshuatorrance we figured out this works. Using screen on setonix

rsync -avz --progress cyb561@gadi-dm.nci.org.au:/g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0 /scratch/pawsey0889/cbull/

cbull · 19 November 2025 00:55

Running OM3 `dev-MC_100km_jra_iaf` (`ceaabe3`)!

Modified config.yaml for
ceaabe3077382e5d6201791cdd21e1a4b039e964

Using payu to setup run dir

module load python/3.11.6
[cbull.setonix-05: access-om3-configs]$ payu setup
laboratory path:  /scratch/pawsey0889/cbull/cbull/access-om3
binary path:  /scratch/pawsey0889/cbull/cbull/access-om3/bin
input path:  /scratch/pawsey0889/cbull/cbull/access-om3/input
work path:  /scratch/pawsey0889/cbull/cbull/access-om3/work
archive path:  /scratch/pawsey0889/cbull/cbull/access-om3/archive
Metadata and UUID generation is disabled. Experiment name used for archival: access-om3-configs
payu: Found modules in /opt/cray/pe/lmod/lmod
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up access-om3
Checking exe, input and restart manifests
Writing manifests/input.yaml
Writing manifests/restart.yaml
Writing manifests/exe.yaml

Note the config.yaml needed to be modified

# PBS configuration

# If submitting to a different project to your default, uncomment line below
# and change project code as appropriate; also set shortpath below
# project: x77

# Force payu to always find, and save, files in this scratch project directory
# (you may need to add the corresponding PBS -l storage flag in sync_data.sh)
# shortpath: /scratch/v45
shortpath: /scratch/pawsey0889/cbull

queue: normal
ncpus: 240
jobfs: 10GB
mem: 960GB

walltime: 02:00:00
jobname: 1deg_jra55do_iaf

model: access-om3

exe: /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3_take2/.spack-env/view/bin/access-om3-MOM6-CICE6
input:

    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-ESMFmesh.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-nomask-ESMFmesh.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-datm-ESMFmesh.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-drof-ESMFmesh.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/share/grids/global.1deg/2020.10.22/topog.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/prerelease/configurations/inputs/access-om3/mom/grids/vertical/share/2025.03.12/ocean_vgrid.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/mom/surface_salt_restoring/global.1deg/2020.05.30/salt_sfc_restore.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/cice/grids/global.1deg/2024.05.14/kmt.nc
    - /scratch/pawsey0889/cbull/om3forcing/vk83/configurations/inputs/access-om3/cice/initial_conditions/global.1deg/2023.07.28/iced.1900-01-01-10800.nc
    #- /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0
    - /scratch/pawsey0889/cbull/om3forcing/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0

collate: false
runlog: false
metadata:
    enable: false

#userscripts:
#    setup: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/setup.sh
#    archive: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/archive.sh
#
#modules:
#    use:
#        - /g/data/vk83/modules
#    load:
#        - access-om3/2025.01.2
#        - nco/5.0.5

payu_minimum_version: 1.1.6

Now running:

[cbull.setonix-05: cbull]$ salloc -p work -n 256 -N 2  --exclusive -A pawsey0889
salloc: Pending job allocation 35030705
salloc: job 35030705 queued and waiting for resources
salloc: job 35030705 has been allocated resources
salloc: Granted job allocation 35030705
salloc: Waiting for resource configuration
salloc: Nodes nid[001515,001526] are ready for job
[680][cbull.nid001515: cbull]$ /scratch/pawsey0889/cbull/access-om3-configs/work
bash: /scratch/pawsey0889/cbull/access-om3-configs/work: Is a directory
[681][cbull.nid001515: cbull]$ cd /scratch/pawsey0889/cbull/access-om3-configs/work
[682][cbull.nid001515: work]$ ls
access-om3-MOM6-CICE6  datm_in           diag_table  drof.streams.xml  fd.yaml  INPUT      log        MOM_input     nuopc.runconfig  RESTART
config.yaml            datm.streams.xml  drof_in     drv_in            ice_in   input.nml  manifests  MOM_override  nuopc.runseq     timing
[683][cbull.nid001515: work]$ ls -lah access-om3-MOM6-CICE6
lrwxrwxrwx 1 cbull pawsey0889 116 Nov 19 08:41 access-om3-MOM6-CICE6 -> /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3_take2/.spack-env/view/bin/access-om3-MOM6-CICE6
[684][cbull.nid001515: work]$ srun -N 2 -n 256 /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3_take2/.spack-env/view/bin/access-om3-MOM6-CICE6
 (t_initf) Read in prof_inparm namelist from: drv_in
 (t_initf) Using profile_disable=          F
 (t_initf)       profile_timer=                      4
 (t_initf)       profile_depth_limit=                4
 (t_initf)       profile_detail_limit=               2
 (t_initf)       profile_barrier=          F
 (t_initf)       profile_outpe_num=                  1
 (t_initf)       profile_outpe_stride=               0
 (t_initf)       profile_single_file=      F
 (t_initf)       profile_global_stats=     T
 (t_initf)       profile_ovhd_measurement= F
 (t_initf)       profile_add_detail=       F
 (t_initf)       profile_papi_enable=      F
NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to  5000000.
NOTE from PE     0: field_manager_init:: You are using the legacy version of the field_table

 Dumping coupler_mod/types tree
types/
      air_sea_gas_flux_generic/
            implementation/
                  ocmip2/
                        num_parameters = 2
                  duce/
                        num_parameters = 1
                  johnson/
                        num_parameters = 2
            num_flags = 0
            use_atm_pressure = T
            use_10m_wind_speed = T
            pass_through_ice = F
            atm/
                  name[1] = 'pcair'
                  name[2] = 'u10'
                  name[3] = 'psurf'
                  long_name[1] = 'Atmospheric concentration'
                  long_name[2] = 'Wind speed at 10 m'
                  long_name[3] = 'Surface atmospheric pressure'
                  units[1] = 'mol/mol'
                  units[2] = 'm/s'
                  units[3] = 'Pa'
            ice/
                  name[1] = 'alpha'
                  name[2] = 'csurf'
                  name[3] = 'sc_no'
                  long_name[1] = 'Solubility w.r.t. atmosphere'
                  long_name[2] = 'Ocean concentration'
                  long_name[3] = 'Schmidt number'
                  units[1] = 'mol/m^3/atm'
                  units[2] = 'mol/m^3'
                  units[3] = 'dimensionless'
            flux/
                  name[1] = 'flux'
                  name[2] = 'deltap'
                  name[3] = 'kw'
                  name[4] = 'flux0'
                  long_name[1] = 'Surface flux'
                  long_name[2] = 'Ocean-air delta pressure'
                  long_name[3] = 'Piston velocity'
                  long_name[4] = 'Surface flux no atm'
                  units[1] = 'mol/m^2/s'
                  units[2] = 'uatm'
                  units[3] = 'm/s'
                  units[4] = 'mol/m^2/s'
      air_sea_gas_flux/
            implementation/
                  ocmip2/
                        num_parameters = 2
                  ocmip2_data/
                        num_parameters = 2
                  linear/
                        num_parameters = 3
            num_flags = 0
            use_atm_pressure = T
            use_10m_wind_speed = T
            pass_through_ice = F
            atm/
                  name[1] = 'pcair'
                  name[2] = 'u10'
                  name[3] = 'psurf'
                  long_name[1] = 'Atmospheric concentration'
                  long_name[2] = 'Wind speed at 10 m'
                  long_name[3] = 'Surface atmospheric pressure'
                  units[1] = 'mol/mol'
                  units[2] = 'm/s'
                  units[3] = 'Pa'
            ice/
                  name[1] = 'alpha'
                  name[2] = 'csurf'
                  long_name[1] = 'Solubility from atmosphere times Schmidt number term'
                  long_name[2] = 'Ocean concentration times Schmidt number term'
                  units[1] = 'mol/m^3/atm'
                  units[2] = 'mol/m^3'
            flux/
                  name = 'flux'
                  long_name = 'Surface flux'
                  units = 'mol/m^2/s'
      air_sea_deposition/
            implementation/
                  dry/
                        num_parameters = 1
                  wet/
                        num_parameters = 1
            num_flags = 0
            use_atm_pressure = F
            use_10m_wind_speed = F
            pass_through_ice = T
            atm/
                  name = 'deposition'
                  long_name = 'Atmospheric deposition'
                  units = 'kg/m^2/s'
            ice/
                  name = NULL
                  long_name = NULL
                  units = NULL
            flux/
                  name = 'flux'
                  long_name = 'Surface deposition'
                  units = 'mol/m^2/s'
      land_sea_runoff/
            implementation/
                  river/
                        num_parameters = 1
            num_flags = 0
            use_atm_pressure = F
            use_10m_wind_speed = F
            pass_through_ice = T
            atm/
                  name = 'runoff'
                  long_name = 'Concentration in land runoff'
                  units = 'mol/m^3'
            ice/
                  name = NULL
                  long_name = NULL
                  units = NULL
            flux/
                  name = 'flux'
                  long_name = 'Concentration in land runoff'
                  units = 'mol/m^3'
NOTE from PE     0: open_param_file: MOM_input has been opened successfully.
NOTE from PE     0: open_param_file: MOM_override has been opened successfully.
NOTE from PE     0: close_param_file: MOM_input has been closed successfully.
NOTE from PE     0: close_param_file: MOM_override has been closed successfully.
 ==>Note from atmos_ocean_fluxes_mod(atmos_ocean_fluxes_init): No gas fluxes
NOTE from PE     0: callTree: ---> ocean_model_init(), ocean_model_MOM.F90
NOTE from PE     0: open_param_file: MOM_input has been opened successfully.
NOTE from PE     0: open_param_file: MOM_override has been opened successfully.
NOTE from PE     0: Found the optimum layout for auto-masking. Terminating iteration...
NOTE from PE     0: MOM_domains_init: reading maskmap information from MOM_auto_mask_table
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   30  30  30  30  30  30  30  30  30  30  30  30
  Y-AXIS =   17  17  17  17  17  17  16  16  16  16  16  16  17  17  17  17  17  17
 MOM_inc domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   15  15  15  15  15  15  15  15  15  15  15  15
  Y-AXIS =    9   9   9   8   8   8   8   8   8   8   8   8   8   8   8   9   9   9
 parse_mask_table: Number of domain regions masked in MOM_in =           22
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
 MOM_inc domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   11  11  11  11  11  10  10  10  10  10  10  10  11  11  11  11  11
  Y-AXIS =   11  11  11  11  11  10  10  10  10  11  11  11  11  11
NOTE from PE     0: diag_manager_mod::diag_manager_init: prepend_date only supported when diag_manager_init is called with time_init present.
NOTE from PE     0: diag_manager_mod::diag_manager_init: You are using the legacy version of the diag table
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   11  11  11  11  11  10  10  10  10  10  10  10  11  11  11  11  11
  Y-AXIS =   11  11  11  11  11  10  10  10  10  11  11  11  11  11
 MOM_MOSAIC domain decomposition
whalo =    9, ehalo =    9, shalo =    9, nhalo =    9
  X-AXIS =   44  44  42  42  42  42  42  42  40  42  42  42  42  42  42  44  44
  Y-AXIS =   44  44  44  42  42  42  42  42  42  42  42  44  44  44
 MOM_MOSAIC domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   30  30  30  30  30  30  30  30  30  30  30  30
  Y-AXIS =   17  17  17  17  17  17  16  16  16  16  16  16  17  17  17  17  17  17
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   15  15  15  15  15  15  15  15  15  15  15  15
  Y-AXIS =    9   9   9   8   8   8   8   8   8   8   8   8   8   8   8   9   9   9
 MOM_MOSAIC domain decomposition
whalo =    9, ehalo =    9, shalo =    9, nhalo =    9
  X-AXIS =   60  60  60  60  60  60  60  60  60  60  60  60
  Y-AXIS =   34  34  34  34  34  34  32  32  32  32  32  32  34  34  34  34  34  34
 MOM_MOSAIC domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   30  30  30  30  30  30  30  30  30  30  30  30
  Y-AXIS =   17  17  17  17  17  17  16  16  16  16  16  16  17  17  17  17  17  17
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   30  30  30  30  30  30  30  30  30  30  30  30
  Y-AXIS =   17  17  17  17  17  17  16  16  16  16  16  16  17  17  17  17  17  17
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   15  15  15  15  15  15  15  15  15  15  15  15
  Y-AXIS =    9   9   9   8   8   8   8   8   8   8   8   8   8   8   8   9   9   9

WARNING from PE     0: Open_file is appending .nc to the filename ./ocean_geometry


WARNING from PE     0: Open_file is appending .nc to the filename ./ocean_geometry

 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   11  11  11  11  11  10  10  10  10  10  10  10  11  11  11  11  11
  Y-AXIS =   11  11  11  11  11  10  10  10  10  11  11  11  11  11

WARNING from PE     0: MEKE_init: Initializing MEKE with a local equilibrium balance.

NOTE from PE     0: MOM_restart: Unable to find MEKE_Le queried by pointer, probably because of the suspect comparison of pointers by ASSOCIATED.
NOTE from PE     0: MOM_restart: Unknown restart variable MEKE_Le queried for initialization.

WARNING from PE     0: MEKE_init: Initializing MEKE with a local equilibrium balance.

NOTE from PE     0: MOM_vert_friction: updateCFLtruncationValue set CFL limit to    0.000E+00
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   11  11  11  11  11  10  10  10  10  10  10  10  11  11  11  11  11
  Y-AXIS =   11  11  11  11  11  10  10  10  10  11  11  11  11  11
 MOM_in domain decomposition
whalo =    0, ehalo =    0, shalo =    0, nhalo =    0
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
NOTE from PE     0: data_override_init: You are using the legacy version of the data table.
NOTE from PE     0: data_override_mod: File data_table does not exist.
NOTE from PE     0: data_table is empty, not doing any data_overrides
NOTE from PE     0: data_override_init: You are using the legacy version of the data table.
NOTE from PE     0: data_override_mod: File data_table does not exist.
NOTE from PE     0: data_table is empty, not doing any data_overrides
 MOM_in domain decomposition
whalo =    4, ehalo =    4, shalo =    4, nhalo =    4
  X-AXIS =   22  22  21  21  21  21  21  21  20  21  21  21  21  21  21  22  22
  Y-AXIS =   22  22  22  21  21  21  21  21  21  21  21  22  22  22
 MOM_in domain decomposition
whalo =    2, ehalo =    2, shalo =    2, nhalo =    2
  X-AXIS =   11  11  11  11  11  10  10  10  10  10  10  10  11  11  11  11  11
  Y-AXIS =   11  11  11  11  11  10  10  10  10  11  11  11  11  11
NOTE from PE     0: MOM_vert_friction: updateCFLtruncationValue set CFL limit to    2.188E-01
NOTE from PE     0: MOM_vert_friction: updateCFLtruncationValue set CFL limit to    3.750E-01
NOTE from PE     0: MOM_vert_friction: updateCFLtruncationValue set CFL limit to    4.688E-01
NOTE from PE     0: MOM_vert_friction: updateCFLtruncationValue set CFL limit to    5.000E-01
 ERROR: 1 nans found in Fioi_meltw
 ABORTING JOB, see PET file for details
MPICH ERROR [Rank 0] [job id 35030705.0] [Wed Nov 19 08:45:40 2025] [nid001515] - Abort(1) (rank 0 in comm 496): application called MPI_Abort(comm=0x84000002, 1) - process 0

srun: error: nid001526: tasks 128-255: Exited with exit code 1
srun: Terminating StepId=35030705.0
slurmstepd: error: *** STEP 35030705.0 ON nid001515 CANCELLED AT 2025-11-19T08:45:41 ***
srun: error: nid001515: tasks 0-127: Exited with exit code 1

cbull · 28 November 2025 00:19

@helen / @harshula as I’ve now started keeping notes / issues / errors on GitHub. I think I’m inclined to delete my posts above (might make a backup offline). That ok?

helen · 28 November 2025 00:32

That works for me. Please keep @harshula’s first post though.

Topic		Replies	Views
Running ACCESS-OM2 on Leonardo supercomputer Technical help , access-om2	42	726	17 November 2024
Creating a regional ACCESS-OM3 TWG 21centuryweather	20	306	20 February 2025
ACCESS-ROM3 setup instructions Regional MOM6 regional , tutorial , om3	80	955	12 June 2025
PAYU issues on Leonardo Technical help , payu	32	387	23 April 2025
PAYU issues on Setonix Technical payu	17	461	26 September 2023

How to build ACCESS-OM3 on Setonix