Setting build options for OM3

[Paul.Gregory]

Hello.

NCI have gotten back to me with the tools they use for MPI debugging. See here

How to Debug Parallel Programs - NCI Help - Opus - NCI Confluence

Some I want to build a MOM6 executable with debugging so I can find out why it fails for this particular case (but runs successfully for many other configurations!).

Now I have to remember the procedures to compiling your own MOM6 versions.

I have installed Spack before, following these instructions.

Modify and build an ACCESS model's source code - ACCESS-Hive Docs

I’ll follow @Helen’s instructions here to compile OM3

ACCESS-ROM3 setup instructions

Introduction These are the steps needed to convert an ACCESS-OM3 model configuration to a regional model configuration run on NUPOC. If you would like to view an example setup there is one here. To download this you need to run git clone --branch dev-regional_jra55do_ryf https://github.com/ACCESS-NRI/access-om3-configs/ if you want to see some of steps described in the instructions below then run cd access-om3-configs git log -16 This will provide a log of the changes performed to conver…

Just checking - those instructions state:

I suggest skipping this step for now and waiting a couple of weeks before trying as there are some changes coming through that will make it easier.

The modifications are when you reach this step:

git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5.git
spack env create mom5_dev ACCESS-ESM1.5/spack.yaml

you need to change to this:

git clone https://github.com/ACCESS-NRI/ACCESS-OM3.git
spack env create mom6_dev ACCESS-OM3/spack.yaml

If you wait a couple of weeks, these next changes will not be necessary

Then, in spack.yaml you need to change

   access-om3-nuopc:
      require:
        - '@git.0.3.1'

to

   access-om3-nuopc:
      require:
        - '@git.0.3.1'
        - configurations=MOM6
        - +mom_symmetric 
        

Are those still the current instructions for compiling OM3?

Last December I did manage to compile an older executable, see here

https://21centuryweather.discourse.group/t/develop-a-regional-coupled-modelling-strategy/168/41

Thanks

[Paul.Gregory]

Ok thanks.

I’ll run through everything from scratch and document every step.

Quick question - where do you specify DEBUG compilation for OM3?

I’m looking through CMakeLists.txt and the CMake .json files, and grepped for ‘debug’ in my OM3 source directory, but I can’t find the switch to enable DEBUG compilation.

Thanks

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

@Paul.Gregory

Ok thanks.

I’ll run through everything from scratch and document every step.

Quick question - where do you specify DEBUG compilation for OM3?

I’m looking through CMakeLists.txt and the CMake .json files, and grepped for ‘debug’ in my OM3 source directory, but I can’t find the switch to enable DEBUG compilation.

Thanks

As I understand it Debug is included in the CMake build and a default spack variant.

So in your spack environment file, add the build_type=Debug variant to the access-om3-nuopc package.

I’ve kicked off an build here to see what happens, this is mostly untested. (We haven’t been regularly doing this, so no promises for example that a debug build will even run ).

[Paul.Gregory]

Hi @anton

I’ve pulled the latest version of GitHub - ACCESS-NRI/ACCESS-OM3: ACCESS-OM3 repository for spack configuration information into my Spack directory, I’ve rebuilt my mom6_env environment and now I’m installing this version.

When I get onto compiling/installing a version using debug flags, do you want me to create a ‘development branch’ as outlined in

Modify and build an ACCESS model's source code - ACCESS-Hive Docs

?

Or is that overkill for a version that will most likely only ever sit on my local disk.

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

If the changes are ONLY to compiler flags, and no source code changes are needed, then there is no need for the spack develop process.

Make the changes to the compiler flags in the spack.yaml, then run concretise and install.

There are some notes here about setting flags:

Spack

and I updated the example here with setting fflags

Note between posts we updated how access-om3 is built. There is now a seperate spack package for each model component now.

Is this for regional work? If using the latest from main, you probably want to change to something like this:

spack:
  specs:
    - access-om3@git.2025.05.000
  packages:
    # Main Dependencies
    access3:
      require:
        - '@git.2025.03.0'
        - configurations=MOM6
    access-mom6:
      require:
        - '@git.2025.02.000'
        - '~asymmetric_mem'
    access3-share:
      require:
        - '@git.2025.03.0'
... #leave Other Dependencies etc unchanged

You can then add build_type and cflags and fflags to each of those packages.

[Paul.Gregory]

Thanks Anton. This is for regional work. I will follow your guidelines.

Thanks

[Paul.Gregory]

Ok in environments/mom6_dev/spack.yaml I changed

    access-mom6:
      require:
        - '@git.2025.02.000'
        - '+asymmetric_mem'

To

    access-mom6:
      require:
        - '@git.2025.02.000'
        - '+asymmetric_mem'
        - build_type=Debug
        - 'fflags="-g"''

To just recompile the mom6 executable with debug symbols.

$ spack install then executed with the following:

	[+] /usr (external glibc-2.28-gkllvhliut5hnwpt7ydwtbnemqcwxc5u)
==> openmpi@4.1.7 : has external module in ['openmpi/4.1.7']
[+] /apps/openmpi/4.1.7 (external openmpi-4.1.7-dsgaldxhkofee6argfqcozt6fd3zcq52)
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-dmmd5lphbqt7scvbujxcotxgmlxvtlvt
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/intel-oneapi-runtime-2025.0.4-otx2alpzeflc56jg5fimh4dygkmmurd2
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/zlib-ng-2.1.6-e6syrfr5fdorrhuf726b2z5nfwwmu5a7
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/zstd-1.5.6-wvekoquwtk3aq5dkfnsy7nmbeklieovx
...
etc

But that hasn’t changed the modified times of any files in

./release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-dmmd5lphbqt7scvbujxcotxgmlxvtlvt/lib/

or

./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/

Do I need to run the concrete step again so it actually re-compiles the source with debug symbols?

EDIT: Yes I do. From the NRI docs

" Concretizing the Spack environment is necessary anytime the environment’s spack.yaml gets changed, to force Spack to update its knowledge of the environment configuration."

So after making those edits, I ran

$ spack concretize -f 
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    access-om3@git.2025.05.000=2025.05.000 is unsatisfiable, errors are:
    internal_error("Reuse version weight used for build package")
    internal_error("must choose a single version to satisfy version constraints")
    internal_error("version weights must exist and be unique")

Clearly I’ve broken the package configuration. Time to do some more reading.

BTW any suggestions about spack tutorials? I’m reading through the slides available here:

Tutorial: Spack 101 — Spack Tutorial documentation

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

1. For regional, you want to use ~asymmetric_mem (i.e. NOT-asymmetric_mem, which is equivalent to the old +MOM_symmetric)

2. Yes you need to run spack concretise again

Possibly spack concretise --reuse-deps -f. If that doesn’t do anything, then spack concretise --fresh -f

Spack concretise should print output to the terminal. If there is no output nothing happened.

Next to each package (on the left) in the concretise printout there should be a + (use existing install) , - (build & install), or e (use external install). There should be a - next to access-mom6

$ spack concretize -f 
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    access-om3@git.2025.05.000=2025.05.000 is unsatisfiable, errors are:
    internal_error("Reuse version weight used for build package")
    internal_error("must choose a single version to satisfy version constraints")
    internal_error("version weights must exist and be unique")

This error means (approximately), spack has tried to match the package you are requesting against one that is already installed. spack found access-om3 package already installed but its different to what your spack.yaml is requesting. If you use the --reuse-deps or --fresh flag with concretise, it will tell spack to replace the existing installation of the access-om3 package with a new build of it.

I am not sure on spack docs - hoping @harshula has suggestions

[Paul.Gregory]

OK this didn’t work.

$ spack concretize --reuse-deps -f 
==> Error: concretization failed for the following reasons:

   1. Cannot select a single "version" for package "access3-share"
   2. Cannot select a single "version" for package "access3"
   3. cannot satisfy a requirement for package 'esmf'.
   4. cannot satisfy a requirement for package 'access3-share'.
   5. cannot satisfy a requirement for package 'access3'.
   6. access3: A configurations variant must be set, can be one or many of ('MOM6', 'CICE6', 'WW3', 'MOM6-WW3', 'MOM6-CICE6', 'CICE6-WW3', 'MOM6-CICE6-WW3')
   7. access3: A configurations variant must be set, can be one or many of ('MOM6', 'CICE6', 'WW3', 'MOM6-WW3', 'MOM6-CICE6', 'CICE6-WW3', 'MOM6-CICE6-WW3')
        required because conflict is triggered when configurations=none 
        required because conflict constraint  

But this did.

 spack concretize --fresh -f 
==> Warning: using "cmake@3.24.2" which is a deprecated version
==> Concretized access-om3@git.2025.05.000=2025.05.000
 -   xiebyhy  access-om3@git.2025.05.000=2025.05.000%oneapi@2025.0.4 build_system=bundle arch=linux-rocky8-x86_64_v4
 -   ohls2xb      ^access3@git.2025.03.0=2025.03.0%oneapi@2025.0.4~ipo build_system=cmake build_type=Release configurations=MOM6-CICE6,MOM6-CICE6-WW3 generator=make arch=linux-rocky8-x86_64_v4
[+]  q7a6p2x          ^access-cice@git.CICE6.6.0-1=6.6.0-1%oneapi@2025.0.4+access3~ipo~openmp build_system=cmake build_type=Release generator=make io_type=PIO arch=linux-rocky8-x86_64_v4
[+]  bab6hiz              ^parallelio@2.6.2%oneapi@2025.0.4+fortran~ipo~logging+mpi~ncint~pnetcdf+shared~timing build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
 -   nz62ock          ^access-mom6@git.2025.02.000=2025.02.000%oneapi@2025.0.4 fflags=-g +access3+asymmetric_mem~ipo~openmp build_system=cmake build_type=Debug generator=make arch=linux-rocky8-x86_64_v4
[+]  kcucsid              ^fms@git.2024.03=2024.01.02-git.26%oneapi@2025.0.4~deprecated_io~gfs_phys+internal_file_nml~ipo+large_file~openmp~pic~quad_precision~yaml build_system=cmake build_type=Release constants=GFDL generator=make precision=32,64 arch=linux-rocky8-x86_64_v4
...
etc

So onto spack install which successfully built and installed access-mom6-git, but failed to build access3. Snippets from the stdout.

==> Installing access-mom6-git.2025.02.000=2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr [24/27]
==> No binary for access-mom6-git.2025.02.000=2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr found: installing from source
==> No patches needed for access-mom6
==> access-mom6: Executing phase: 'cmake'
==> access-mom6: Executing phase: 'build'
==> access-mom6: Executing phase: 'install'
==> access-mom6: Successfully installed access-mom6-git.2025.02.000=2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr
  Stage: 8.35s.  Cmake: 19.74s.  Build: 2m 6.01s.  Install: 1.47s.  Post-install: 1.16s.  Total: 2m 40.88s
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/access-mom6-git.2025.02.000_2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/access-cice-git.CICE6.6.0-1_6.6.0-1-q7a6p2xadrqkpwqitsvuufoudeisgxi4
==> Installing access3-git.2025.03.0=2025.03.0-ohls2xbbwpj5gzygz6fx7lwveepcdxvc [26/27]
==> No binary for access3-git.2025.03.0=2025.03.0-ohls2xbbwpj5gzygz6fx7lwveepcdxvc found: installing from source
==> No patches needed for access3
==> access3: Executing phase: 'cmake'
==> access3: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'

Where can I find the build log?

Thanks for your help - I can pick this up again next week.

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

Hi Paul

Normally it prints the path the spack build log to the terminal. I think its typically 'spack-build.out` or something like that.

I realised my instructions weren’t very clear. I made a new example here. The pre-release infrastructure we have will make a (temporary) build from that environment file (spack.yaml) and leave a comment with the path to the new build, if you want to use it.

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

My tests all failed with linking errors.

Do debug builds rely on having their dependent libraries also using debug build types ?

[Paul.Gregory]

Hi Anton

I’m at the CoE hackathon this week so I’ll be slow in replying.

Generally, if you are building an executable for debugging, all libraries will need to be compiled with debug symbols.

I’m unsure how the MOM6 code interacts with the other binaries (esmf, nuopcy etc) but we may have to add ‘-g’ to all other libraries used within MOM6?

I had assumed that the if you didn’t have debug symbols compiled w/esmf or nuopcy libraries, the debugger will just pass through any modules imported without debug symbols and carry on as normal.

(As you can tell, it’s been a while since I’ve been into the weeks debugging complex Fortran source).

When building rMOM6, spack correctly builds and installs the access-mom6 repo but fails with with access3 repo. What does that code do? Is is the Fortran source which combines the MOM6 and CICE modules?

I might try to add ‘-g’ to the CICE build flags. I’ll also update my spack.yaml to reflect the changes your made to specify symmetric memory etc.

I will also check out some old Fortran source from an older project I build, play around with ‘-g’ to remember what I’ve forgotten about using ‘-g’ symbols in some libraries.

BTW I couldn’t find any .out or .log files that were created last Friday night.

$ find . -name '*.out'
./spack/var/spack/repos/tutorial/packages/hdf5/test/dump.out
./spack/var/spack/repos/builtin/packages/hdf/test/storm110.out
./spack/var/spack/repos/builtin/packages/sqlite/test/dump.out
./spack/var/spack/repos/builtin/packages/m4/test/hello.out
./spack/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out
./spack/var/spack/repos/builtin/packages/pocl/example1.out
./spack/var/spack/repos/builtin/packages/hdf5/test/dump.out
$ find . -name '*.log'
./environments/mom6_dev/.spack-env/._view/4i2bucb5onztwymj4nixfyixsvoh2aks/.spack/pkgconf/archived-files/spack-src/config.log
./environments/mom6_dev/.spack-env/._view/4i2bucb5onztwymj4nixfyixsvoh2aks/.spack/fortranxml/archived-files/spack-src/config.log
./environments/mom6_dev/.spack-env/._view/4i2bucb5onztwymj4nixfyixsvoh2aks/.spack/netcdf-fortran/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/netcdf-fortran-4.6.1-cgiqrwtnubuunpbnldlr6bzwnvr7axwh/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/libiconv-1.17-ke6wac5eiwo6iunhzyiuzua7rm7vomti/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/pkgconf-2.2.0-uceimu4ka7s5img5e2a3si4gobfot7f3/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/libsigsegv-2.14-tg4yy6k6if2clgyedtkxjrn352ielq3j/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/fortranxml-4.1.2-w3qlf2kquynkgzx56gw4qp5gugzsrdp2/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/netcdf-fortran-4.6.1-2rodmlet2itybhmrtv6c4mxwwuf365gk/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/netcdf-c-4.9.2-ziq6jbzf6iolb2sfsbacme66hcxfroar/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/libsigsegv-2.14-ou7jkjx57rak4vlkm6ifyoodv4jgxed7/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/fortranxml-4.1.2-bcbruwvclii2pzaistgkxzakmq4hqglj/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/libiconv-1.17-55ipnyeeqcpbfgaqfanu36viaqqni3sx/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/netcdf-fortran-4.6.1-3iyx7hnvowobxccwlf3isebu5ykypflz/.spack/archived-files/spack-src/config.log
./release/linux-rocky8-x86_64/intel-2021.10.0/pkgconf-2.2.0-vrupasu7smpgcbarzpdwap45fcvjbjoa/.spack/archived-files/spack-src/config.log

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

No worries

Sorry for the vague instructions - try looking in /scratch/{USER}/{PROJECT}/tmp/spack-stage/... for spack-build-* files.

access-MOM6 builds a library with the MOM6 code.

access3 builds the final executable and links to MOM6 and the mediator/data/shared code (from the access3-share repo)

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

It looks like we tend to already set -g as a build flag when the build type of Debug is specified. (e.g. here it is for MOM6)