Setting build options for OM3

ANCOMS Regional MOM6
1

Hello.

NCI have gotten back to me with the tools they use for MPI debugging. See here

Some I want to build a MOM6 executable with debugging so I can find out why it fails for this particular case (but runs successfully for many other configurations!).

Now I have to remember the procedures to compiling your own MOM6 versions.

I have installed Spack before, following these instructions.

I’ll follow @Helen’s instructions here to compile OM3

Just checking - those instructions state:

I suggest skipping this step for now and waiting a couple of weeks before trying as there are some changes coming through that will make it easier.

The modifications are when you reach this step:

git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5.git
spack env create mom5_dev ACCESS-ESM1.5/spack.yaml

you need to change to this:

git clone https://github.com/ACCESS-NRI/ACCESS-OM3.git
spack env create mom6_dev ACCESS-OM3/spack.yaml

If you wait a couple of weeks, these next changes will not be necessary

Then, in spack.yaml you need to change

   access-om3-nuopc:
      require:
        - '@git.0.3.1'

to

   access-om3-nuopc:
      require:
        - '@git.0.3.1'
        - configurations=MOM6
        - +mom_symmetric

Are those still the current instructions for compiling OM3?

Last December I did manage to compile an older executable, see here

https://21centuryweather.discourse.group/t/develop-a-regional-coupled-modelling-strategy/168/41

Thanks

2

Ok thanks.

I’ll run through everything from scratch and document every step.

Quick question - where do you specify DEBUG compilation for OM3?

I’m looking through CMakeLists.txt and the CMake .json files, and grepped for ‘debug’ in my OM3 source directory, but I can’t find the switch to enable DEBUG compilation.

Thanks

3

As I understand it Debug is included in the CMake build and a default spack variant.

So in your spack environment file, add the build_type=Debug variant to the access-om3-nuopc package.

I’ve kicked off an build here to see what happens, this is mostly untested. (We haven’t been regularly doing this, so no promises for example that a debug build will even run ).

2 Likes
4

Hi @anton

I’ve pulled the latest version of GitHub - ACCESS-NRI/ACCESS-OM3: ACCESS-OM3 repository for spack configuration information into my Spack directory, I’ve rebuilt my mom6_env environment and now I’m installing this version.

When I get onto compiling/installing a version using debug flags, do you want me to create a ‘development branch’ as outlined in

?

Or is that overkill for a version that will most likely only ever sit on my local disk.

5

If the changes are ONLY to compiler flags, and no source code changes are needed, then there is no need for the spack develop process.

Make the changes to the compiler flags in the spack.yaml, then run concretise and install.

There are some notes here about setting flags:

and I updated the example here with setting fflags

Note between posts we updated how access-om3 is built. There is now a seperate spack package for each model component now.

Is this for regional work? If using the latest from main, you probably want to change to something like this:

spack:
  specs:
    - access-om3@git.2025.05.000
  packages:
    # Main Dependencies
    access3:
      require:
        - '@git.2025.03.0'
        - configurations=MOM6
    access-mom6:
      require:
        - '@git.2025.02.000'
        - '~asymmetric_mem'
    access3-share:
      require:
        - '@git.2025.03.0'
... #leave Other Dependencies etc unchanged

You can then add build_type and cflags and fflags to each of those packages.

1 Like
6

Thanks Anton. This is for regional work. I will follow your guidelines.

Thanks

7

Ok in environments/mom6_dev/spack.yaml I changed

    access-mom6:
      require:
        - '@git.2025.02.000'
        - '+asymmetric_mem'

To

    access-mom6:
      require:
        - '@git.2025.02.000'
        - '+asymmetric_mem'
        - build_type=Debug
        - 'fflags="-g"''

To just recompile the mom6 executable with debug symbols.

$ spack install then executed with the following:

	[+] /usr (external glibc-2.28-gkllvhliut5hnwpt7ydwtbnemqcwxc5u)
==> openmpi@4.1.7 : has external module in ['openmpi/4.1.7']
[+] /apps/openmpi/4.1.7 (external openmpi-4.1.7-dsgaldxhkofee6argfqcozt6fd3zcq52)
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-dmmd5lphbqt7scvbujxcotxgmlxvtlvt
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/intel-oneapi-runtime-2025.0.4-otx2alpzeflc56jg5fimh4dygkmmurd2
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/zlib-ng-2.1.6-e6syrfr5fdorrhuf726b2z5nfwwmu5a7
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/zstd-1.5.6-wvekoquwtk3aq5dkfnsy7nmbeklieovx
...
etc

But that hasn’t changed the modified times of any files in

./release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-dmmd5lphbqt7scvbujxcotxgmlxvtlvt/lib/

or

./release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/

Do I need to run the concrete step again so it actually re-compiles the source with debug symbols?

EDIT: Yes I do. From the NRI docs

" Concretizing the Spack environment is necessary anytime the environment’s spack.yaml gets changed, to force Spack to update its knowledge of the environment configuration."

So after making those edits, I ran

$ spack concretize -f 
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    access-om3@git.2025.05.000=2025.05.000 is unsatisfiable, errors are:
    internal_error("Reuse version weight used for build package")
    internal_error("must choose a single version to satisfy version constraints")
    internal_error("version weights must exist and be unique")

Clearly I’ve broken the package configuration. Time to do some more reading.

BTW any suggestions about spack tutorials? I’m reading through the slides available here:

8

  1. For regional, you want to use ~asymmetric_mem (i.e. NOT-asymmetric_mem, which is equivalent to the old +MOM_symmetric)

  2. Yes you need to run spack concretise again

Possibly spack concretise --reuse-deps -f. If that doesn’t do anything, then spack concretise --fresh -f

Spack concretise should print output to the terminal. If there is no output nothing happened.

Next to each package (on the left) in the concretise printout there should be a + (use existing install) , - (build & install), or e (use external install). There should be a - next to access-mom6

$ spack concretize -f 
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    access-om3@git.2025.05.000=2025.05.000 is unsatisfiable, errors are:
    internal_error("Reuse version weight used for build package")
    internal_error("must choose a single version to satisfy version constraints")
    internal_error("version weights must exist and be unique")

This error means (approximately), spack has tried to match the package you are requesting against one that is already installed. spack found access-om3 package already installed but its different to what your spack.yaml is requesting. If you use the --reuse-deps or --fresh flag with concretise, it will tell spack to replace the existing installation of the access-om3 package with a new build of it.

I am not sure on spack docs - hoping @harshula has suggestions

1 Like
9

OK this didn’t work.

$ spack concretize --reuse-deps -f 
==> Error: concretization failed for the following reasons:

   1. Cannot select a single "version" for package "access3-share"
   2. Cannot select a single "version" for package "access3"
   3. cannot satisfy a requirement for package 'esmf'.
   4. cannot satisfy a requirement for package 'access3-share'.
   5. cannot satisfy a requirement for package 'access3'.
   6. access3: A configurations variant must be set, can be one or many of ('MOM6', 'CICE6', 'WW3', 'MOM6-WW3', 'MOM6-CICE6', 'CICE6-WW3', 'MOM6-CICE6-WW3')
   7. access3: A configurations variant must be set, can be one or many of ('MOM6', 'CICE6', 'WW3', 'MOM6-WW3', 'MOM6-CICE6', 'CICE6-WW3', 'MOM6-CICE6-WW3')
        required because conflict is triggered when configurations=none 
        required because conflict constraint

But this did.

 spack concretize --fresh -f 
==> Warning: using "cmake@3.24.2" which is a deprecated version
==> Concretized access-om3@git.2025.05.000=2025.05.000
 -   xiebyhy  access-om3@git.2025.05.000=2025.05.000%oneapi@2025.0.4 build_system=bundle arch=linux-rocky8-x86_64_v4
 -   ohls2xb      ^access3@git.2025.03.0=2025.03.0%oneapi@2025.0.4~ipo build_system=cmake build_type=Release configurations=MOM6-CICE6,MOM6-CICE6-WW3 generator=make arch=linux-rocky8-x86_64_v4
[+]  q7a6p2x          ^access-cice@git.CICE6.6.0-1=6.6.0-1%oneapi@2025.0.4+access3~ipo~openmp build_system=cmake build_type=Release generator=make io_type=PIO arch=linux-rocky8-x86_64_v4
[+]  bab6hiz              ^parallelio@2.6.2%oneapi@2025.0.4+fortran~ipo~logging+mpi~ncint~pnetcdf+shared~timing build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
 -   nz62ock          ^access-mom6@git.2025.02.000=2025.02.000%oneapi@2025.0.4 fflags=-g +access3+asymmetric_mem~ipo~openmp build_system=cmake build_type=Debug generator=make arch=linux-rocky8-x86_64_v4
[+]  kcucsid              ^fms@git.2024.03=2024.01.02-git.26%oneapi@2025.0.4~deprecated_io~gfs_phys+internal_file_nml~ipo+large_file~openmp~pic~quad_precision~yaml build_system=cmake build_type=Release constants=GFDL generator=make precision=32,64 arch=linux-rocky8-x86_64_v4
...
etc

So onto spack install which successfully built and installed access-mom6-git, but failed to build access3. Snippets from the stdout.

==> Installing access-mom6-git.2025.02.000=2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr [24/27]
==> No binary for access-mom6-git.2025.02.000=2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr found: installing from source
==> No patches needed for access-mom6
==> access-mom6: Executing phase: 'cmake'
==> access-mom6: Executing phase: 'build'
==> access-mom6: Executing phase: 'install'
==> access-mom6: Successfully installed access-mom6-git.2025.02.000=2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr
  Stage: 8.35s.  Cmake: 19.74s.  Build: 2m 6.01s.  Install: 1.47s.  Post-install: 1.16s.  Total: 2m 40.88s
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/access-mom6-git.2025.02.000_2025.02.000-nz62ocks3w3h7kapo2dgvgxt527podyr
[+] /g/data/gb02/pag548/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.0.4/access-cice-git.CICE6.6.0-1_6.6.0-1-q7a6p2xadrqkpwqitsvuufoudeisgxi4
==> Installing access3-git.2025.03.0=2025.03.0-ohls2xbbwpj5gzygz6fx7lwveepcdxvc [26/27]
==> No binary for access3-git.2025.03.0=2025.03.0-ohls2xbbwpj5gzygz6fx7lwveepcdxvc found: installing from source
==> No patches needed for access3
==> access3: Executing phase: 'cmake'
==> access3: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'

Where can I find the build log?

Thanks for your help - I can pick this up again next week.