Modifying ACCESS-OM2 spack package

Aidan · 22 February 2024 06:34

I want to try modifying the mom5 spack package to add a +repro variant.

I’m following the instructions for developers v6 and I’ve built access-om2 fine.

Now I’m trying to use the version of spack-packages I have modified.

I did this to add my modified package repo:

spack repo add ~/code/spack/spack-packages/

but when I try to install that variant it says it doesn’t exist:

spack install access-om2+repro ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^parallelio@2.5.2 ^openmpi@4.0.2 %intel@19.0.5.281
==> Error: trying to set variant "repro" in package "access-om2", but the package has no such variant [happened during concretization of access-om2+repro ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^openmpi@4.0.2%intel@19.0.5.281 ^parallelio@2.5.2]

The output of

spack config blame

output from spack config blame

---                                                                        packages:                                                                            /g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:2              all:                                                                              
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:3                providers:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:5                  mpi: [openmpi]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:20        awk: [gawk]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:21        blas: [openblas, amdblis]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:22        D: [ldc]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:23        daal: [intel-oneapi-daal]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:24        elf: [elfutils]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:25        fftw-api: [fftw, amdfftw]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:26        flame: [libflame, amdlibflame]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:27        fuse: [libfuse]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:28        gl: [glx, osmesa]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:29        glu: [mesa-glu, openglu]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:30        golang: [go, gcc]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:31        go-or-gccgo-bootstrap: [go-bootstrap, gcc]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:32        iconv: [libiconv]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:33        ipp: [intel-oneapi-ipp]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:34        java: [openjdk, jdk, ibm-java]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:35        jpeg: [libjpeg-turbo, libjpeg]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:36        lapack: [openblas, amdlibflame]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:37        libglx: [mesa+glx, mesa18+glx]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:38        libllvm: [llvm]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:39        libosmesa: [mesa+osmesa, mesa18+osmesa]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:40        lua-lang: [lua, lua-luajit-openresty, lua-luajit]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:41        luajit: [lua-luajit-openresty, lua-luajit]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:42        mariadb-client: [mariadb-c-client, mariadb]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:43        mkl: [intel-oneapi-mkl]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:44        mpe: [mpe2]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:46        mysql-client: [mysql, mariadb-c-client]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:47        opencl: [pocl]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:48        onedal: [intel-oneapi-dal]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:49        pbs: [openpbs, torque]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:50        pil: [py-pillow]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:51        pkgconfig: [pkgconf, pkg-config]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:52        rpc: [libtirpc]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:53        scalapack: [netlib-scalapack, amdscalapack]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:54        sycl: [hipsycl]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:55        szip: [libaec, libszip]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:56        tbb: [intel-tbb]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:57        unwind: [libunwind]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:58        uuid: [util-linux-uuid, libuuid]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:59        xxd: [xxd-standalone, vim]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:60        yacc: [bison, byacc]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:61        ziglang: [zig]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:18      compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:62      permissions:
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:63        read: world
/g/data/tm70/aph502/spack-build/spack/etc/spack/defaults/packages.yaml:64        write: user
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:6              perl:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:7                externals:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:8                - spec: perl@5.26.3~cpanm+shared+threads
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:9                  prefix: /usr
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:10               buildable: False
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:11             cmake:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:12               externals:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:13               - spec: cmake@3.24.2
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:14                 prefix: /apps/cmake/3.24.2
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:15               buildable: False
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:16             openmpi:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:17               externals:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:18               - spec: openmpi@4.0.2
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:19                 prefix: /apps/openmpi/4.0.2
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:20                 modules:
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:21                 - openmpi/4.0.2
/g/data/tm70/aph502/spack-build/spack/etc/spack/packages.yaml:22               buildable: False

Doesn’t show the package repo I added, but when I try and add it again it seems to think it knows about it. Listing the repos shows it:

$ spack repo list                                                                                                            
==> 3 package repositories.                                                                                                                                    
access.nri    /home/502/aph502/code/spack/spack-packages                                                                                                       
access.nri    /g/data/tm70/aph502/spack-build/spack-packages                                                                                                   
builtin       /g/data/tm70/aph502/spack-build/spack/var/spack/repos/builtin

I tried removing both the repos and then adding back my modified one:

$ spack repo list
==> 2 package repositories.
access.nri    /home/502/aph502/code/spack/spack-packages
builtin       /g/data/tm70/aph502/spack-build/spack/var/spack/repos/builtin

but it still won’t build. It doesn’t seem to want to find my updated SPD.

Any suggestions @harshula?

Aidan · 22 February 2024 06:44

Addendum to above, the output of spack info shows it doesn’t know about the variant

$ spack info access-om2
BundlePackage:   access-om2

Description:
    ACCESS-OM2 bundle contains the coupled CICE5 and MOM5 models.

Homepage: https://www.access-nri.org.au

Preferred version:  
    latest    

Safe versions:  
    latest    

Deprecated versions:  
    None

Variants:
    Name [Default]           When    Allowed values    Description
    =====================    ====    ==============    ======================================

    build_system [bundle]    --      bundle            Build systems supported by the package
    deterministic [off]      --      on, off           Deterministic build.

Build Dependencies:
    cice5  libaccessom2  mom5

Link Dependencies:
    cice5  libaccessom2  mom5

Run Dependencies:
    None

I have made a PR here so you can check my changes in case I’ve made a mistake

harshula · 22 February 2024 06:47

Why didn’t you modify the existing mom5 SPD in spack-packages?

Aidan · 22 February 2024 06:49

Good question. Dunno. Will try that tomorrow.

harshula · 22 February 2024 06:52

Please create a new topic if you want help to modify ACCESS-OM2.

Aidan · 22 February 2024 11:06

Uh duh. I didn’t have a +repro variant in the access-om2 package, so it couldn’t pass it through to mom.

Fixed that and it is finding the +repro variant now and has successfully built.

Motivation for this is to test reproducibility across different restart boundaries

github.com/ACCESS-NRI/access-om2-configs

Restart reproducibility tests are failing

opened 06:42AM - 22 Feb 24 UTC

aidanheerdegen

The tests for reproducibility across restarts are currently not being run in the… CI tests because they fail. https://github.com/ACCESS-NRI/access-om2-configs/blob/main/test/test_bit_reproducibility.py#L78 The original COSIMA ACCESS-OM2 tests from which these were adapted used reproducibility flags turned on for their builds https://github.com/COSIMA/access-om2/blob/master/test/exp_test_helper.py#L217-L218 though this was not used in production builds https://github.com/COSIMA/access-om2/blob/master/install.sh#L49 We need a `repro` variant for the spack package to test if we can replicate the COSIMA reproducibility results, and see what effect that option has on performance.

harshula · 22 February 2024 11:40

Try setting the REPRO variable to a non-empty string for the repro variant.

harshula · 23 February 2024 07:23

Did you try?:

spack install access-om2 ^mom5+repro ^netcdf-c@4.7.4 ^netcdf-fortran@4.5.2 ^parallelio@2.5.2 ^openmpi@4.0.2 %intel@19.0.5.281

Topic		Replies	Views
Shared access spack configuration spack python , spack	32	1133	10 May 2023
How to build ACCESS-OM2 on Gadi using Intel OneAPI compilers? spack spack , access-om2	3	223	15 November 2023
How to create a Docker container to build ACCESS-OM2 (x86) spack spack	2	233	18 September 2023
Random spack errors/tips/fixes spack spack	36	905	22 April 2024
Spack v0.22 vs v0.21: dependency reuse spack	0	55	16 May 2024

Modifying ACCESS-OM2 spack package

Related Topics