# Comparability of output from AM3 spack executable against that from locally-built modified AM3 **URL:** \ **Category:** AM3\ **Tags:** spack\ **Created:** [24 March 2026 09:18 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099 "2026-03-24T09:18:34Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [24 March 2026 09:18 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/1 "2026-03-24T09:18:34Z")
Hi team, @Jiachen_Lu has been adding urban land cover to AM3, and would like to test this against the unmodified model. (Link to Cumulus post for those with access to Cumulus [here](https://21centuryweather.discourse.group/t/am3-global-experiments/2113/72)). We are already doing an n512 run using the AM3 spack executable for another project. We had assumed we could use this as the baseline run to test Jiachen’s work against, however over in regional world today I noticed a big caveat on the [rAM3-configs pages](https://access-ram3-configs.access-hive.org.au/executables/) that says: > There are however slight differences between the current SPACK-built and rose/cylc built UM version 13.5 executable, as seen in minor differences in the model output. This is because the UM executable is sensitive to the order in which modules are loaded into the environment and there are subtle differences module load ordering. Does this mean that if we want to test modified AM3 that’s built locally with rose/cylc, we need to build a local rose/cylc vanilla AM3 and re-run our baseline?? The baseline run costs about 0.5 MSU at n512, so it would be preferable if we can avoid having to re-run with a new local build. ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [24 March 2026 23:20 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/2 "2026-03-24T23:20:01Z")
I was under the impression that we had achieved bitwise reproducibility between the suite and Spack builds, but this was before I joined the project- @ben @MartinDix do you remember the details on that? ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [24 March 2026 23:55 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/3 "2026-03-24T23:55:51Z")
I think we did manage to resolve compatibility between the Spack and in-suite builds, see the discussion in [this issue](https://github.com/ACCESS-NRI/access-am3-configs/issues/103). So I think the comparison between builds should be a fair one (for AM3). ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [24 March 2026 23:58 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/4 "2026-03-24T23:58:40Z")
Thanks @lachlanswhyborn, that’s the answer I was hoping to hear 😆 ---
**Author:** ![clairecarouge](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/clairecarouge/32/380_2.png) [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)\ **Post date:** [25 March 2026 23:31 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/5 "2026-03-25T23:31:39Z")
I would also encourage @Jiachen_Lu to build the model using the spack infrastructure. We want to deprecate the direct build in the suite since Spack provides a lot more provenance information. It might be good for Jiachen to learn the spack way then. ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [26 March 2026 02:43 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/6 "2026-03-26T02:43:16Z")
Hi @clairecarouge, this statement concerns me: > We want to deprecate the direct build in the suite since Spack provides a lot more provenance information. A significant amount of science use involves making changes to the source code and needing to recompile - as a very simple example, building 5-10 versions of the model for sensitivity testing of latent heating strengths. Although not the case for all science work (which is sometimes development), changes like this example are not something that are ever intended to be pushed back into the release branch. A system where the model can be built and run from submitting a single cylc workflow for each model run works well for this (and creates no extra overhead in separate user tasks). If the option to build directly in the suite with rose/cylc is deprecated in favour of pushing everyone to use spack, there will need to be a lot more user training and support provided from NRI around spack. This will be particularly important for new-starting PhD students. ---
**Author:** ![Jiachen\_Lu](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jiachen_lu/32/807_2.png) [@Jiachen\_Lu](https://forum.access-hive.org.au/u/Jiachen_Lu)\ **Post date:** [26 March 2026 02:46 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/7 "2026-03-26T02:46:23Z")
Thanks Claire, I’m doing some cleaning to our code after the discussion with Jhan, and will try to build a spack version of the executables. I agree with Bethan that a direct build in the suite is better to be preserved for R&D purpose. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [26 March 2026 02:50 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/8 "2026-03-26T02:50:15Z")
> [@clairecarouge](#): > > I would also encourage @Jiachen_Lu to build the model using the spack infrastructure. We want to deprecate the direct build in the suite since Spack provides a lot more provenance information. Also the suite build relies on the software dependencies being available as modules in `/g/data/access`. ACCESS-NRI will not be keeping these updated as the `spack` build compiles all required dependencies automatically. _When_ the `spack` build starts using newer dependencies the suite build will diverge, which is another reason to use the `spack` build infrastructure from the beginning. ACCESS-NRI has [automated build-infrastructure](https://docs.access-hive.org.au/models/build_a_model/create_a_prerelease/#backporting-bugfixes) that removes the need to compile the model and has a number of other benefits including making the work discoverable and available for others to test. Assistance is available if you need it. ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [26 March 2026 03:14 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/9 "2026-03-26T03:14:10Z")
Just chiming in here that a few rAM3 users have been running experiments using their own UM executables with modified source code. So before NRI removes the software dependencies in `/g/data/access` and fully commits to `spack` builds, the user community will need to develop competency in deploying UM spack builds. ---
**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\ **Post date:** [26 March 2026 04:10 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/10 "2026-03-26T04:10:57Z")
From my point of view maintaining functionality of people’s existing runs is extremely important, so I expect that nothing will be removed from `/g/data/access`. The climate and weather community actively uses a wide variety of model versions from different eras so there’s a large history to support - from UM7.3 CMIP runs to the latest UM14 now being developed on Github. This does have the downside that we’ve accumulated a lot of cruft over the last 15 or so years in `/g/data/access` - NRI using new technologies like spack to manage these dependencies in a new separate area where they’re not tied down by history is welcome 😄. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [26 March 2026 05:58 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/11 "2026-03-26T05:58:49Z")
> [@Paul.Gregory](#): > > So before NRI removes the software dependencies in `/g/data/access` and fully commits to `spack` builds, the user community will need to develop competency in deploying UM spack builds. Just to echo what @Scott said above **ACCESS-NRI HAS NO INTENTION OF DELETING OR MODIFYING** anything in `/g/data/access`. ACCESS-NRI is using other projects (principally `vk83`) for deploying software, in part for the reasons @Scott explained above. But we also are not intending to add to the existing software dependencies that would be required to upgrade in-suite builds. e.g. these are the current AM3 suite build dependencies that will _not_ be updated: ```bash module load gcom/7.8_ompi.4.1.4 module load eccodes/2.8.0 module load drhook/1.1_ompi.4.1.4 module load fcm/2019.09.0 ``` ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [26 March 2026 22:26 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/12 "2026-03-26T22:26:11Z")
Thanks for the confirmation Aidan. Moving forward, will all LFRic / Psyclone builds use spack? ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [26 March 2026 22:46 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/13 "2026-03-26T22:46:33Z")
> [@Paul.Gregory](#): > > Moving forward, will all LFRic / Psyclone builds use spack? ACCESS-NRI builds? Yes. `spack` is necessary for all infrastructure, as it allows for generic (model agnostic) workflows and [build provenance](https://reporting.access-nri-store.cloud.edu.au/release-provenance/releases), as well other desirable features. The UKMO is moving to `spack` too, so I’d expect it to become fairly ubiquitous in the future, so the “yes” above might be for all LFRic builds, but @Scott would know more about it than I. ---
**Author:** ![Scott](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/scott/32/37_2.png) [@Scott](https://forum.access-hive.org.au/u/Scott)\ **Post date:** [26 March 2026 22:58 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/14 "2026-03-26T22:58:00Z")
Momentum will be using spack to install dependencies, using the same package recipes and environment definitions across the partnership, but models themselves will continue to be built by the workflows as is done with current UM runs. ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [26 March 2026 22:59 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/15 "2026-03-26T22:59:41Z")
> [@Aidan](#): > > The UKMO is moving to `spack` too Ah, that’s great news. We’re ahead of the curve 😉 ---
**Author:** ![Jhan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/jhan/32/721_2.png) [@Jhan](https://forum.access-hive.org.au/u/Jhan)\ **Post date:** [27 March 2026 10:46 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/16 "2026-03-27T10:46:56Z")
Any progress on the local spack build? ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [30 March 2026 04:43 UTC](https://forum.access-hive.org.au/t/comparability-of-output-from-am3-spack-executable-against-that-from-locally-built-modified-am3/6099/17 "2026-03-30T04:43:16Z")
There is an [existing user guide for modifying and building ACCESS models](https://docs.access-hive.org.au/models/build_a_model/build_source_code/#changing-number-of-processes-for-linaro-ddt) directly. However models using newer UM versions were always a bit problematic because the source code was only available on the MOSRS subversion server, or the local mirror. We’re in the process of moving to entirely GitHub based builds, as well as transitioning to spack v 1.1 which will allow us to have a much simpler setup process to use the installed ACCESS-NRI spack instance and just `module load spack`. So right now is a uniquely bad time for this! I hope to be able to give you a much more satisfactory answer in a couple of weeks.