Comparability of output from AM3 spack executable against that from locally-built modified AM3

Atmosphere AM3
1

Hi team,

@Jiachen_Lu has been adding urban land cover to AM3, and would like to test this against the unmodified model.
(Link to Cumulus post for those with access to Cumulus here).

We are already doing an n512 run using the AM3 spack executable for another project. We had assumed we could use this as the baseline run to test Jiachen’s work against, however over in regional world today I noticed a big caveat on the rAM3-configs pages that says:

There are however slight differences between the current SPACK-built and rose/cylc built UM version 13.5 executable, as seen in minor differences in the model output. This is because the UM executable is sensitive to the order in which modules are loaded into the environment and there are subtle differences module load ordering.

Does this mean that if we want to test modified AM3 that’s built locally with rose/cylc, we need to build a local rose/cylc vanilla AM3 and re-run our baseline??

The baseline run costs about 0.5 MSU at n512, so it would be preferable if we can avoid having to re-run with a new local build.

2

I was under the impression that we had achieved bitwise reproducibility between the suite and Spack builds, but this was before I joined the project- @ben @MartinDix do you remember the details on that?

1 Like
3

I think we did manage to resolve compatibility between the Spack and in-suite builds, see the discussion in this issue. So I think the comparison between builds should be a fair one (for AM3).

1 Like
4

Thanks @lachlanswhyborn, that’s the answer I was hoping to hear :laughing:

5

I would also encourage @Jiachen_Lu to build the model using the spack infrastructure. We want to deprecate the direct build in the suite since Spack provides a lot more provenance information. It might be good for Jiachen to learn the spack way then.

1 Like
6

Hi @clairecarouge, this statement concerns me:

We want to deprecate the direct build in the suite since Spack provides a lot more provenance information.

A significant amount of science use involves making changes to the source code and needing to recompile - as a very simple example, building 5-10 versions of the model for sensitivity testing of latent heating strengths. Although not the case for all science work (which is sometimes development), changes like this example are not something that are ever intended to be pushed back into the release branch.

A system where the model can be built and run from submitting a single cylc workflow for each model run works well for this (and creates no extra overhead in separate user tasks).

If the option to build directly in the suite with rose/cylc is deprecated in favour of pushing everyone to use spack, there will need to be a lot more user training and support provided from NRI around spack. This will be particularly important for new-starting PhD students.

7

Thanks Claire, I’m doing some cleaning to our code after the discussion with Jhan, and will try to build a spack version of the executables. I agree with Bethan that a direct build in the suite is better to be preserved for R&D purpose.

8

Also the suite build relies on the software dependencies being available as modules in /g/data/access. ACCESS-NRI will not be keeping these updated as the spack build compiles all required dependencies automatically.

When the spack build starts using newer dependencies the suite build will diverge, which is another reason to use the spack build infrastructure from the beginning.

ACCESS-NRI has automated build-infrastructure that removes the need to compile the model and has a number of other benefits including making the work discoverable and available for others to test. Assistance is available if you need it.

9

Just chiming in here that a few rAM3 users have been running experiments using their own UM executables with modified source code.

So before NRI removes the software dependencies in /g/data/access and fully commits to spack builds, the user community will need to develop competency in deploying UM spack builds.

10

From my point of view maintaining functionality of people’s existing runs is extremely important, so I expect that nothing will be removed from /g/data/access. The climate and weather community actively uses a wide variety of model versions from different eras so there’s a large history to support - from UM7.3 CMIP runs to the latest UM14 now being developed on Github.

This does have the downside that we’ve accumulated a lot of cruft over the last 15 or so years in /g/data/access - NRI using new technologies like spack to manage these dependencies in a new separate area where they’re not tied down by history is welcome :grinning_face_with_smiling_eyes:.

1 Like
11

Just to echo what @Scott said above ACCESS-NRI HAS NO INTENTION OF DELETING OR MODIFYING anything in /g/data/access.

ACCESS-NRI is using other projects (principally vk83) for deploying software, in part for the reasons @Scott explained above.

But we also are not intending to add to the existing software dependencies that would be required to upgrade in-suite builds. e.g. these are the current AM3 suite build dependencies that will not be updated:

module load gcom/7.8_ompi.4.1.4 
module load eccodes/2.8.0
module load drhook/1.1_ompi.4.1.4
module load fcm/2019.09.0
1 Like
12

Thanks for the confirmation Aidan.

Moving forward, will all LFRic / Psyclone builds use spack?

13

ACCESS-NRI builds? Yes.

spack is necessary for all infrastructure, as it allows for generic (model agnostic) workflows and build provenance, as well other desirable features.

The UKMO is moving to spack too, so I’d expect it to become fairly ubiquitous in the future, so the “yes” above might be for all LFRic builds, but @Scott would know more about it than I.

2 Likes
14

Momentum will be using spack to install dependencies, using the same package recipes and environment definitions across the partnership, but models themselves will continue to be built by the workflows as is done with current UM runs.

3 Likes
15

Ah, that’s great news. We’re ahead of the curve :wink:

1 Like
16

Any progress on the local spack build?

17

There is an existing user guide for modifying and building ACCESS models directly.

However models using newer UM versions were always a bit problematic because the source code was only available on the MOSRS subversion server, or the local mirror.

We’re in the process of moving to entirely GitHub based builds, as well as transitioning to spack v 1.1 which will allow us to have a much simpler setup process to use the installed ACCESS-NRI spack instance and just module load spack.

So right now is a uniquely bad time for this! I hope to be able to give you a much more satisfactory answer in a couple of weeks.

2 Likes