# ACCESS-rAM3 run reproducibility **URL:** \ **Category:** General\ **Tags:** help\ **Created:** [10 September 2026 06:25 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980 "2026-09-10T06:25:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [10 September 2026 06:25 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/1 "2026-09-10T06:25:32Z")
Hi @cbengel and team I am using ACCESS-rAM3 and I want to know if I run the exactly same (identical) suite twice then can i expect differences in the output? I expect the output to be exactly same, pixel by pixel. Runs should be reproducible. Is it a common behaviour of the model or anyone have seen such issue? Thank you in advance! ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [10 September 2026 06:27 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/2 "2026-09-10T06:27:31Z")
Hi @Leena, The results should be exactly the same. If you are having issues with reproducibility can you please provide extra information so people can chase this up? Thanks, Chermelle ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [10 September 2026 06:32 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/3 "2026-09-10T06:32:41Z")
Hi @cbengel , Thanks Chermelle. I am using stashpack6 in SHEP, in which they mention about packing=5 .. what does it do with model? Thanks? ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [10 September 2026 07:03 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/4 "2026-09-10T07:03:52Z")
Hi @Leena, to view the contents of a STASHpack via the UI, please do the following steps (please pay attention to the green information box telling you to take a copy of your original stash request so you can revert back to the original behaviour): > **[View optional STASHpack contents via UI - ACCESS-rAM3-configs](https://access-ram3-configs.access-hive.org.au/model_outputs/view_stashpack_contents_via_ui/)** Then in the rose UI go to: _um → namelist → Model Input and Output → STASH Requests and Profiles → STASH requests_ You can then double-click on each STASH request individually to bring up the properties of each request. This should be able to give you more information. Packing is likely related to the precision-level in the output files. ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [10 September 2026 07:15 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/5 "2026-09-10T07:15:26Z")
@cbengel , in any case, so packing is not an issue. My concern is, I used same stashpack6 .. its identical suite, ran twice, then how can i get different results. What else do i need to check for this? ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [10 September 2026 07:16 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/6 "2026-09-10T07:16:58Z")
There are many different reasons this could happen. Some innocent, some that would need more chasing down. Can you double/triple check that you have not accidentally modified your setting between runs? ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [10 September 2026 07:18 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/7 "2026-09-10T07:18:40Z")
Hi Leena, can you tell us what you’re seeing that is different? Are you comparing to runs done before / after Paul’s optiisation of the SHEP suite? I know that Paul reverted the compiler he was using on the sapphire rapids to a different version from the one recommended by NCI. A compiler change will cause non-bit-reproducible output between the two executables compiled with different compilers. ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [10 September 2026 07:20 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/8 "2026-09-10T07:20:30Z")
HI @bethanwhite , I started both the runs today morning. just ran same suite twice. ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [10 September 2026 07:22 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/9 "2026-09-10T07:22:12Z")
Also, i havent updated the suite recently so not sure if thats the case ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [10 September 2026 07:23 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/10 "2026-09-10T07:23:04Z")
Hi @Leena, This would take a bit more digging around. I have been running the ACCESS-rAM3 (non-Flagship) suite recently, comparing outputs and coming up with identical results. It might be worth seeing if those in the community running the Flagship run are finding similar issues or not. I am away tomorrow but will look into this next week. ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [10 September 2026 07:24 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/11 "2026-09-10T07:24:51Z")
Hmmm ok. Can you leave the suite ID here so Chermelle can access it when she is back? Can you also tell us what fields you are seeing differences in, and the level of difference (% of change between the two runs)? ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [10 September 2026 07:30 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/12 "2026-09-10T07:30:03Z")
Hi Bethan and Chermelle, its u-dx356. the differences are around 0-1.5%. These differences are in almost all fields. ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/4/4ac3ae1626fa1962ffc2086687da07fdd5861982.png) ---
**Author:** ![edoyango](https://avatars.discourse-cdn.com/v4/letter/e/bbe5ce/32.png) [@edoyango](https://forum.access-hive.org.au/u/edoyango)\ **Post date:** [10 September 2026 22:46 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/13 "2026-09-10T22:46:51Z")
---
**Author:** ![sdahl](https://avatars.discourse-cdn.com/v4/letter/s/f9ae1b/32.png) [@sdahl](https://forum.access-hive.org.au/u/sdahl)\ **Post date:** [11 September 2026 02:35 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/14 "2026-09-11T02:35:14Z")
Hi all, I’ve done additional testing of this issue. My version of u-dx356 has not yet incorporated any of the minor changes Leena has been making this week, so we can consider it independent. I ran two runs of the exact same case in parallel (only changing the name of the output). The model outputs were quantitatively different. For example, here are some time series of 12p2 and 4p4 output below for different grid cells around the domain: ![download-15](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/e/ebed80645b2591ac56dee725754818c118a0f704.png) ![download-14](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/8/8f23334ed4f50b975ad9e1800a8762ef53b65da7.png) ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [13 September 2026 05:24 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/15 "2026-09-13T05:24:32Z")
Hi @bethanwhite and @cbengel , We have now updated the suite, as Bethan mentioned that @Paul.Gregory had made changes. The suite is `u-dx356`. However, we still observe the reproducibility issue in this version. @sdahl found differences of approximately 8–9%. ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [14 September 2026 00:07 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/16 "2026-09-14T00:07:01Z")
In the interest of not flooding the ACCESS Hive with posts I will email you @Leena and @sdahl offline and summarise later what we find. ---
**Author:** ![cbengel](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@cbengel](https://forum.access-hive.org.au/u/cbengel)\ **Post date:** [14 September 2026 22:52 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/17 "2026-09-14T22:52:49Z")
First of all, thank you for raising the issue, @Leena and confirming @sdahl. I went through and ran the u-dx356 suite four times. Twice with the SPACK-executable (identical inputs) (top row: run 1, run 2, then the difference), and twice with the suite-built executable (identical inputs) (bottom row: run 1, run 2, then the difference). The SPACK-built executable runs (top row) had differing results between runs, confirming the reproducibility issue you raised. The suite-built executable runs though produced identical results (bottom row). I do my testing on the suite-built executable and have verified this many times. \*\* Please note, the statistics shown are from the difference plots. ![Reproducibility Investigation Kelvin](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/a/a62656d971f17ec252cc4771d0d4cd2a58cf1def.jpeg) Here is a screenshot of a zoomed in portion of the SPACK-built run difference plot. ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/4/4eaa5ece2b4cc41ac3c467db6bb9ac341882dec7.png) It was a known issue that outputs from the SPACK-built executable run did not match the outputs from the suite-built run (and this was being looked into by members of the ACCESS-NRI team). However, it was not known that the SPACK-built runs were producing inconsistent results between runs. For your purposes, you can produce reproducible results by using a suite-built executable. It takes about 10-minutes to build an executable, so you don’t want to have to do that every time. In practice we build the executable once, then use the same executable for subsequent runs. To build the executable the first time go to: Suite conf → Nesting Suite → General run option → BUILD\_MODE. Set **BUILD\_MODE = Build a new executable.** The next time you run, set **BUILD\_MODE = Use an executable made by a previous run of this suite.** You can thenceforth use that setting for subsequent runs as long as you don’t delete the executable. In the meantime, I will feedback the issue to the team that built the SPACK executable. Best regards, @cbengel ---
**Author:** ![Leena](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/leena/32/4309_2.png) [@Leena](https://forum.access-hive.org.au/u/Leena)\ **Post date:** [14 September 2026 23:07 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/18 "2026-09-14T23:07:23Z")
Excellent @cbengel , Thank you for identifying the exact issue and providing the solution. Much appreciated. Have a great day ahead! ---
**Author:** ![bethanwhite](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/bethanwhite/32/3080_2.png) [@bethanwhite](https://forum.access-hive.org.au/u/bethanwhite)\ **Post date:** [14 September 2026 23:54 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/19 "2026-09-14T23:54:57Z")
Hi @Leena and @sdahl, If you need any extra information on how to build your own code, @Paul.Gregory has written these notes: > **[Custom UM builds for rAM3 — 21st Century Weather Software Wiki](https://21centuryweather.github.io/21st-Century-Weather-Software-Wiki/NRI-RNS/build-UM.html)** @cbengel is right, it’s an easy process and it only takes a few minutes to compile, so you’ll be back up and running very soon. Once the code has built, it will create these two directories (the `.exe` files sit inside a `/bin` subdirectory in each): `~/cylc-run//share/fcm_make_um/build-atmos` `~/cylc-run//share/fcm_make_um/build-recon` Because this is on scratch, and because it is in your cylc suite directory, they are at risk of being deleted by the eventual scratch purge policy and / or if you accidentally run a `rose suite-clean`. To mitigate this risk, you can copy the _entire_ directories above (i.e. not just the .exe files) to somewhere on your home directory or on gdata, and then point your suite to that location (see the final section of Paul’s documentation for how to do this). e.g. you could make a directory on gdata called `model_builds/rAM3-SHEP/`, and place the `build-recon` and `build-atmos` directories inside that. ---
**Author:** ![lachlanswhyborn](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/lachlanswhyborn/32/1837_2.png) [@lachlanswhyborn](https://forum.access-hive.org.au/u/lachlanswhyborn)\ **Post date:** [21 September 2026 23:33 UTC](https://forum.access-hive.org.au/t/access-ram3-run-reproducibility/6980/20 "2026-09-21T23:33:07Z")
Some excellent detective work by @MartinDix has identified the cause of this problem, and why it does appear when using the Spack build but not for the in-suite build. There is a UM namelist option `global_sum_method` which determines how to perform global horizontal sums. rAM3 uses `global_sum_method=3`, which is just an `mpi_reduce` under the hood (AM3 uses `2`, which is why it hasn’t shown up there). This method is not _guaranteed_ to be reproducible, and it turns out that `openmpi/4.1.5` _does_ produce reproduce across runs (which the in-suite build uses) while `openmpi/4.1.7` _doesn’t_ reproduce (which the Spack build uses). I think the best solution in the immediate is to set `global_sum_method=2`. This will reproduce results with a negligible effect on performance. We’re in the process of upgrading all of our models to use `openmpi/5.0.8`, which should hopefully fix the reproducibility issue even with `global_sum_method=3`.