Hi @cbengel and team
I am using ACCESS-rAM3 and I want to know if I run the exactly same (identical) suite twice then can i expect differences in the output? I expect the output to be exactly same, pixel by pixel. Runs should be reproducible. Is it a common behaviour of the model or anyone have seen such issue?
Thank you in advance!
Hi @Leena,
The results should be exactly the same.
If you are having issues with reproducibility can you please provide extra information so people can chase this up?
Thanks,
Chermelle
Hi @cbengel , Thanks Chermelle. I am using stashpack6 in SHEP, in which they mention about packing=5 .. what does it do with model? Thanks?
Hi @Leena, to view the contents of a STASHpack via the UI, please do the following steps (please pay attention to the green information box telling you to take a copy of your original stash request so you can revert back to the original behaviour):
Then in the rose UI go to:
um → namelist → Model Input and Output → STASH Requests and Profiles → STASH requests
You can then double-click on each STASH request individually to bring up the properties of each request.
This should be able to give you more information.
Packing is likely related to the precision-level in the output files.
@cbengel , in any case, so packing is not an issue. My concern is, I used same stashpack6 .. its identical suite, ran twice, then how can i get different results. What else do i need to check for this?
There are many different reasons this could happen. Some innocent, some that would need more chasing down.
Can you double/triple check that you have not accidentally modified your setting between runs?
Hi Leena, can you tell us what you’re seeing that is different?
Are you comparing to runs done before / after Paul’s optiisation of the SHEP suite?
I know that Paul reverted the compiler he was using on the sapphire rapids to a different version from the one recommended by NCI. A compiler change will cause non-bit-reproducible output between the two executables compiled with different compilers.
HI @bethanwhite , I started both the runs today morning. just ran same suite twice.
Also, i havent updated the suite recently so not sure if thats the case
Hi @Leena,
This would take a bit more digging around.
I have been running the ACCESS-rAM3 (non-Flagship) suite recently, comparing outputs and coming up with identical results.
It might be worth seeing if those in the community running the Flagship run are finding similar issues or not.
I am away tomorrow but will look into this next week.
Hmmm ok. Can you leave the suite ID here so Chermelle can access it when she is back?
Can you also tell us what fields you are seeing differences in, and the level of difference (% of change between the two runs)?
Hi all, I’ve done additional testing of this issue. My version of u-dx356 has not yet incorporated any of the minor changes Leena has been making this week, so we can consider it independent.
I ran two runs of the exact same case in parallel (only changing the name of the output). The model outputs were quantitatively different. For example, here are some time series of 12p2 and 4p4 output below for different grid cells around the domain:
Hi @bethanwhite and @cbengel ,
We have now updated the suite, as Bethan mentioned that @Paul.Gregory had made changes. The suite is u-dx356.
However, we still observe the reproducibility issue in this version. @sdahl found differences of approximately 8–9%.
First of all, thank you for raising the issue, @Leena and confirming @sdahl.
I went through and ran the u-dx356 suite four times.
Twice with the SPACK-executable (identical inputs) (top row: run 1, run 2, then the difference), and twice with the suite-built executable (identical inputs) (bottom row: run 1, run 2, then the difference).
The SPACK-built executable runs (top row) had differing results between runs, confirming the reproducibility issue you raised.
The suite-built executable runs though produced identical results (bottom row). I do my testing on the suite-built executable and have verified this many times.
** Please note, the statistics shown are from the difference plots.
Here is a screenshot of a zoomed in portion of the SPACK-built run difference plot.
It was a known issue that outputs from the SPACK-built executable run did not match the outputs from the suite-built run (and this was being looked into by members of the ACCESS-NRI team). However, it was not known that the SPACK-built runs were producing inconsistent results between runs.
For your purposes, you can produce reproducible results by using a suite-built executable.
It takes about 10-minutes to build an executable, so you don’t want to have to do that every time. In practice we build the executable once, then use the same executable for subsequent runs.
To build the executable the first time go to:
Suite conf → Nesting Suite → General run option → BUILD_MODE.
Set
BUILD_MODE = Build a new executable.
The next time you run, set
BUILD_MODE = Use an executable made by a previous run of this suite.
You can thenceforth use that setting for subsequent runs as long as you don’t delete the executable.
In the meantime, I will feedback the issue to the team that built the SPACK executable.
Best regards,
Excellent @cbengel , Thank you for identifying the exact issue and providing the solution. Much appreciated. Have a great day ahead!
If you need any extra information on how to build your own code, @Paul.Gregory has written these notes:
@cbengel is right, it’s an easy process and it only takes a few minutes to compile, so you’ll be back up and running very soon.
Once the code has built, it will create these two directories (the .exe files sit inside a /bin subdirectory in each):
~/cylc-run/<suite-id>/share/fcm_make_um/build-atmos
~/cylc-run/<suite-id>/share/fcm_make_um/build-recon
Because this is on scratch, and because it is in your cylc suite directory, they are at risk of being deleted by the eventual scratch purge policy and / or if you accidentally run a rose suite-clean.
To mitigate this risk, you can copy the entire directories above (i.e. not just the .exe files) to somewhere on your home directory or on gdata, and then point your suite to that location (see the final section of Paul’s documentation for how to do this).
e.g. you could make a directory on gdata called model_builds/rAM3-SHEP/, and place the build-recon and build-atmos directories inside that.
Some excellent detective work by @MartinDix has identified the cause of this problem, and why it does appear when using the Spack build but not for the in-suite build.
There is a UM namelist option global_sum_method which determines how to perform global horizontal sums. rAM3 uses global_sum_method=3, which is just an mpi_reduce under the hood (AM3 uses 2, which is why it hasn’t shown up there). This method is not guaranteed to be reproducible, and it turns out that openmpi/4.1.5 does produce reproduce across runs (which the in-suite build uses) while openmpi/4.1.7 doesn’t reproduce (which the Spack build uses).
I think the best solution in the immediate is to set global_sum_method=2. This will reproduce results with a negligible effect on performance. We’re in the process of upgrading all of our models to use openmpi/5.0.8, which should hopefully fix the reproducibility issue even with global_sum_method=3.
