# Technical requirements for MOM6 node testing **URL:** \ **Category:** TWG\ **Tags:** mom6, testing\ **Created:** [2 August 2023 05:57 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052 "2023-08-02T05:57:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [2 August 2023 05:57 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/1 "2023-08-02T05:57:49Z")
@angus-g has put out a call for MOM6 model configurations to include in a testing suite when there is an Australian MOM6 “node” [https://forum.access-hive.org.au/t/a-call-for-community-model-configurations-that-use-mom6/1049/5](https://forum.access-hive.org.au/t/a-call-for-community-model-configurations-that-use-mom6/1049/5) This is a related discussion about the technical aspects of any such testing. ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [2 August 2023 05:23 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/2 "2023-08-02T05:23:22Z")
Here’s a couple of relevant papers discussing reproducibility and testing. The [first one](http://dx.doi.org/10.1098/rsta.2017.0344) defines 4 categories of reproducibility, and statistical tests to automate categorising the non-bit-for-bit cases > Changes, additions and updates to CICE fall into four categories: (I) BFB [bit-for-bit] with no further assessment required; (II) non-BFB but unlikely to be climate changing; (III) non-BFB and climate changing; and (IV) a new model configuration option requiring separate scientific assessment. This section describes the automated methods used to flag the first three categories. [http://dx.doi.org/10.1098/rsta.2017.0344](http://dx.doi.org/10.1098/rsta.2017.0344) > **[Machine dependence and reproducibility for coupled climate simulations: the...](https://gmd.copernicus.org/articles/13/139/2020/)** > > Abstract. When the same weather or climate simulation is run on different high-performance computing (HPC) platforms, model outputs may not be identical for a given initial condition. While the role of HPC platforms in delivering better climate... ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [2 August 2023 05:24 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/3 "2023-08-02T05:24:15Z")
> [@A call for community model configurations that use MOM6](https://forum.access-hive.org.au/t/a-call-for-community-model-configurations-that-use-mom6/1049/1): > > a git repository containing the control files (`MOM_input`, `config.yaml`, etc.); Are [payu](https://github.com/payu-org/payu) configurations desirable? Preferred? Required? How do other sites run their testing? Do they version control their inputs? If so, how? Is this even the right topic to discuss this? Happy to move to another topic if not. ---
**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\ **Post date:** [2 August 2023 05:42 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/4 "2023-08-02T05:42:10Z")
> [@Aidan](#): > > Are [payu](https://github.com/payu-org/payu) configurations desirable? Preferred? Required? I suppose most of the preexisting configurations around would probably be using payu, which is why I suggested the `config.yaml` (also gives an idea of resource requirements). But this probably ends up being a question for the technical implementation of the actual running of the tests. There’d probably be a little bit of modification required to give a testing-suitable run anyway. I think a lower barrier to entry by not requiring payu is fine? > [@Aidan](#): > > How do other sites run their testing? Do they version control their inputs? If so, how? I know that GFDL runs their tests through a pipeline on an internal Gitlab instance. I wouldn’t be surprised if there are a range of solutions from manual running, to Makefiles handed down from a supreme being (some of the developers use this for their own tests), to modern pipelines. I can try to dig around for a bit more info there. I think the control inputs are often version controlled. GFDL has MOM6-examples, ESMG has an equivalent with their configurations, etc. There are probably private configurations, but they’d be within version control on the inside of the firewall. As for other (binary) inputs, I’m not sure! That is probably an issue we’ll have to think about too, particularly for full-chain reproducibility and provenance. > [@Aidan](#): > > Is this even the right topic to discuss this? Happy to move to another topic if not. Sure 🙂 But we might want to spin out a discussion on the technical implementation (running tests, validating tests, how to organise configurations, etc.). Ideally it can all be authoritative, so we don’t get a desync between what we’re testing and what’s actually being run. But also the testing is only as valuable as the tests capturing codepaths in the model that people are actually interested in. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [2 August 2023 06:00 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/5 "2023-08-02T06:00:13Z")
I was worried I had derailed your topic @angus-g, so I’ve moved the discussion to this topic. Hope you don’t mind being scooped up and moved to @aekiss, but your post seemed to fit here quite well. I can move it back if you want me to. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [2 August 2023 06:13 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/6 "2023-08-02T06:13:26Z")
> [@angus-g](#): > > I suppose most of the preexisting configurations around would probably be using payu, which is why I suggested the `config.yaml` (also gives an idea of resource requirements). But this probably ends up being a question for the technical implementation of the actual running of the tests. There’d probably be a little bit of modification required to give a testing-suitable run anyway. I think a lower barrier to entry by not requiring payu is fine? Sorry, I didn’t notice the `config.yaml` reference. Yes I don’t think there is a problem with a lower barrier of entry, but if `payu` is the preferred way to go (and I think it is), then non-`payu` configs will have to be converted to run with `payu` in any case. I’m wondering aloud about a few things: - @MartinDix was enquiring a while ago about ways to version inputs for the rose+cylc experiments. It got me thinking about [IPFS](https://en.wikipedia.org/wiki/InterPlanetary_File_System) - Are there modifications we might usefully make to `payu` to facilitate running test cases programatically like this? The [ACCESS-OM2 testing](https://github.com/COSIMA/access-om2/blob/master/test/exp_test_helper.py) writes to `config.yaml` files. We could make it more seamless than that I reckon. > [@angus-g](#): > > But also the testing is only as valuable as the tests capturing codepaths in the model that people are actually interested in. True. The intel compiler can also generate codecov data, which might be worthwhile thinking about to quantify coverage. ---
**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\ **Post date:** [2 August 2023 06:17 UTC](https://forum.access-hive.org.au/t/technical-requirements-for-mom6-node-testing/1052/7 "2023-08-02T06:17:24Z")
> [@Aidan](#): > > True. The intel compiler can also generate codecov data, which might be worthwhile thinking about to quantify coverage. GFDL actually uses codecov on their fork, e.g.: [Fix a bug in the OMP directive for plume\_flux by Hallberg-NOAA · Pull Request #427 · NOAA-GFDL/MOM6 · GitHub](https://github.com/NOAA-GFDL/MOM6/pull/427#issuecomment-1656011146). Although that applies to the smaller regression tests that are run through GitHub Actions. You can also see an example of the Gitlab pipeline link in that PR.