Apologies - I’ve recently discovered the ocean and sea ice data are inconsistent in 01deg_jra55v140_iaf_cycle4 from 2011-07-01 (output962) to the end of the cycle (2018-12-31).
How this occurred:
Last year I accidentally deleted sea ice data from April 2002 to the end of 2018 (apart from 6-hourly ice area from 2014-2016 inclusive) during post-processing prior to sharing the data.
I re-generated the deleted sea ice data in 01deg_jra55v140_iaf_cycle4_rerun_from_2002 (which also added daily aicen, vicen and 3-hourly uvel, vvel, divu, shear for 2016).
This re-generated data (including the extra 2016 outputs) was added to the 01deg_jra55v140_iaf_cycle4 output directory on cj50.
The re-run 01deg_jra55v140_iaf_cycle4_rerun_from_2002 was identical to the original 01deg_jra55v140_iaf_cycle4 for about half the run (from April 2002 until the run starting 2011-07-01). The re-run ocean and sea ice then started to deviate from the original, for reasons I’m yet to figure out (we’ve seen this once before). By the end of 2018 there are visually obvious differences in eddy locations between the runs (thanks to chaos).
Consequently
all non-6-hourly sea ice data in 01deg_jra55v140_iaf_cycle4 is inconsistent with the ocean data from 2011-07-01 onwards
the 6-hourly ice area (2014-2016 inclusive) is inconsistent with the other sea ice variables, but consistent with the ocean
I’ll do some experiments to see whether the reproducibility failure occurred in the original run or the re-run. If the latter, it may be possible to correct it.
01deg_jra55v140_iaf_cycle4_jra55v150_extension was not affected by this inconsistency issue, in case anyone was wondering. The initial condition was from 01deg_jra55v140_iaf_cycle4.
I’ve confirmed that there was a non-reproducible error of some sort in the original 01deg_jra55v140_iaf_cycle4 experiment, so it’s the ocean data that’s bad, not the sea ice.
The difference starts as a very small perturbation, so as far as I can see the ocean data is still credible (a different sample from the same statistical distribution in this turbulent flow). We probably should retain this data despite this flaw, as it has been used in publications such as @LaurieM’s. This is an analogous situation to the glitch in 01deg_jra55v140_iaf_cycle3, which affected all subsequent runs.
It’s unclear as yet what field had the initial error. sea_level first loses reproducibility on 2011-09-27 (almost the end of the run starting 2011-07-01). Other fields would also have lost reproducibility by that time, due to sea_level being coupled directly or indirectly to everything else.
If it’s just Laurie’s paper that uses that data, I would vote for NOT keeping the old data on gdata. We could move it to massdata and replace with the new data. Laurie’s data availability statement is: “The modelling outputs presented here have been deposited on UNSWorks (https://unsw-primo.hosted.exlibrisgroup.com/primo-explore/search?vid=UNSWORKS). A doi will be generated upon acceptance of the manuscript.”
We have serious storage limitations on cj50 and ik11 right now (~300 Tb of data on scratch that we don’t have space to transfer to gdata!).
I guess deleting the old data would mean rerunning the extension also? What’s the cost of that?
The costs would be around 2MSU for the last 8 years of cycle 4 plus 1.2MSU for the extension.
And probably 2 weeks of somebody’s time.
We should also set up some system of versioning to track output data issues like this.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
11
Would it be possible to run the simulation backwards from the first non-reproducible restart and re-create the ice data to the point where the simulations diverge?
If the change is small and random, and has not massive impact on the utility/fidelity of the output, then it really only matters that ice data that matches the ocean data can be created, and that the problem is noted in the metadata so simulations aren’t forked from the before that error.
We only have annual restarts from the original run. We could run forwards from the first available non-reproducible restart from the original run (2012-01-01), which would hopefully remain consistent with the original run to the end, giving us ice data consistent with the ocean data from 2012-01-01 to 2018-12-31.
This seems like the most economical solution in terms of storage, but since we can’t reproduce the glitch in the original run on 2011-09-27 I see no way to generate sea ice data consistent with the original run ocean state between 2011-09-27 and 2012-01-01. There is data from 01deg_jra55v140_iaf_cycle4_repro_test over this period which is consistent between ocean and ice within that run but differs from the ocean in 01deg_jra55v140_iaf_cycle4 from 2011-09-27 onwards.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
13
Maybe I don’t understand the problem correctly!
I am wondering if it is possible to run the model backwards? I have no idea. In theory models can be run backwards, but in practice that might be impossible.
If it were possible, then it should be possible to run the model backwards from the original control run 01deg_jra55v140_iaf_cycle4 backwards from 2012-01-01 thus creating ocean and ice data to 2011-09-27, when the simulations diverge.
The point where they join becomes slightly problematic, but presumably it could be stitched together in some way. No?
That would be nice, but it’s not possible with these models. Diffusion and advective stretching map multiple model states to the same result if run forward in time (due to limited numerical precision and grid resolution), so when run backward in time you generally wouldn’t get the state you started with (i.e. it’s deterministic in both directions, but information is lost in the forward direction so the initial state can’t reconstructed by running backwards). The diffusive terms would amplify grid-scale variability when run backwards, creating a noisy mess.
1 Like
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
15
01deg_jra55v140_iaf_cycle4 had a non-reproducible glitch at roughly 2011-09-27
This seems to have been a tiny initial perturbation, but the difference eventually became macroscopic due to chaos.
I think the ocean data is therefore credible, since it’s just another realisation of turbulence from the same statistical distribution; the glitch was possibly as insignificant as the (reproducible) roundoff errors that happen all the time.
We don’t know how often these non-reproducible glitches happen (since they are only detected if we re-run) so there may be several other examples of this issue in our datasets (we know there was another case in cycle 3 affecting all subsequent cycles).
The main problem is we don’t have any sea ice data (except 6-hourly) that is consistent with the 01deg_jra55v140_iaf_cycle4 ocean data from 2011-09-27 onwards, and we can’t generate it due to the non-reproducible glitch in the original run.
the post-2002 non-6-hourly sea ice data in 01deg_jra55v140_iaf_cycle4 was actually generated by 01deg_jra55v140_iaf_cycle4_rerun_from_2002, and is inconsistent with the ocean state from 2011-09-27 onwards
The simplest solution would be to remove the post-2011-09 non-6-hourly sea ice data from 01deg_jra55v140_iaf_cycle4 and put it into 01deg_jra55v140_iaf_cycle4_rerun_from_2002, which would have no ocean data.
That way users could combine these if they wish but it would be clear that they are different experiments.
It gets more difficult if there’s anyone who needs consistent ocean and sea ice data post-2011-09. This would require a re-run of the last 8 years of cycle 4, starting 1 Jan 2011, costing about 2MSU, 2 weeks and 6TB.
