/g/data/ik11/ cleanup

[aekiss (Andrew Kiss)]

/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_iaf_4hourly
was for generating a movie - I’ll delete unless somebody tells me today that they need it

[AndyHoggANU (Andy Hogg)]

OK, now started looking more closely at panant. I think the following are out of date and no longer needed:

• panant-01-hycom
• panant-01-zstar
• panant-hycom1
• panant-v2

I will delete these next week unless I hear objections.

These are all older runs that have been superseded. I would quite like to keep panant-hycom1-v13 for a while, in addition to panant-01-zstar-v13 if we can.

Lastly, I see that panant-005-zstar-ACCESSyr2 is enormous. It’s an important run, but is there a chance we can trim some diagnostics?

Once we do all this we’ll need to replace panant.db. Note that we don’t yet have a catalogue builder for MOM6-SIS2 — so can’t yet switch these outputs to the intake catalogue system … that could be rectified if important.

[adele157 (Adele Morrison)]

Thanks @AndyHoggANU and @aekiss for jumping in and deleting stuff! So good.

Yes, @schmidt-christina @Wilton_Aguiar it would be great if we could trim down panant-005-zstar-ACCESSyr2.

There are a lot of restarts there, each 100 GB, we probably only need yearly restarts at a maximum (ideally for end of Dec - but looks like we’ve preserved them at a weird non-annual frequency). Anyway we can delete a bunch.

We then have ~6 TB output per year for years 2000-2010, which includes daily u, v, T, S, T_adx_2d (which confusingly seems to be 3d not 2d!), T_ady_2d on z levels AND daily umo, vmo on rho levels. Who is using which of these variables for which years? Maybe the easiest approach is to trim down to only 5 years or so not 11 years of daily data? Or think about which variables we want most? Having both u, v and umo, vmo seems rather overkill?

@Matthis_Auger @ongqingyee you may want to weigh in here also. Which variables/years have you used?

[rmholmes (Ryan Holmes)]

@rmholmes

g_era5_iaf → active use. Keep.
025deg_era5_ryf → Cut down storage by deleting everything except scalars and 3D monthly T/S for all years except the last 10.
025deg_jra55_iaf_era5comparison → Apples-to-apples comparison with 025deg_era5_iaf, keep.
025deg_jra55_iaf_omip2_cycle1-5 → first five cycles of OMIP-2. I don’t think anyone looks at these, so I would propose deleting everything except scalars.
025deg_jra55_iaf_omip2_cycle6 and 025deg_jra55_iaf_omip2_cycle6_jra55v150_extension → active use. Keep.
025deg_jra55_iaf_omip2_straits* → Looks like these were done by @AndyHoggANU?
025deg_jra55_ryf9091_bgc → Done by @Aidan.
025deg_jra55_ryf9091_gadi → A well spunup RYF run (550 years). I would propose cutting down on storage by deleting most outputs (looks like last ~250 years have more output saved, we could cut that down to last ~50).
025deg_jra55_ryf9091_gadi_noGM/norediGM → Take very little space.
025deg_jra55_ryf9091_gadi_GMhi → @AndyHoggANU
025deg_jra55_ryf_era5comparison → Apples to apples comparison with 025deg_era5_ryf. Again, I would propose cutting down storage by deleting everything except scalars and 3D monthly T/S for all years except the last 10.

I don’t know anything about the runs in /g/data/cj50/access-om2/cf-compliant/access-om2-025/. The readme in there suggests they were done by Andy, Andrew and Abhishek?

If I don’t hear back by the end of the week I will follow through with my deletion plan as above for the 025deg_era_ryf, 025deg_jra55_iaf_omip2_cycle1-5, 025deg_jra55_ryf9091_gadi and 025deg_jra55_ryf_era5comparison runs.

[anton (Anton Steketee, ACCESS-NRI Oceans Team)]

@rmholmes

025deg_jra55_iaf_omip2_cycle1-5 → first five cycles of OMIP-2. I don’t think anyone looks at these, so I would propose deleting everything except scalars.

I am far from knowing all the context here, but I guess we want these for comparison to the same cycles when run with OM3. Also, they are “official” control experiments and published in OMIP, so at worst they should go onto massdata?

[aekiss (Andrew Kiss)]

Same goes for 025deg_jra55_ryf9091_gadi: it’s also on the list of control experiments and would be useful to keep for comparison with ACCESS-OM3, so we should be choosy about what variables we discard.

[aekiss (Andrew Kiss)]

@rmholmes

I don’t know anything about the runs in /g/data/cj50/access-om2/cf-compliant/access-om2-025/. The readme in there suggests they were done by Andy, Andrew and Abhishek?

These are from the 2020 GMD paper. They’ve been superseded, so I’m not sure if they are of much interest to anyone anymore.

[adele157 (Adele Morrison)]

@aekiss

/g/data/cj50/access-om2/cf-compliant/access-om2-025/.

These are from the 2020 GMD paper. They’ve been superseded, so I’m not sure if they are of much interest to anyone anymore.

Except once they’re on cj50 we can’t ever delete right?

[rmholmes (Ryan Holmes)]

@anton

I am far from knowing all the context here, but I guess we want these for comparison to the same cycles when run with OM3. Also, they are “official” control experiments and published in OMIP, so at worst they should go onto massdata?

Comparison to similar cycles for OM3 is possibly the only use case I can think of for these, as most users will just want cycle6. Putting them on massdata is a good idea - do we have a massdata space on ik11 ?

@aekiss

Same goes for 025deg_jra55_ryf9091_gadi: it’s also on the list of control experiments and would be useful to keep for comparison with ACCESS-OM3, so we should be choosy about what variables we discard.

As I said, my plan would be to delete most outputs except for the last 50 years. At the moment there is relatively comprehensive output for the last 250 years, which I think is overkill. Maybe 100 years is the right amount to keep?

But again, if you want an apples-to-apples comparison with OM3, then I guess it’s easier to keep some of the earlier data so you don’t have to spinup OM3 for as long.

[aekiss (Andrew Kiss)]

Our initial tests of ACCESS-OM3 will start by comparing to the initial spinups, so the early parts of these runs are needed for that. Also, some applications might prefer to use the early IAF cycles which will be closer to the initial conditions (and hence observed climatology).

We are talking about 25TB though, which seems rather a lot.

5.1T /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi
20T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle[1-5]

I can’t think of many uses for the 5.3T of daily outputs for the IAF

5.3T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle[1-5]/output*/ocean/ocean_daily.nc

[PSpence (Paul Spence)]

I’m happy to tar and move things to massdata for us if given a list of expts/data to move. Its a bit of pain, but I’ve streamlined the process lately. Which massdata project can we use?

[aekiss (Andrew Kiss)]

Thanks @PSpence. There’s an mdss tool here that might be useful GitHub - coecms/mdssdiff: Look for differences between local filesystem and it's copy on a mass data store system

[adele157 (Adele Morrison)]

@aekiss

5.1T /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi
20T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle[1-5]

If we need these 1/4deg runs for OM3 development comparison, that’s probably happening within the next year right? So perhaps best to keep them on gdata and then just delete next year when that development is done?

And I agree, deleting the daily data now seems like a good idea.

[PSpence (Paul Spence)]

Yeah, I am a pro at that tool now Its super useful. Let me know which sims to tar and mdss and i can start.

[ongqingyee (Ellie Ong)]

For panant-005-zstar-ACCESSyr2 and panant-01-zstar-ACCESSyr2, I’ve used the daily zos, aice (daily and monthly), and tau_x and tau_y. I would also like to keep umo_2d vmo_2d just in case but I don’t know if it is as much of a data problem?

For the record, I was using the whole 1950-2180 years of eta_t daily timeseries of the 01deg_jra55v13_ryf9091 run and think this long timeseries would be good to keep in general.

[rmholmes (Ryan Holmes)]

Ok if I don’t hear further by the end of the day today I will delete 025deg_jra55_iaf_omip2_cycle[1-5]/output*/ocean/ocean_daily.nc, as well as cutting out most of the diagnostics from earlier years in 025deg_era5_ryf and 025deg_jra55_ryf_era5comparison. Everything else I’ll leave as is.

[Aidan (Aidan Heerdegen, ACCESS-NRI Release Team Lead)]

@rmholmes

025deg_jra55_ryf9091_bgc → Done by @Aidan.

Not guilty m’lud. I just made the directory, it is @PSpence’s simulation.

Sorry for the late reply.

[Midway-X (Yinghuan Xie)]

I recently ran a 60-year 0.25-degree BGC cycle with IAF from the existing 025deg_jra55_iaf_omip2_cycle6 (which I call cycle7-bgc). As there is no IAF-BGC cycle for 0.25-degree yet, I think my outputs and restarts may be useful for the community. Should I copy these files to ik11 or cj50?

• Due to storage limits, outputs only include carbonate variables (adic, dic, paco2, alkalinity), oxygen, and a very limited number of physical variables (temperature, salinity, potential density, and meridional transport) for diagnostics. Restart files are saved every year.

[adele157 (Adele Morrison)]

Hi @Midway-X, how much storage are we talking about? Maybe people can comment if they think they would use this simulation.

[Lizzie (elizabeth ellison)]

I would be interested to look at the dic and paco2 fields