# /g/data/ik11/ cleanup **URL:** \ **Category:** COSIMA\ **Created:** [12 June 2024 00:37 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153 "2024-06-12T00:37:33Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [14 June 2024 01:50 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/21 "2024-06-14T01:50:20Z")
`/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_iaf_4hourly` was for generating a movie - I’ll delete unless somebody tells me today that they need it ---
**Author:** ![AndyHoggANU](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/andyhogganu/32/43_2.png) [@AndyHoggANU](https://forum.access-hive.org.au/u/AndyHoggANU)\ **Post date:** [14 June 2024 02:22 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/22 "2024-06-14T02:22:35Z")
OK, now started looking more closely at panant. I think the following are out of date and no longer needed: - `panant-01-hycom` - `panant-01-zstar` - `panant-hycom1` - `panant-v2` **I will delete these next week unless I hear objections.** These are all older runs that have been superseded. I would quite like to keep `panant-hycom1-v13` for a while, in addition to `panant-01-zstar-v13` if we can. Lastly, I see that `panant-005-zstar-ACCESSyr2` is enormous. It’s an important run, but is there a chance we can trim some diagnostics? Once we do all this we’ll need to replace panant.db. Note that we don’t yet have a catalogue builder for MOM6-SIS2 — so can’t yet switch these outputs to the intake catalogue system … that could be rectified if important. ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [14 June 2024 05:35 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/23 "2024-06-14T05:35:48Z")
Thanks @AndyHoggANU and @aekiss for jumping in and deleting stuff! So good. Yes, @schmidt-christina @Wilton_Aguiar it would be great if we could trim down `panant-005-zstar-ACCESSyr2`. There are a lot of restarts there, each 100 GB, we probably only need yearly restarts at a maximum (ideally for end of Dec - but looks like we’ve preserved them at a weird non-annual frequency). Anyway we can delete a bunch. We then have ~6 TB output per year for years 2000-2010, which includes daily `u, v, T, S, T_adx_2d` (which confusingly seems to be 3d not 2d!), `T_ady_2d` on z levels AND daily `umo, vmo` on rho levels. Who is using which of these variables for which years? Maybe the easiest approach is to trim down to only 5 years or so not 11 years of daily data? Or think about which variables we want most? Having both `u, v` and `umo, vmo` seems rather overkill? @Matthis_Auger @ongqingyee you may want to weigh in here also. Which variables/years have you used? ---
**Author:** ![rmholmes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/rmholmes/32/117_2.png) [@rmholmes](https://forum.access-hive.org.au/u/rmholmes)\ **Post date:** [18 June 2024 02:33 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/24 "2024-06-18T02:33:43Z")
> [@rmholmes](#): > > `g_era5_iaf` → active use. Keep. > `025deg_era5_ryf` → Cut down storage by deleting everything except scalars and 3D monthly T/S for all years except the last 10. > `025deg_jra55_iaf_era5comparison` → Apples-to-apples comparison with `025deg_era5_iaf`, keep. > `025deg_jra55_iaf_omip2_cycle1-5` → first five cycles of OMIP-2. I don’t think anyone looks at these, so I would propose deleting everything except scalars. > `025deg_jra55_iaf_omip2_cycle6` and `025deg_jra55_iaf_omip2_cycle6_jra55v150_extension` → active use. Keep. > `025deg_jra55_iaf_omip2_straits*` → Looks like these were done by @AndyHoggANU? > `025deg_jra55_ryf9091_bgc` → Done by @Aidan. > `025deg_jra55_ryf9091_gadi` → A well spunup RYF run (550 years). I would propose cutting down on storage by deleting most outputs (looks like last ~250 years have more output saved, we could cut that down to last ~50). > `025deg_jra55_ryf9091_gadi_noGM/norediGM` → Take very little space. > `025deg_jra55_ryf9091_gadi_GMhi` → @AndyHoggANU > `025deg_jra55_ryf_era5comparison` → Apples to apples comparison with `025deg_era5_ryf`. Again, I would propose cutting down storage by deleting everything except scalars and 3D monthly T/S for all years except the last 10. > > I don’t know anything about the runs in `/g/data/cj50/access-om2/cf-compliant/access-om2-025/`. The readme in there suggests they were done by Andy, Andrew and Abhishek? If I don’t hear back by the end of the week I will follow through with my deletion plan as above for the `025deg_era_ryf`, `025deg_jra55_iaf_omip2_cycle1-5`, `025deg_jra55_ryf9091_gadi` and `025deg_jra55_ryf_era5comparison` runs. ---
**Author:** ![anton](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/anton/32/1836_2.png) [@anton](https://forum.access-hive.org.au/u/anton)\ **Post date:** [18 June 2024 04:35 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/25 "2024-06-18T04:35:06Z")
> [@rmholmes](#): > > `025deg_jra55_iaf_omip2_cycle1-5` → first five cycles of OMIP-2. I don’t think anyone looks at these, so I would propose deleting everything except scalars. I am far from knowing all the context here, but I guess we want these for comparison to the same cycles when run with OM3. Also, they are [“official” control experiments](https://forum.access-hive.org.au/t/access-om2-control-experiments/258) and published in OMIP, so at worst they should go onto massdata? ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [18 June 2024 05:27 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/26 "2024-06-18T05:27:09Z")
Same goes for `025deg_jra55_ryf9091_gadi`: it’s also on the list of control experiments and would be useful to keep for comparison with ACCESS-OM3, so we should be choosy about what variables we discard. ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [18 June 2024 05:31 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/27 "2024-06-18T05:31:11Z")
> [@rmholmes](#): > > I don’t know anything about the runs in `/g/data/cj50/access-om2/cf-compliant/access-om2-025/`. The readme in there suggests they were done by Andy, Andrew and Abhishek? These are from the [2020 GMD paper](https://doi.org/gmd-13-401-2020). They’ve been superseded, so I’m not sure if they are of much interest to anyone anymore. ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [18 June 2024 05:43 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/28 "2024-06-18T05:43:07Z")
> [@aekiss](#): > > `/g/data/cj50/access-om2/cf-compliant/access-om2-025/`. > > These are from the [2020 GMD paper](https://doi.org/gmd-13-401-2020). They’ve been superseded, so I’m not sure if they are of much interest to anyone anymore. Except once they’re on cj50 we can’t ever delete right? ---
**Author:** ![rmholmes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/rmholmes/32/117_2.png) [@rmholmes](https://forum.access-hive.org.au/u/rmholmes)\ **Post date:** [18 June 2024 22:44 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/31 "2024-06-18T22:44:27Z")
> [@anton](#): > > I am far from knowing all the context here, but I guess we want these for comparison to the same cycles when run with OM3. Also, they are [“official” control experiments](https://forum.access-hive.org.au/t/access-om2-control-experiments/258) and published in OMIP, so at worst they should go onto massdata? Comparison to similar cycles for OM3 is possibly the only use case I can think of for these, as most users will just want cycle6. Putting them on massdata is a good idea - do we have a massdata space on `ik11` ? > [@aekiss](#): > > Same goes for `025deg_jra55_ryf9091_gadi`: it’s also on the list of control experiments and would be useful to keep for comparison with ACCESS-OM3, so we should be choosy about what variables we discard. As I said, my plan would be to delete most outputs except for the last 50 years. At the moment there is relatively comprehensive output for the last 250 years, which I think is overkill. Maybe 100 years is the right amount to keep? But again, if you want an apples-to-apples comparison with OM3, then I guess it’s easier to keep some of the earlier data so you don’t have to spinup OM3 for as long. ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [19 June 2024 05:42 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/32 "2024-06-19T05:42:30Z")
Our initial tests of ACCESS-OM3 will start by comparing to the initial spinups, so the early parts of these runs are needed for that. Also, some applications might prefer to use the early IAF cycles which will be closer to the initial conditions (and hence observed climatology). We are talking about 25TB though, which seems rather a lot. ```auto 5.1T /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi 20T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle[1-5] ``` I can’t think of many uses for the 5.3T of daily outputs for the IAF ```auto 5.3T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle[1-5]/output*/ocean/ocean_daily.nc ``` ---
**Author:** ![PSpence](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/pspence/32/298_2.png) [@PSpence](https://forum.access-hive.org.au/u/PSpence)\ **Post date:** [19 June 2024 07:46 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/33 "2024-06-19T07:46:01Z")
I’m happy to tar and move things to massdata for us if given a list of expts/data to move. Its a bit of pain, but I’ve streamlined the process lately. Which massdata project can we use? ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [19 June 2024 10:04 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/34 "2024-06-19T10:04:17Z")
Thanks @PSpence. There’s an mdss tool here that might be useful [GitHub - coecms/mdssdiff: Look for differences between local filesystem and it's copy on a mass data store system](https://github.com/coecms/mdssdiff) ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [19 June 2024 10:48 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/35 "2024-06-19T10:48:45Z")
> [@aekiss](#): > > ```auto > 5.1T /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi > 20T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle[1-5] > > ``` If we need these 1/4deg runs for OM3 development comparison, that’s probably happening within the next year right? So perhaps best to keep them on gdata and then just delete next year when that development is done? And I agree, deleting the daily data now seems like a good idea. ---
**Author:** ![PSpence](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/pspence/32/298_2.png) [@PSpence](https://forum.access-hive.org.au/u/PSpence)\ **Post date:** [19 June 2024 23:37 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/36 "2024-06-19T23:37:26Z")
Yeah, I am a pro at that tool now 🙂 Its super useful. Let me know which sims to tar and mdss and i can start. ---
**Author:** ![ongqingyee](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ongqingyee/32/126_2.png) [@ongqingyee](https://forum.access-hive.org.au/u/ongqingyee)\ **Post date:** [19 June 2024 23:59 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/37 "2024-06-19T23:59:33Z")
For `panant-005-zstar-ACCESSyr2` and `panant-01-zstar-ACCESSyr2`, I’ve used the daily `zos`, `aice` (daily and monthly), and `tau_x` and `tau_y`. I would also like to keep `umo_2d` `vmo_2d` just in case but I don’t know if it is as much of a data problem? For the record, I was using the whole 1950-2180 years of `eta_t` daily timeseries of the `01deg_jra55v13_ryf9091` run and think this long timeseries would be good to keep in general. ---
**Author:** ![rmholmes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/rmholmes/32/117_2.png) [@rmholmes](https://forum.access-hive.org.au/u/rmholmes)\ **Post date:** [20 June 2024 22:50 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/38 "2024-06-20T22:50:16Z")
Ok if I don’t hear further by the end of the day today I will delete `025deg_jra55_iaf_omip2_cycle[1-5]/output*/ocean/ocean_daily.nc`, as well as cutting out most of the diagnostics from earlier years in `025deg_era5_ryf` and `025deg_jra55_ryf_era5comparison`. Everything else I’ll leave as is. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [21 June 2024 06:45 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/39 "2024-06-21T06:45:34Z")
> [@rmholmes](#): > > `025deg_jra55_ryf9091_bgc` → Done by @Aidan. Not guilty m’lud. I just made the directory, it is @PSpence’s simulation. Sorry for the late reply. ---
**Author:** ![Midway-X](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/midway-x/32/1980_2.png) [@Midway-X](https://forum.access-hive.org.au/u/Midway-X)\ **Post date:** [5 July 2024 05:38 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/40 "2024-07-05T05:38:35Z")
I recently ran a 60-year 0.25-degree BGC cycle with IAF from the existing 025deg\_jra55\_iaf\_omip2\_cycle6 (which I call cycle7-bgc). As there is no IAF-BGC cycle for 0.25-degree yet, I think my outputs and restarts may be useful for the community. **Should I copy these files to ik11 or cj50?** - Due to storage limits, outputs only include carbonate variables (adic, dic, paco2, alkalinity), oxygen, and a very limited number of physical variables (temperature, salinity, potential density, and meridional transport) for diagnostics. Restart files are saved every year. ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [5 July 2024 05:57 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/41 "2024-07-05T05:57:27Z")
Hi @Midway-X, how much storage are we talking about? Maybe people can comment if they think they would use this simulation. ---
**Author:** ![Lizzie](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@Lizzie](https://forum.access-hive.org.au/u/Lizzie)\ **Post date:** [5 July 2024 06:11 UTC](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153/42 "2024-07-05T06:11:09Z")
I would be interested to look at the dic and paco2 fields [Previous page](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153.md?page=1) [Next page](https://forum.access-hive.org.au/t/g-data-ik11-cleanup/2153.md?page=3)