Hi COSIMA folks,
/g/data/ik11 is very full, so it would be great if we could delete or archive some of the outputs there if they’re no longer needed.
First up, there are a bunch of ACCESS-OM2-025 experiments in /g/data/ik11/outputs/access-om2-025/. I guess we want to keep the main omip ones? Although is there any double up there with what’s here: /g/data/cj50/access-om2/cf-compliant/access-om2-025/ ? Is anyone still using all the different experiments here? @rmholmes @AndyHoggANU ?
Next up, old panan runs. Does anyone mind if we delete the following highlighted experiments? (Obviously keeping the current unhighlighted 1/10th, 1/20th and 1/40th simulations - and moving these to /g/data/ol01/.) Has anyone published with these blue highlighted runs? Maybe @JuliaN ? @AndyHoggANU
/g/data/ik11/outputs/access-om2-01/ is a much harder task to deal with!
There are many folders in there that I don’t know what they are (e.g. 01deg_jra55v13_iaf_4hourly; how is 01deg_jra55v13_ryf9091_qian_ctrl different from the normal ryf again? should we move that to cj50?; 01deg_jra55v140_iaf_cycle3_HF; 01deg_jra55v140_iaf_cycle4_rerun_from_*).
Also, can we consider deleting any of it? e.g. Does anyone use the start of the RYF or need long time series of it? Could we delete outputs000-195? And then perhaps also outputs280-739? And outputs 800-1075?
Maybe we can discuss at COSIMA meeting tomorrow @edoddridge since there seems to be time?
I’ve added it to the agenda for tomorrow.
Many of the runs in access-om2-025 are mine. Here are some thoughts/proposals:
025deg_era5_iaf → active use. Keep.
025deg_era5_ryf → Cut down storage by deleting everything except scalars and 3D monthly T/S for all years except the last 10.
025deg_jra55_iaf_era5comparison → Apples-to-apples comparison with 025deg_era5_iaf, keep.
025deg_jra55_iaf_omip2_cycle1-5 → first five cycles of OMIP-2. I don’t think anyone looks at these, so I would propose deleting everything except scalars.
025deg_jra55_iaf_omip2_cycle6 and 025deg_jra55_iaf_omip2_cycle6_jra55v150_extension → active use. Keep.
025deg_jra55_iaf_omip2_straits* → Looks like these were done by @AndyHoggANU?
025deg_jra55_ryf9091_bgc → Done by @Aidan.
025deg_jra55_ryf9091_gadi → A well spunup RYF run (550 years). I would propose cutting down on storage by deleting most outputs (looks like last ~250 years have more output saved, we could cut that down to last ~50).
025deg_jra55_ryf9091_gadi_noGM/norediGM → Take very little space.
025deg_jra55_ryf9091_gadi_GMhi → @AndyHoggANU
025deg_jra55_ryf_era5comparison → Apples to apples comparison with 025deg_era5_ryf. Again, I would propose cutting down storage by deleting everything except scalars and 3D monthly T/S for all years except the last 10.
I don’t know anything about the runs in /g/data/cj50/access-om2/cf-compliant/access-om2-025/. The readme in there suggests they were done by Andy, Andrew and Abhishek?
I back Ryan’s comments on iaf_jra55_025deg runs keep cylce6 runs and comparisons with era5comparioson as they are still actively used. I am not users of the ryf runs but others will be.
As Ryan says the data on cj50 looks like more it has been formatted for omip style entry to CMIP6. Not sure which cycle it matches.
Thanks Siobhan.
I ran the OMIP-2 run for CMIP6 and I’m not mentioned in the cj50 readme, so I was assuming these were older runs (not any of the ones listed above) that Abhi did ages ago?
Hi Adele,
we also need to keep panant-01-zstar-ACCESSyr2_first_run (I am using it atm) but might be able to delete some output. panant-01-zstar-ACCESSyr2_first_run and panant-01-zstar-ACCESSyr2 are identical but we saved high resolution output for panant-01-zstar-ACCESSyr2 which we hadn’t saved before. panant-01-zstar-ACCESSyr2 however only goes to 2005 whereas panant-01-zstar-ACCESSyr2_first_run was run to 2010.
Good to know Christina. We could perhaps delete /g/data/ik11/outputs/mom6-panan/panant-01-zstar-ACCESSyr2_first_run up to the end of 2005 then? Looks like we don’t have the first 10 years in any case.
@adele157 re the RYF: I’m using outputs 990-1059 as the control for the end of 21st century simulations, so would request to keep those.
Comments from COSIMA meeting today:
01deg_jra55v13_iaf_4hourly and 01deg_jra55v140_iaf_cycle3_HF.01deg_jra55v13_ryf9091_qian_ctrl is with Qian.01deg_jra55v140_iaf_cycle4_rerun_from_* are @pat_wongpan’s runs to output some extra BGC variables that were missing from the original cycle4. He is still using them.I used all of the 0.1 ryf, as well as 0.25 cycle 5. I don’t need them for anything more and have calculated the indexes I needed, so they could theoretically be deleted without causing me problems (though if you’re going to delete give me a warning to triple-check), depending on what needs to be stored to meet journal requirements
This incantation lists all the output dirs which don’t contain any .nc files which have been accessed (nb: creation date ignored) in the previous year (caution: it also lists dirs with newly-copied .nc files that nobody’s accessed yet, and dirs with no .nc files, and non-directories, so there are some “false positives”). Directory size and owner are shown, together with the directory modification date.
for d in /g/data/ik11/outputs/*/*; do if [ `find $d -name "*.nc" -atime -365 | wc -l` -eq "0" ]; then du -hs $d; ls -ld $d; echo; fi; done
it yields this (scroll in box to see full list)
277G /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_iaf_4hourly
drwxrws---+ 24 aek156 ik11 4096 May 13 2022 /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_iaf_4hourly
348K /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_21mbath_5mdz
drwxrws---+ 2 aek156 ik11 4096 May 31 09:47 /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_21mbath_5mdz
2.8T /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3_HF
drwxrws---+ 21 amh157 ik11 4096 May 12 2022 /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3_HF
1.3T /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_rerun_from_1980
drwxrws---+ 9 aek156 ik11 4096 Apr 6 2023 /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_rerun_from_1980
170G /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_rerun_from_1983
drwxrws---+ 8 aek156 ik11 4096 Apr 6 2023 /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_rerun_from_1983
85G /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_rerun_from_1986
drwxrws---+ 6 aek156 ik11 4096 Apr 6 2023 /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_rerun_from_1986
find: ‘/g/data/ik11/outputs/access-om2-01/basal_melt_outputs/accessom2-GPC039/restarts/restart060’: Permission denied
find: ‘/g/data/ik11/outputs/access-om2-01/basal_melt_outputs/accessom2-GPC039/restarts/restart050’: Permission denied
find: ‘/g/data/ik11/outputs/access-om2-01/basal_melt_outputs/accessom2-GPC026/output030’: Permission denied
4.0K /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle6_ext
drwxrws---+ 2 aek156 ik11 4096 May 19 2022 /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle6_ext
1.9T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo2_cycle1
drwxrws---+ 33 amh157 ik11 4096 Nov 19 2021 /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo2_cycle1
1.9T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo_cycle1
drwxrws---+ 32 amh157 ik11 4096 Nov 14 2021 /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo_cycle1
58G /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi_noGM
drwxr-sr-x+ 84 rmh561 ik11 4096 Aug 14 2020 /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi_noGM
32G /g/data/ik11/outputs/access-om2/1deg_iamip2_con
drwxrws---+ 162 hh0162 ik11 12288 Dec 14 2021 /g/data/ik11/outputs/access-om2/1deg_iamip2_con
145G /g/data/ik11/outputs/access-om2/1deg_iamip2_EC-Earth3ssp126
drwxrws---+ 100 hh0162 ik11 4096 Dec 14 2021 /g/data/ik11/outputs/access-om2/1deg_iamip2_EC-Earth3ssp126
144G /g/data/ik11/outputs/access-om2/1deg_iamip2_EC-Earth3ssp585
drwxrws---+ 99 hh0162 ik11 4096 Dec 14 2021 /g/data/ik11/outputs/access-om2/1deg_iamip2_EC-Earth3ssp585
69G /g/data/ik11/outputs/access-om2/1deg_iamip2_exc
drwxrws---+ 74 hh0162 ik11 4096 Dec 14 2021 /g/data/ik11/outputs/access-om2/1deg_iamip2_exc
23G /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle6_bgcinit
drwxrws---+ 13 hh0162 ik11 4096 May 8 2023 /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle6_bgcinit
5.0G /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2dEdd_cycle1
drwxrws---+ 6 hh0162 ik11 4096 Apr 22 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2dEdd_cycle1
14G /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2fe03_cycle1
drwxrws---+ 5 hh0162 ik11 4096 Apr 22 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2fe03_cycle1
6.0G /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2femip_cycle1
drwxrws---+ 6 hh0162 ik11 4096 Apr 22 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2femip_cycle1
140G /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_v2.0.0rc3
drwxrws---+ 14 aek156 ik11 4096 Jul 7 2020 /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_v2.0.0rc3
140G /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_v2.0.0rc3_nonuniform_albedo
drwxrws---+ 13 aek156 ik11 4096 Jul 7 2020 /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_v2.0.0rc3_nonuniform_albedo
23G /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_con
drwxrws---+ 16 hh0162 ik11 4096 Apr 22 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_con
1.4G /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_conflux
drwxrws---+ 4 hh0162 ik11 4096 May 27 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_conflux
1.4G /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_control
drwxrws---+ 6 hh0162 ik11 4096 May 27 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_control
1.4G /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_dedd
drwxrws---+ 4 hh0162 ik11 4096 May 27 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_dedd
1.4G /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_ktherm
drwxrws---+ 6 hh0162 ik11 4096 Jun 7 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_ktherm
1.5G /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_mp
drwxrws---+ 4 hh0162 ik11 4096 May 27 2021 /g/data/ik11/outputs/access-om2/1deg_jra55_ryf_iamip2_sens_mp
5.8G /g/data/ik11/outputs/access-om2/1deg_jra55v131_ryf_const_albedo
drwxrws---+ 8 aek156 ik11 4096 Jul 7 2020 /g/data/ik11/outputs/access-om2/1deg_jra55v131_ryf_const_albedo
11M /g/data/ik11/outputs/mom6-om4-025/mom6.db
-rw-r-----+ 1 amh157 ik11 10932224 Nov 19 2021 /g/data/ik11/outputs/mom6-om4-025/mom6.db
479G /g/data/ik11/outputs/mom6-panan/panant-01-hycom1
drwxr-s---+ 50 amh157 ik11 4096 Oct 22 2021 /g/data/ik11/outputs/mom6-panan/panant-01-hycom1
7.7M /g/data/ik11/outputs/mom6-panan/panant.db
-rw-r-----+ 1 amh157 ik11 7966720 Dec 17 2021 /g/data/ik11/outputs/mom6-panan/panant.db
@AndyHoggANU are these still needed?
2.8T /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3_HF
drwxrws---+ 21 amh157 ik11 4096 May 12 2022 /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3_HF
1.9T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo2_cycle1
drwxrws---+ 33 amh157 ik11 4096 Nov 19 2021 /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo2_cycle1
1.9T /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo_cycle1
drwxrws---+ 32 amh157 ik11 4096 Nov 14 2021 /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo_cycle1
The 0.25° runs were for Improve topography at 1deg and 025deg? · Issue #158 · COSIMA/access-om2 · GitHub
This one is mine. after transferring its data to oz91 I left some symlinks to maintain the cc.dbase working. Im deleting the symlinks and folder now
No need to delete anything that’s small (eg symlinks). I’d count this as a “false positive” from my command.
I didn’t mean to imply that everything listed should be deleted, only that they are things to consider (if big).
Following up @rmholmes requests, I’m actively analysing all the below:
025deg_jra55_iaf_era5comparison
025deg_era5_iaf
025deg_jra55_iaf_omip2_cycle6
025deg_jra55_iaf_omip2_cycle6_jra55v150_extension
OK, I’ve started to have a look through these. I think that:
025deg_jra55_iaf_omip2_straits* → these were runs we did when running the 0.25° coupled model, to help tune the bathymetry. I don’t need them any more and I propose deleting. Let me know if you disagree.
025deg_jra55_ryf9091_gadi_GMhi → This run was for comparison with the coupled model simulations. I’m concerned that @wghuneke might be using these for comparison, so propose not deleting for now until we can determine that.
01deg_jra55v140_iaf_cycle3_HF → This was a short run with high-frequency output for movie-making. I’m happy to delete that.
Sorry for the late reply! I have used panant-01-zstar-v13 for the vorticity budget paper. If its not possible to keep it I can try to find alternative storage (massdata in e14 maybe?)