Experiment title:
Testing the effect of Indo-Pacific SST variability on mid-latitude droughts
Summary:
Using ACCESS-ESM1.5 to test the effects of increasing or decreasing SST variability in the Indian and Pacific oceans affects the occurrence, length, and/or intensity of mid-latitude droughts.
Scientific motivation:
Palaeoclimate evidence suggests that in the first half of the past millennium, decreased ENSO and IOD variability co-occurred with increased megadrought occurrence across the mid-latitudes. This suggests a relationship between Indo-Pacific SST variability and mid-latitude hydroclimate variability.
We are testing this hypothesis by running ACCESS-ESM1.5 in pacemaker format where the magnitude of tropical Indian and Pacific Ocean SST variability is altered using 4 forcing scenarios: 60% reduction, 30% reduction, 30% enhancement, 60% enhancement. We have created SST forcing files by taking the 500 years from the existing 1000-year-long control run, altering the magnitude of SST variability, and using these as SST restoring files (with all other model components allowed to evolve freely).
In the first instance we will alter both Indian and Pacific ocean SST variability. In later experiments we will alter SST variability in the individual ocean basins, to aid in diagnosing the results.
Design and details
Experiment Name: m60/m30/p30/p60 (for Indo-Pacific SST variability plus/minus 60/30%)
People: @georgyfalster@spencerwong + Nerilie Abram
Model: ACCESS-ESM1.5
Configuration: release-preindustrial+concentrations
Link to experiment repo: GitHub - ACCESS-Community-Hub/access-esm1.5-sst-perturbation-indo-pacific: Magnitude of tropical Indian and Pacific ocean SST variability altered relative to ACCESS-ESM1.5 preindustrial control
Initial conditions: restart files for the 4 experiments in /scratch/zr75/gf6872/access-esm/archive/
Run plan: 500 model years for each forcing scenario = 2000 model years total
Simulation details: for each model year, SST variability is prescribed according to the experiment outlined above. All other fields evolve freely.
Total KSUs required: 2000 model years x 1.1 kSU per year = 2200 kSU
Time required: 4 parallel runs of 500 years @ 16 years/day = 32 days
Total storage required: ~3 TB
Storage lifetime: approx. 12 months - for analysis, write-up and review
Long term data plan: a very reduced set of relevant outputs to be permanently archived in a suitable repository
Outputs: ‘spin-up’ preset output profiles as per Preset output profiles for ESM1.5 - #2 by spencerwong
Restarts: Currently saving every 10 years
Related articles: TBA
Analysis:
[in-depth analysis to be provided on completion of simulations]
Example Nino 3.4 SST anomaly output from the first ~300-400 model years of each experiment
Hi Georgy,
Thanks for the proposal. Just wondering, how soon could you potentially start these simulations? We currently have 400 KSU remaining in lg87 for Q1, but there could potentially be some unused SUs in other working groups.
So, if you can start promptly there might be some scope to grab additional unused SUs in Q1.
Regards,
David
Thanks! That would be ace, because to run all the simulations I’d like would take a lot more SU than I currently have.
As per discussions elsewhere: how quickly this happens depends on how quickly I can get set up with the Met Office account. After that everything is mostly in place to start within ~a week (depending on how much time I am inclined to take out of my between-jobs holiday)
I am more or less up and running now - I’ve successfully done a test run with the ESM and now just need to add the tweaks to run it in pacemaker mode. If there are still kSU that need to be used I could probably start getting through some from next week? That is, if someone shows me how to draw from two projects at once
That’s great news! Thanks so much to folks at ACCESS-NRI for making that happen so quickly. @spencerwong@heidi@Aidan@clairecarouge I know it hasn’t always been that simple.
Hmmm… well for now I’d suggest to go ahead and run using project lg87 for your simulations to use what’s there for Q1 (~400 KSU). If there are other unused resources in the other WGs we can discuss if some portion can be redistributed.
Hi lg87 chairs: this experiment is ongoing. Can I please request some Q2 compute for this? I am approximately halfway through the simulations (with around 1000 kSU-worth to go). So: anything would help. If a couple of hundred kSU wouldn’t be too much to ask that would be amazing, but also I of course don’t want to snuffle it all up this early in the quarter.
Hi @georgyfalster,
At the moment, it appears that there are no competing proposals for the lg87 resources in Q2, so you can in principle use these resources.
May I suggest you go ahead and use 500 KSU as soon as possible. Then check in with us again at that point, and we will re-assess if there are any other potential users of lg87 this quarter. (And if not, we might suggest you use more.)
Regards,
Dave (and ESM co-chairs)
hi lg87: I’ve now used my 500 kSU (thanks!). It looks like there haven’t been any other users so far this quarter - but there is still a fair while to go. If there are other potential users I am of course happy to leave the remaining compute for them, or otherwise I’ll happily use more (or wait to do so until a bit later in the quarter).
Hi ACCESS-ESM team - I have a potentially sticky question. How are lg87’s gdata and scratch capacity looking? And if it’s ok, could I ‘borrow’ ~3TB from one or the other (ideally gdata)?! Just because I am starting to run up against my scratch iQuota which is slowing things down a bit.
In terms of timeframe: I’d just need the storage for maybe ~4 months or so while I finish the simulations and confirm which variables we’ll analyse. Then I can just pull those out (won’t be a lot of memory) and archive the full outputs to massdata.
Hi Georgy,
At the moment, I think there is space on gdata or scratch lg87 for 3TB of temporary storage. Since it’s the iQuota that is limiting you, may I ask how many files we are talking (i.e. what is your iQuota?)
There is approx. 6.5M remaining in the gdata iAllocation and 3.6M in the scratch iAllocation.
And yes, please do follow through on archiving to massdata when the jobs are done, because we want to maintain the working space for future use (storage tends to get full and forgotten very easily).
Regards,
David
My iUsage is 295368 and I’ve run 78% of my model years. I could theoretically almost fit the full outputs in my scratch (iLimit 381150) except for the occasional collation error which produces wild amounts of tiny files. I’ve been been fixing these as I go but now that I am approaching the limit, just one model year failing to collate properly pops me over(!) But that shouldn’t be an issue under lg87’s limits, I think.
And yes have no fear I absolutely will transfer the outputs to massdata as expediently as possible.
1 Like
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
14
I guess this is using the io_layout that produces prodigious numbers of files?
@spencerwong would it be feasible to swap to an uncollated io_layout for these configurations?
erm I’m not sure what causes the collation error but it only seems to happen once per ~30 or so years and is easily fixed with payu collate (and all outputs in these experiments are currently properly collated)
My understanding is that there is no io_layout specified at all in ACCESS-ESM1.5, so the uncollated outputs are simply one file for every CPU. When you have the diagnostics split into ~190 single-variable files and 180 CPUs for the ocean, a single uncollated run can generate ~34K files.
and removing the collation step entirely. This has a small impact on walltime, however guarantees that the collation/conversion failures won’t occur.
@georgyfalster, we could try swapping your configurations to use this layout to prevent the conversion failures. The one thing to check is whether you will need any of the 1D ocean variables highlighted here. Unfortunately due to a bug in MOM, these variables don’t work when the io_layout setting is used.
