Restarting CSIRO CMIP6 historical runs

Technical
, , ,
1

Hello everyone!

I’ve been trying to restart a historical run with no success. I want to restart it from year 1960 and apply some perturbations.

What I’ve done so far:

  1. git cloned the esm-historical configuration directory

  2. changed start year at config.yaml to 1960

  3. changed ‘inidate’ and ‘init_date’ to 19600101 in ice/input.nml

  4. changed ‘model_basis_time’ to 1960

  5. changed the following in warm_start.sh: project=p66; user=cm2704; expname=HI-14; source_year=1960 and run the warm start scripts

  6. when running the warm start script I’ve got the following warnings:
    /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.04/lib/python3.9/site-packages/mule/stashmaster.py:259: UserWarning:
    Unable to load STASHmaster from version string, path does not exist
    Path: $UMDIR/vn7.3/ctldata/STASHmaster/STASHmaster_A
    Please check that the value of mule.stashmaster.STASHMASTER_PATH_PATTERN is correct for your site/configuration
    warnings.warn(msg)
    /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.04/lib/python3.9/site-packages/mule/validators.py:198: UserWarning:
    File: work/atmosphere/restart_dump.astart
    Field validation failures:
    Fields (1114,1115,1116)
    Field grid longitudes inconsistent
    File grid : 0.0 to 358.125, spacing 1.875
    Field grid: 0.5 to 359.5, spacing 1.0
    Extents should be within 1 field grid-spacing
    Field validation failures:
    Fields (4935,4937,6676,6715)
    Skipping Field validation due to irregular lbcode:
    Field lbcode: 31320
    warnings.warn(msg)

  7. If I try to put the model to run it crashes at the very begging with payu error code 134. This seems to be related to forcing/restart files and the same warning as above appears in the payu error log file.

Any ideas on how to make this restart work are welcome.

Thanks,
Gabriel

1 Like
2

Hi Gabriel,

I will look into this issue.
I will try to get back to you within the end of the week :slight_smile:

1 Like
5

Hi @gpontes,

I am sorry but I totally forgot to come back to this issue after our chat at the ACCESS Workshop.

Did you manage to solve the issue or is it still happening?

Cheers
Davide

6

Hello @atteggiani,

No worries. Thanks getting back this.

I’m still not able to restart any historical CSIRO run. For now, I’ve started my own historical simulation from scratch (previous PI control). However, I will need to run an ensemble; so, figuring out how to restart historical CSIRO run would save lots of computational time.

In time, I’ll also need to run a CMIP6 SSP126 scenario with some perturbations applied. I could find a payu configuration for the SSP585 scenario (GitHub - coecms/esm-ssp585), but not SSP126. Is the SSP126 setup somewhere else? If not, is it possible to configure this run for payu?

Thanks,
Gabriel

7

Hi @gpontes. I am not familiar (yet) with the payu framework and I am still running ACCESS-ESM1.5 in a script based environment. However there are a number of people who have done restarts from different runs and points in time. Maybe @tammasloughran or @ariaan have any suggestions?

8

Hi @gpontes. With regards to setting up an SSP126 run, this should be relatively straight forward. We ran SSP126 with ACCESS-ESM1.5 for CMIP6 (but not with payu) and all the forcing files are available. @holger has been helping in the past with converting the script based setup with ACCESS-ESM1.5 into payu. However, since we already have an SSP585 payu setup it should be relatively easy to copy this and replace the forcings etc. with SSP126. I can help with providing the relevant forcings, but again I am not familiar with payu.

Cheers,
Tilo

9

Hi @gpontes,

sorry for not getting back to your request.

Related to the error you were getting:
I have poor knowledge about the payu configurations as well, but I get the same error code (134) you are getting, even when I try to run a simulation freshly cloned from the repo (after setting source=PAYU in the warm-start.sh file).
So I believe the error does not relate to the change in the restart date. I am going to write an issue on the Github repo and email @holger about it (unless he replies to this post).

For the setup of SSP126 simulation, you can follow what @tiloz suggested (thank you @tiloz for your reply!)

Cheers
Davide

10

Hello @tiloz,

Many thanks. I’'l try to simply replace the SSP585 forcing files with the ones from SSP126.
I get back to you in case I have any issues or am unsure about the forcing files.

Cheers,
Gabriel

11

Hi @atteggiani,

Thanks for going after the restart issue.
Let me know whenever you’ve got any updates on that.

Cheers,
Gabriel

12

Hi @gpontes,

Unfortunately, I have not been able to get any response on my help request for ACCESS-ESM1.5 historical repo error.

Just to make sure we’re talking about the same repo though, can you please point out which repo you were trying to use that produced the error code 134?

Thank you

14

Hi @gpontes,

Just checking in whether you still require help with this topic, otherwise we are going to close it.

Thank you.

Cheers
Davide

15

Hi @atteggiani,

Sorry for the late reply on this topic.

I still can’t restart these runs. It would be good if we could figure out a way of doing this in case I need some more ensemble members of my runs.

Thanks,
Gabriel

16

Hi @gpontes,

Are you already using the new ACCESS-ESM1.5 Github Repo?

17

Hi @atteggiani,

Yes, I’m using this same repo.

1 Like
18

Hi @gpontes ,

I tried replicating your error, and what you described happens to me as well.
I am going to write an email to @holger to hopefully solve this issue.

Just a question on the error you are getting:
When you run the warm-start.sh file, after the warnings you mentioned, do you also get the following output?

cdo    selyear (Warning): Year 101 not found!

cdo    selyear (Abort): No timesteps selected!
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/atm/HI-14.astart-19600101' -> './archive/restart000/atmosphere/restart_dump.astart'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/cpl/a2i.nc-19591231' -> './archive/restart000/coupler/a2i.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/cpl/i2a.nc-19591231' -> './archive/restart000/coupler/i2a.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/cpl/o2i.nc-19591231' -> './archive/restart000/coupler/o2i.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/csiro_bgc.res.nc-19591231' -> './archive/restart000/ocean/csiro_bgc.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/csiro_bgc_sediment.res.nc-19591231' -> './archive/restart000/ocean/csiro_bgc_sediment.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_age.res.nc-19591231' -> './archive/restart000/ocean/ocean_age.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_barotropic.res.nc-19591231' -> './archive/restart000/ocean/ocean_barotropic.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_bih_friction.res.nc-19591231' -> './archive/restart000/ocean/ocean_bih_friction.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_density.res.nc-19591231' -> './archive/restart000/ocean/ocean_density.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_frazil.res.nc-19591231' -> './archive/restart000/ocean/ocean_frazil.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_lap_friction.res.nc-19591231' -> './archive/restart000/ocean/ocean_lap_friction.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_neutral.res.nc-19591231' -> './archive/restart000/ocean/ocean_neutral.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_pot_temp.res.nc-19591231' -> './archive/restart000/ocean/ocean_pot_temp.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_sbc.res.nc-19591231' -> './archive/restart000/ocean/ocean_sbc.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_sigma_transport.res.nc-19591231' -> './archive/restart000/ocean/ocean_sigma_transport.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_solo.res-19591231' -> './archive/restart000/ocean/ocean_solo.res'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_temp_salt.res.nc-19591231' -> './archive/restart000/ocean/ocean_temp_salt.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_thickness.res.nc-19591231' -> './archive/restart000/ocean/ocean_thickness.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_tracer.res-19591231' -> './archive/restart000/ocean/ocean_tracer.res'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_velocity_advection.res.nc-19591231' -> './archive/restart000/ocean/ocean_velocity_advection.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ocn/ocean_velocity.res.nc-19591231' -> './archive/restart000/ocean/ocean_velocity.res.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ice/ice.restart_file-19591231' -> './archive/restart000/ice/ice.restart_file'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ice/mice.nc-19591231' -> './archive/restart000/ice/mice.nc'
'/g/data/p66/cm2704/archive/ACCESS-ESM1-5/HI-14/restart/ice/iced.19600101' -> './archive/restart000/ice/iced.19600101'
Setting restart year in ./archive/restart000 to 1960
ice init_date 1960-01-01 00:00:00, runtime0 0:00:00
Finished
19

Hi @atteggiani,

Yes, I do get that too.

Cheers,
Gabriel

1 Like
20

Hi
I was gone for a long time, now I’m back.

Do you still have an issue?

Cheers
Holger

1 Like
21

Hello Holger,

No worries.

I haven’t tried restarting these runs for a while. But since there was no update on this, I believe the same issues reported above still persists.

Cheers,
Gabriel

22

Hi Gabriel

I’m trying to reproduce the error, but it seems as if there is no experiment HI-14 on /g/data/p66/cm2704/archive

Can you confirm which experiment’s restart file you want to use?

Cheers
Holger