Restarting ACCESS-AM3 mid run

Hi! I have an experiment that ran from Jan 2007 to Dec 2008, the experiment has already finished. For a reason, I have to re-run Aug 2007. What would be the correct way to restart the model to run only that month?

I would usually change the initial conditions to use the corresponding dump file and initial date and start a new run. But if I do that, the suite will run the reconfiguration step before running the model. I’d like to avoid it since it is not what happens in a “normal” run.

I already tried starting the suite and skipping the recon step, but the model complains that there is no astart file. When I copy the initial conditions with that name, the model complains that there is no xhist :frowning:

I’ve been exploring the suite and the documentation and I found an option BITCOMP_NRUN that activates l_nrun_as_crun=.true.. It sounded like it was the solution, according to the documentation:

can be set to treat a new run in the same way as a continuation run. This effectively bypasses special behaviour on timestep one.

However, the model still complains that it needs the am3.xhist file. I can potentially copy one of those from a previous run, but the content won’t match anyway.

Ahh good question- I think what you should be able to do is play the workflow (cylc play) and then trigger the specific task you want to re-run, before Cylc times out and shuts the workflow down again (which I think is 5mins)? I think the date should be the cycle point, so cylc trigger access-am3-configs/<workflow-name>//<date>/atmos?

Thanks for that one!

It didn’t work immediately for me, it returned Command queued but then nothing else (no job sent to the queue). I tried a few things and this worked:

cylc set --pre=all access-am3-configs_21st/run16//20070801T0000Z/atmos_main 

It seems you need to set all the dependencies to success, even when the workflow has already finished successfully. So, now I got atmos_main in the queue :tada:

Note for people in the future. Those double // are not a typo, trust me :sweat_smile:

But, it didn’t run the model. I got a bunch of warnings like this:

IO: Open: /home/565/pc2687/cylc-run/access-am3-configs_21st/run16/share/data/History_Data/am3a.pg20081231 on unit  19 (Server =   23)
loadHeader: The file on unit  19 does not have enough content to load the header

????????????????????????????????????????????????????????????????????????????????
??????????????????????????????      WARNING       ??????????????????????????????
?  Warning code: -10
?  Warning from routine: INIT_PP_CRUN
?  Warning message: Error: Failed to load fixed header.
?  Warning from processor: 0
?  Warning number: 66
????????????????????????????????????????????????????????????????????????????????

I think it is still reading the last am3.xhist with the last cycle information. I’m tempted to manually edit the file and see what happens.

It worked!!

So, I changed the am3.xhist file to match what I imagine it would need to run Aug 2007 (dump and output stream file names) and the model ran without any complaints, triggering with:

cylc set --pre=all access-am3-configs_21st/run16//20070801T0000Z/atmos_main

I also checked a few variables to see if they match and yes, diff = 0 :tada:

I have been reading along wondering if there would be a (relatively) simple fix to be discovered. This is great news, Pao!

Nice, good to hear you found a solution, it has always bothered me that I often couldn’t get cylc trigger to work the way I wanted it to.

@paocorrales did you need to cylc play the workflow before calling cylc set --pre=all ...?

Yes, the suite has stopped so cylc play and then cylc set --pre=all ....

Well, I had to trigger the netCDF conversion also and it worked with cylc trigger. There is something in the logic of the suite/cylc that I still don’t understand.

There’s some docs on retrying sections of a workflow at Interventions — Cylc 8.6.6 documentation