Transport matrix diagnostic: using payu to run many related jobs

Technical
1

I am much like Callum in that I have lots of experiment names that I’ve chosen and can make sense of. For my purposes, if the archive folder name matches the control name it all makes sense. If instead I have to append the archive name with a random string (that I can’t control or set in advance) it causes inconvenience.

On one application I am running a transport matrix diagnostic which requires copying an experiment and running it 42 times. Each experiment requires a “tiled” input tracer, which I can easily make with an ordered set of tiles from 01..42. My control dirs follow this pattern, i.e.

Expt_01, Expt_02, etc,

and it all makes sense if my archives follow that pattern too.

Also I am a dinosaur and quite like the 2015 version of payu which I used in Sweden for a long time. :smiley:

2 Likes
Payu functionality, naming of output folders
2

Hi @dkhutch for your use case you might find our new tooling helpful

They’re designed to create and run large numbers of experiments automatically.

Could you share a bit more detail about this specific workflow? Eg, does this involve changes to the codebase itself, or are you modifying the tiled tracer input 42 times and need to generate 42 separate experiments?

1 Like
3

@dkhutch if you’re interested in trying out the new tools, please open a new topic so we can continue the discussion there and keep this thread tidy!

4

Hi Minghang,
Thanks for your tips on the experiment runner. Yes the model stays the same (and uses the same restart folder) for all 42 experiments. If you’re curious, I put one of the tiles into a github repo:

The tracer that changes between each experiment is read in through the field_table:

And this is also included each time in the config.yaml:

The tracer_set_01.nc increments from 01 to 42. Anyhow, don’t worry too much about my specific case - this is just if you wanted to see what I’m talking about.

5

Great thanks for sharing! This is definitely something the new tooling can support. I’ve seen quite a few use cases where people need to modify the field_table or data_table, but those aren’t supported yet. I’ll add support for them soon and ping you for a look/review. I’ll also add examples there too.

1 Like
6

My specific hack looks something like this

import glob
filename_i_expect = 'Expt_01'
actual_filename = glob.glob('/path/to/'+filename_i_expect+'*')[0]

(sorry to upset the NRI folk, but if people are being stupid you may as well know about it?)

2 Likes