"gh repo create" - workflow to enable a repository with many experiments, rather than just one

Technical
,
1

The creation of a repo using gh in the command line is awesome. Love it.

However, I like running my experiments in parallel (at the same time). This means that having many branches within the one run directory isn’t really an option. Instead, I have many run directories.

So, I’d like to be able to create a repo using this command that included many experiments, perhaps sensitivity experiments, contained within their own run directories. In essence, a gh repo create command from a higher level directory that contained all the run directories.

Is there a workflow that we could standardise to make this nice?

2 Likes
2

Would this fit the bill? GitHub - aekiss/ensemble: Generate ensemble of ACCESS-OM2 experiments

It only allows one parameter to be perturbed at a time, but @minghangli has something more general.

3

@minghangli - I think this is similar question to the tool you’ve been working on

4

@pearseb and I were chatting at the training session about this.

I think a good approach is to leverage the payu branching features, have a tool that creates a bunch of branches in a single repo, checking out branch one at a time.

Then you could have a generic run tool that you point at a repo and it loops over all branches, creates a separate sub-dir for each branch and spins off a simulation in each separate sub-dir.

Separating the functions has the advantage that the run tool is agnostic to how the repo and branches are created.

5

Forgot to say, leveraging Jinja2 would make creating a bunch of related configs programmatically exceptionally easy

this is a “dumb” approach, as it doesn’t need to know anything specific about the structure of the files themselves.

The tool @minghangli is working on is more structured, but also specialised to a specific model

6

ping @claireyung because I know you were involved in the conversation with @pearseb at the payu training session.

1 Like