How do I evaluate new developments in CABLE?

As a developer for CABLE, I want to know how my code changes affect the simulation outputs. That is I want to ensure there is no impact on unrelated parts of the code and diagnostic variables and evaluate the effect on the related diagnostic variables.

  1. How do I know what configurations to run for testing?
  2. How do I set up the codes and file structures to run all configurations correctly?
  3. How do I find what analysis to run?
  4. How do I find evaluation results from preview developments?
  5. How do I compare my results to other developments?
  6. How do I share and archive my analysis results?

Current Workflow (as of 17/01/2023):

  1. Each user figures it out on their own or within their team.
  2. Each user is responsible for their own setup.
  3. Users might be able to use modelevaluation.org or ILAMB if it fits their needs. Most of the time, users write their own analysis script or need to know someone who can share a script.
  4. Ask the developer if they are known. In most cases, the analysis will not be available and will need to be recreated from scratch.
  5. Each user has to figure that one out. Usually, the simulations and analyses of the various developments don’t match and need to be redone by the user before any comparison is possible.
  6. Each user shares their results as they see fit. They are responsible for archiving their results as well. That means in most cases, the results are shared only between a few people and they are not archived.
1 Like

ACCESS-NRI would like to use user-stories as qualitative measures of impact and improvement for the community. In many cases what we might do can’t be well measured in metrics, but the community will “just know” it is a lot easier than it used to be. User stories are a way to capture this improvement, by documenting the improvement in the workflow for a particular user story.

Since working on improving model evaluation and testing for CABLE has emerged as a huge priority I have started a user story to document where we started from and how we improve.

The first post is a wiki post which means it is editable by everyone so everyone can help document the improvements.

@gab @inh599 I’m tagging you in if you have any comments/edits on what is there.

1 Like