Poster: Proposed architecture for weather, climate and CMIP data provenance

ACCESS Workshop 2024 Workshop Posters
, , ,
1

Title

A proposed architecture for weather, climate and CMIP data provenance tracking

About

Model input provenance should be available to anyone who uses model output data, since model input determines model output. Unified Model ancillary files contain model input data such as static initial conditions or periodic forcings. Their provenance includes a data source, an ancillary source and an ancillary file on a specific grid. Unified Model ancillary files are currently not FAIR: Their provenance is in generation tool documents, project Wiki pages, tickets and suites.

Provena (provena.io) is a provenance system supporting large modelling and simulation workflows. It resulted from a CSIRO project led by Jonathan Yu to support the Reef Restoration and Adaptation Program.

To add provenance capture to UM ancillary generation:

On the input data side:

  • Add Provena API calls to UM ancillary preprocessing and generation scripts.
  • Initially use Provena on AWS.

On the provenance side:

  • Adapt Provena data models to CF conventions.
  • Migrate Provena servers and stores from AWS to NCI.

Poster

ACCESS-NRI_provenance_architecture.pdf (1.1 MB)

ACCESS-NRI_provenance_architecture
ACCESS-NRI_provenance_architecture795×1124 163 KB

Note: this topic is part of the 2024 ACCESS Community Workshop Poster session

1 Like
2

Hi Paul,

Would the intent be to integrate Proverna into ANTS?

How would provenance information be attached to ancillary files, would it be something like a text file kept beside the ancil, or would you need to look up the file in a database?

3

If you ping @paulleopardi he will get a notification and you might get a response.

4

The intent would be to set up a Python library to record provenance, using the Provena API. The provenance info would be available via a web interface and API, hopefully integrated with the Intake catalogue. Details and feasibility are yet to be investigated.