Axiom - High performance metadata utility

Technical
, , ,
1

Hi All, just letting you know about a little utility we’re developing for ACS to process model data into standards-compliant datasets.

https://axiom.readthedocs.io/en/latest/

Contributions welcome.

Cheers,

Ben

2 Likes
2

Hi @ben, Welcome to ACCESS-Hive!

Thanks, that looks interesting. Could you provide a bit more background about this tool? What problem does it solve? From my limited experience it looks like a much more general CF checker tool. Is that a fair characterisation?

Is this currently being used for validating ACS data sets?

If axiom tells you your data is non-compliant, are there also tools to rectify that? I hacked together addmeta for this purpose, but perhaps there is now something better?

Do you have any specific contributions you would like? Do you need users/testers in order to get more feedback about usability, missing features and/or bugs? If so then you will want to make this widely available to this community I guess? If it was available on conda-forge then it would be simple to request it be installed in the widely used CLEX CMS conda environment.

Ping @rbeucher ICYMI

3

Hi @Aidan, thanks.

There is an element of what utilities like CF Checker and CMOR to what Axiom can do. It was originally developed to validate and apply metadata standards for legacy data but soon grew to something that does parallelised DRS processing for the most part. I see your add meta utility does a very similar thing with a YAML configuration file - I’ve gone with a similar JSON option which includes not only the required metadata, but the valid entries for what should be contained in those keys. The whole thing is driven by configuration which makes the system very generalisable to different requirements and hardware.

Axiom is available to the ACS community for validating data sets, we are currently using it to produce our standards-compliant data for CCAM ourselves.

As far as contributions go - any and all are welcome. Bug reports, fixes and enhancements, documentation. I am in the process of setting it up as an installable package, starting with pip then moving to conda-forge.

Cheers,

Ben

1 Like
4

Thanks for the extra information @ben.

Is there a public repo where the scripts and metadata configuration you use for producing your standards compliant data for CCAM is available to use as an example?

Does axiom require or impost a DRS-like structure? I ask because some of the high resolution ocean model data is very large, so there is a requirement to do as little transformation as possible, so the data is generally kept as it was output from the model. However it would be useful to use a tool like axiom to check and clean up metadata, but it would have to be done in-place, only altering the metadata. Is that possible with Axiom?