No worries, I had already included the fs38 storage, specifically because I want to use ACCESS data as well. On that note, is there a recommended way to merge the oi10 and fs38 NCI-maintained catalogs? (Your CMS catalog contains both the oi10 and fa38 data as far as I understand, right?)
I’ve got a rather specific question about CMIP6 on gadi. I generally use the CMSclef tool to interrogate the CMIP data holdings, and also to request new downloads, but I get the impression that this will be deprecated and we’re moving to intake - is this correct?
One of the really useful aspects of clef is that it’s also able to interrogate the ESGF archive, which is really helpful to find out what’s missing from gadi and also to then request new data. I’m assuming that’s not possible to replicate in with intake?
Maybe @Paola-CMS or @dougiesquire can comment what the plan is for CMIP7/post-CLEX?
1 Like
Hi Will,
The short answer is there’s no plan post-CLEX to maintain clef and the intake catalogues as they are don’t make comparison with the remote colelction.
The long answer is, it might be possible to adapt clef to use intake catalogues.
However, we use file checksums for the remote query in clef.
This is because especially for CMIP5 data this was the only reliable way to make sure what was available locally was identical to the remote version.
Intake catalogues don’t record checksums, it could be possible but it would require NCI to have a special workflow for the CMIP data.
It might be that this is less of an issue with CMIP7 if they are using tracking-ids more consistently and if these tracking-ids are available in the intake catalogues.
If that’s the case, it would be relatively easy for someone to update clef to use them instead of the postgres database.
I was considering having a go at this update but I’m running out of time. I might still have a go if there’s enough interest.
2 Likes
Hi Paola, thanks for the clarification - I was a bit afraid of that
Hi all, I was wondering if there are replicas for CMIP6 daily output of ocean variables (such as tos) in Gadi? I could only find daily outputs for atmospheric variables in oi10.
Any kind of help would be appreciated. Thanks!
Apologies for raising this zombie @Aidan - but 2 years down the road, are you aware of anyone making open progress in this regard - open, xarray-based workflows, xmip, cf_xarray, datatree?
Hi @sb4233, have you tried looking at the ACCESS-NRI Intake catalog and searching for ocean variables on a daily frequency?
Assuming you are in a python workflow, these docs should get you there (?) - ACCESS-NRI Intake catalog — ACCESS-NRI Intake catalog documentation
2 Likes
Sorry no, but it’s not my bailiwick these days so I’m less likely to stumble across this stuff.
@rbeucher or @CharlesTurner may know more.
1 Like
Hi @Thomas-Moore ,
We are working on recipes using ESMValcore in Jupyter notebooks.
Plan is to release evaluation recipes for ENSO, Atmosphere and Ocean in the next couple of months.
Priority is on using ESMValcore and Iris but we will eventually explore other workflows.
Romain
1 Like
Thank you! @Thomas-Moore, this is very helpful!
1 Like
Nice!
And hope you get some waves, @sb4233
1 Like
Thanks @rbeucher - assume ESMValcore does the distributed computing ( Dask’s role ) and Iris is your xarray alternative? For bringing non-CF compliant data ( observations to validate models, etc ) into Iris do you have thoughts on a workflow? Will CF standardising non-model data be part of the workflow? Thanks for taking the time to reply here.
also - don’t thank me, thank @dougiesquire et al at ACCESS-NRI for building the catalog and great docs.
1 Like
Yes, that’s correct.
ESMValCore provides robust support for CMIP data, offering a comprehensive collection of preprocessors for tasks such as regridding, spatial and temporal statistics, and automated weighting.
While it’s possible to ingest non-CF-compliant data, doing so requires the development of drivers to perform minimal CMORisation, often referred to as CMOR-lite. We’ve recently released a version of ESMValCore that supports a subset of atmospheric variables, and Rhaegar Zeng is currently working on ocean variables.
I’m developing ocean recipes using ESMValCore and have had good success with the tripolar grid so far. From a Python workflow perspective, ESMValCore/Iris is somewhat less flexible than Xarray, but converting workflows to ESMValTool recipes is relatively straightforward and enables ensemble and multi-model analysis.
I plan to release new training materials in the coming weeks.
2 Likes
Thank you @rbeucher for this big-picture overview and your efforts.
1 Like