Forcing data sets for the the CMIP7 DECK are becoming available and it would be useful to start testing them with ACCESS-ESM1.5. We would require help to convert the format published for input4MIPs to the required format in ESM1.5.
Status and links to the forcing data sets can be found here:
Cheers,
Tilo
Broadly speaking, what transformation is required? Conversion to grib?
Hi Aidan,
the format depends on the data. The GHG concentrations for example are provided in text format, but other data such as aerosols and ozone fields need to be converted to the UM ancil format. This also requires re-gridding. Martin (@MartinDix ) has done most of this for CMIP6 for both model versions. But we also benefited a lot from work that has been done by the UKMO and we didn’t have to start from the input4MIPs data set in many cases.
Cheers,
Tilo
And also unit checking!
Yes, and probably a few other things as well.
Do we have an update on preparing the forcing data for CMIP7 so that we can start testing with ACCESS-ESM1.5/ESM1.6? I am attending a climate forcing dataset workshop this week and have been asked to report on status of data implementation in CSIRO models.
Thanks.
Tilo
As a note, you can follow along with the status of the various datasets at: input4MIPs CMIP6Plus delivery summary
Each dataset then has its own more detailed page (index here: Summary - input4MIPs Controlled Vocabularies (CVs)). For example there is:
If you have any problems, please don’t hesitate to raise an issue/discussion in the underlying repository:
A number of groups have already raised issues and discussions, so it’s worth having a look at what is there and can help. (For example, quite some discussion on volcanic forcing: UOEXETER-CMIP-1-1-3: Volcanic optical properties and SO2 emissions · PCMDI/input4MIPs_CVs · Discussion #142 · GitHub). I know the UKESM team is getting spun up too, so they may able to also help.
Here is an update on preparation of CMIP7 forcing data, from my perspective.
Questions by Paul Leopardi. Answers by Jane Mulcahey as of 2024-12-20.
As far as I understand, ancillary processing generally follows a two-step process: there is always a preprocessing step to produce an ancillary master file from an original input file, followed by an ancillary generation step that produces one or more ancillary files for a specific grid. How does the UK Met Office plan to track the provenance of the ancillary master files produced from CMIP7 forcings? Will there be a single persistent source of truth that tracks each master file back to its forcings?
For the CMIP forcings we don’t produce this intermediate step – our scripts should go straight from source data to ancillary at requested resolution. This is the same as what we did for CMIP6. You are right that the ancillary generation of many of the standard ancils required to run GC systems have the above step but we don’t do this in the CMIP project.
Is there any generic documentation on how to run the ancil/contrib applications, or does each application just have its own documentation somewhere in the ANCIL project Wiki? Is there an index of ancil/contrib application documentation somewhere, or do I just need to search the ANCIL project for keywords?
Our scripts will make use of the ants library documented here: https://code.metoffice.gov.uk/doc/ancil/ants/latest/index.html . Our ambition then is to set up a rose suite to run our CMIP scripts using above plus other required libraries. The rose suite is being designed to run on both JASMIN as well as our internal machines. Note all our code is still under development and is not yet on contrib but the ambition is that most scripts for the different forcing datasets will get lodged over the next 6 months or so. If there are specific forcings you are interested in it would be good to know this. With time we will be moving to github, which I think will make a lot of this collaboration easier but I don’t think this will happen before we need to do the processing for Fast Track.
Can I use the ancil/contrib rose-stem test https://code.metoffice.gov.uk/trac/ancil/browser/contrib/trunk/rose-stem/suite.rc as a set of examples on how to run each ancil/contrib application?
I will have to refer you to the MIAO and Partnerships team to provide best advice on how to run these applications externally. My understanding from Julia is that NCI has a version of ants installed but perhaps not the latest version? @Julia Cantarano can you follow up on what ants version in installed at NCI ?
The ancil/contrib rose-stem tests have a site directory, but no nci-gadi site. This indicates to me that the tests have not yet been run at NCI. Would porting the rose-stem test to NCI help ACCESS-NRI to understand how to run and contribute to the ancil/contrib applications?
See my response above
The documentation for UKCA ancillaries refers to Python 2.7 and looks to be out of date: https://code.metoffice.gov.uk/trac/ancil/wiki/ukca/NetcdfCode . The code itself looks to be unmaintained: https://code.metoffice.gov.uk/trac/ancil/browser/contrib/trunk/Unmaintained/AerosolChemistryEmissions/ukca_netcdf
Are these ancillaries relevant to CMIP6 or CMIP7 or are they only used for other purposes?
Is this the correct Wiki page for UKCA ancillaries?
What is the replacement for the unmaintained code?
This code is under development and will be updated (python3 based) for CMIP7 processing – as I said above all of this is still under active development and so is not yet lodged. My understanding is the old code does not work with the latest version of ants – so you are right to assume this is out of date. I wouldn’t advise picking up this code.
The documentation for CMIP6 greenhouse gas ancillary generation is at https://code.metoffice.gov.uk/trac/UKESM/wiki/CmipGhg with https://code.metoffice.gov.uk/trac/ancil/wiki/CMIP6/ForcingData/GreenhouseGases describing the forcing data. The corresponding Trac ticket for CMIP7 is https://code.metoffice.gov.uk/trac/ancil/ticket/2416 , which includes updating the Python 2 ancil/contrib code to Python 3.
Is this the right codebase to start with when porting the CMIP7 ancillary generation code to NCI?
What is the relationship between the CMIP6 greenhouse gas ancillary files and the UKCA ancillaries as described in1.e. above? Which fields do each of these sets of ancillary files contain?
This code is under active development. You are looking at the right ticket but this code has not been science tested or validated yet so please use with caution. Once lodged it will go into https://code.metoffice.gov.uk/trac/ancil/browser/contrib/trunk/Projects/CMIP/CMIP7
An example of a CMIP7 global ancillary generation suite is u-dh360: https://code.metoffice.gov.uk/trac/roses-u/browser/d/h/3/6/0/trunk . It uses Cylc 8 and has a site directory containing jasmin, meto_86 and spice.
This suite is under active development and is not yet ready for use.
Will the CMIP7 global ancillary generation suites all be based on Cylc 8?
yes
Would it be a good idea for ACCESS-NRI to start development and testing on Jasmin while also porting the ancil Python code, rose-stem tests and ancillary generation suites to NCI?
At this point I would say No, as everything is still under development and has not been sufficiently tested. Once ready for testing we can absolutely let you know, the same goes for the code. I wouldn’t advise porting old (CMIP6-based) code or code under active development. I will chat with Stephen Pring, the developer of the suite to make him aware of your interest and put you in touch once he’s got something at a more advanced state of readiness.
Is it true that the CMIP7 global ancillary generation suites carry out only the second stage of ancillary generation as per 1.a. above, and assume that the preprocessing has already occurred, with files stored in known directories, such as those defined by the variables in files stored in known directories, such as those defined by the variables in https://code.metoffice.gov.uk/trac/roses-u/browser/d/h/3/6/0/trunk/site/meto_x86/variables.rc ?
No, see my reply to your question 1 above. I would also take the directory locations here with a pinch of salt as this suite is currently under development. We are also in the process of setting up and defining the directory structure for the CMIP7 forcing data. I hope to have this in place for JASMIN as well as on our internal systems later in Jan 2025.
Do all of the directories defined by the variables in https://code.metoffice.gov.uk/trac/roses-u/browser/d/h/3/6/0/trunk/site/meto_x86/variables.rc have corresponding directories on Jasmin that would allow mirroring to NCI?
See my response above. They will but its not currently in place.
I have added a suggested development strategies to the CMIP7-Input README. For your convenience, here is a copy. Please comment on the strategies by replying to this post.
NOTE: The following development strategies are suggestions, subject to discussion and confirmation
As much code as possible would be developed via this GitHub repository. The main exceptions would be the code and suites currently developed via MOSRS repositories such as ANTS, the ANCIL Contrib project, Rose/Cylc suites, and Rose Stem tests. These would initially be developed via contributing to the relevant MOSRS repository according to current ANTS working practices and would be migrated to GitHub when appropriate. The GitHub Continuous Integration infrastructure for such MORSR code would be developed in parallel with the code development via MOSRS.
Similarly, as much documentation as possible would be contained in the Wiki for this repository, then published to other locations when appropriate. The main exception would be documentation that describes some of the internal workings of some UK Met Office and Momentum Partnership projects, such as the ANCIL project, the UK CMIP6 project, and the UKESM project. In this case, only a top-level description would be given in the Wiki for this repository, with details in an appropriate MOSRS project Wiki.