Ants module availability in xp65 analysis3 kernels

Technical Infrastructure
, ,
21

I’ve not looked at ANTS for a while, it is now public on github with the environment definition and conda lock file that it wants to be using - GitHub - MetOffice/ANTS: A Python library for generating UM ancillary files, developed at the Met Office (the contrib code that actually generates the ancillaries is still private I think?)

There are two use cases to consider:

  • If we’re generating ancillaries for future use we should make use of the lock file to generate a conda environment, this way the ancillaries can be re-created exactly the same in the future.

  • If we’re developing ANTS code then it would make sense to install it into a larger environment with Jupyter etc.

22

I agree with Scott, and I would add that we are planning to release new ANTS versions in a dedicated containerised environment on Gadi (similar to Scott’s ngm one).

However, the main issues remain:

  • ANTS strict dependency pinning (which makes it very hard to install in an environment together with other packages)
  • no distributions are available (meaning we have to manage distributions, for example Conda or PyPI, ourselves)

Now that ANTS, as other UKMO tools, are on GitHub, I think it will be easier to have released distributions of ANTS and other tools (e.g., mule, shumlib, ug-ants, etc.) directly connected to their respective GitHub repos and managed in collaboration with the MetOffice.

There is a related issue for mule, for which Scott also has a conda-forge package proposal in queue for review.

I think similar procedures should be put in place for ANTS.

23

Thanks @atteggiani, can you please confirm that analysis3-25.08 will remain active (currently throwing warning on load that it will soon be depreciated) until the release of the dedicated seperate ANTS environment? Can you also please confirm which other packages are planned to be included in the seperate ANTS environment?

24

I’ll defer to @rbeucher on this, as the ACCESS-NRI MED team is managing the analysis3 environments.

There hasn’t been a discussion about this yet.
When the deployment infrastructure is ready (hopefully within a month), I think this could be a good topic to discuss in a Working Group (maybe the Atmosphere one?).

25

Hi @mlipson

We will not deprecate the environment without providing an alternative for your workflow.
As @atteggiani said, we have flagged the issue with the developers. There have been some changes recently, and we are hopeful we will be able to make the new packages available soon.

26

FYI, ants is now released as a stand-alone containerised environment.
This will atomatically be aupdated with new ants releases.

27

Hi all. I’m building a rose/cylc task for the regional Climate model ancillary suite which regrids a MOM6 ocean mask.

My task used to

  • Load conda/analysis3-25.12 (because it contains the mule dependency)
  • Use the version of ants installed with ACCESS-rAM3

However, this now fails with the following warnings and errors

 module that was compiled using NumPy 1.x cannot be run in
NumPy 2.4.6 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
..
    from mule.packing import wgdos_pack_field, wgdos_unpack_field
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.12/lib/python3.11/site-packages/mule/packing.py", line 107, in <module>
    raise ImportError(err.args + (msg,))
ImportError: ('numpy.core.multiarray failed to import', 'SHUMlib Packing library found, but failed to import')

I note the release of the new ants package here :

However, this environment doesn’t contain xarray. (It does contain numpy, iris and espy), which means my current script won’t run : https://github.com/21centuryweather/rCM3-ancil-suite/blob/main/bin/regrid_MOM6_UM.py

My task follows the working practises of @lachlanswhyborn and the AM3/CM3 ancillary suite :

https://github.com/ACCESS-NRI/CCI-Ancillary-Suite/tree/main/app/landseamask_from_ocean_mesh

https://github.com/ACCESS-NRI/CCI-Ancillary-Suite/blob/main/flow.cylc

The default python environment used in this suite is

		init-script = """
			module use /g/data/xp65/public/modules
			module load conda/analysis3-25.08
			module load nco
		"""

Can the global CCI ancillary suite still with this environment?

If so, what am I missing?

28

The existing ancillary suite definitely still runs with conda/analysis-25.08, but that version was chosen specifically because it was the last environment that shipped with ants (or at least I thought it was- maybe it was mule that was no longer shipped). I suspect ants doesn’t specify its dependencies correctly, which is why the environment managed to concretize with an invalid combination of ants and numpy.

We can deploy a new environment that contains the UKMO modules and xarray- I don’t remember the reason we didn’t include it initially. It feels a bit superfluous to use them in the same task, since they have significantly overlapping functionality- can you load the desired modules for the respective tasks? A newer conda/analysis version for xarray tasks, and the ants package for ants/iris tasks?

29

Yeah I could split the workflow so that it creates the land-sea mask grids and required metadata and save that to netcdf.

And then write a seperate task to load the netcdf file and use the new ants library to save it as a UM ancillary.

I’ll let you know how I go.

30

Ah yep I forgot about that bit. The most recent versions of the conda environment do contain the iris package, and the ants savers do just wrap the iris savers AFAIK. I’ve been trying to move to iris where possible, as ANTS is not public and people generally do not have visibility of the contents of ants.

31

I recall that ants has extra functionality when saving ancillary files that iris doesn’t. i.e. iris won’t transfer all the cube metadata to the ancillary header file.

I’ll double check that and get back to you.

32

There is nothing stopping someone making a PR to add xarray to the ants environment:

Note that it is pretty lean at the moment, which is a good thing. The more packages you add, the harder it becomes to maintain. Eventually you could end up with an entire analysis environment with ants and a bunch of out-dated packages because ants is so prescriptive about dependencies.

But if xarray is obviously required for ancillary processing workflows (and I can see it might be) then we can add it.

33

Ok I will add an xarray issue to the repo (I don’t think I have permissions to make branches).

That would ensure that @lachlanswhyborn 's scripts will work with the new ants environment.

It also help those using the ants environment to post-process and diagnose ancillary and PP files.

34

Eventually you could end up with an entire analysis environment with ants and a bunch of out-dated packages because ants is so prescriptive about dependencies.

I don’t think this is necessarily such a bad thing- the main point of the creation of this ants environment was to be able to remove them from our analysis environments, so that everyone that doesn’t have to deal with UM nonsense are not held back by the people that do have to deal with it. I’m on board with adding xarray to this environment.