xp65 analysis3 environments from 25.06 onwards are incompatible with NCI’s cylc7 environment.
A minimal example:
module purge
module use /g/data/xp65/public/modules
module use /g/data/hr22/modulefiles
module load conda/analysis3-25.07
module load cylc7/23.09
rose task-env
Produces a stack trace
Traceback (most recent call last):
File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 548, in <module>
main()
File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 530, in main
known_paths = addusersitepackages(known_paths)
File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 275, in addusersitepackages
user_site = getusersitepackages()
File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 250, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 240, in getuserbase
USER_BASE = get_config_var('userbase')
File "/g/data/hr22/apps/cylc7/lib/python2.7/sysconfig.py", line 530, in get_config_var
return get_config_vars().get(name)
File "/g/data/hr22/apps/cylc7/lib/python2.7/sysconfig.py", line 482, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/g/data/hr22/apps/cylc7/lib/python2.7/sysconfig.py", line 365, in _init_posix
_temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ImportError: No module named _sysconfigdata_x86_64_conda_linux_gnu
This appears to be because the analysis3 modulefile from 25.06 is setting an environment variable
Haven’t had a look yet but I’m working on some other help stuff this morning so I’ll take a look at this too.
Off the top of my head, there are have been no major package changes/upgrades in the past few months, so sticking to the old version of the environment for now shouldn’t be an issue.
Okay: it looks like _CONDA_PYTHON_SYSCONFIGDATA_NAME changed from _sysconfigdata_x86_64_conda_cos6_linux_gnu to _sysconfigdata_x86_64_conda_linux_gnu in 25.06.
I’m not particularly sure that we can resolve this (without downgrading the version of conda we are using). The _CONDA_PYTHON_SYSCONFIGDATA_NAME variable appears to be set by micromamba as part of the environment creation process. My guess is that that cyclc7/23.09 is expecting an older version of conda, which we lost compatibility with sometime between the 24.11.3 and 25.5.0 versions.
Is there a reason for the module file to set this variable? It should only be required inside the container so maybe all the variables the module sets can be moved to the wrapper script instead.
The variables were needed in the modulefile before containers as we were effectively doing a ‘conda activate’ inside the module file, but that ‘conda activate’ should be runnable in the container wrapper script (or in the container runscript, not sure exactly how it’s been set up) directly. Then the container has the environment that it needs but it’s not affecting other applications.
I’m not sure we can fix this without aligning conda versions - the environment variable is set by micromamba when it creates the conda environment, and so looks like it would require either an upgrade to the cylc conda version or a downgrade to the analysis3 version.
Unless there are packages in newer versions of the analysis3 environment you need, I think it’s going to be easier to just stick to using conda/analysis3-25.05
I would prefer finding a solution for making newer conda/analysis3 versions to be still compatible with cylc7.
If this is really not possible, we would have to make it very clear to users that, in case they are using cylc7, the max conda/analysis3 version they can use is 25.05.
A note on that: what happens in some time when 25.05 will be considered “obsolete” and maybe even removed? That will completely remove support for conda/analysis3 and cylc7?
I agree with @atteggiani. I hope to start deprecating the old analysis3 soon so it would be good to find a solution and make the newer analysis3 work with cylc7
Just want to note that some supported models, e.g. ACCESS-rAM3, use cycl7, so I assume will need to be supported in general. However, I don’t have a need to use cylc7 with analysis3, so I am not personally troubled with it dropping support from xp65.