Analysis3-25.06 onwards incompatible with cylc7

General
,
1

xp65 analysis3 environments from 25.06 onwards are incompatible with NCI’s cylc7 environment.

A minimal example:

module purge

module use /g/data/xp65/public/modules
module use /g/data/hr22/modulefiles

module load conda/analysis3-25.07
module load cylc7/23.09

rose task-env

Produces a stack trace

Traceback (most recent call last):
  File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 275, in addusersitepackages
    user_site = getusersitepackages()
  File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 250, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/g/data/hr22/apps/cylc7/lib/python2.7/site.py", line 240, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/g/data/hr22/apps/cylc7/lib/python2.7/sysconfig.py", line 530, in get_config_var
    return get_config_vars().get(name)
  File "/g/data/hr22/apps/cylc7/lib/python2.7/sysconfig.py", line 482, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/g/data/hr22/apps/cylc7/lib/python2.7/sysconfig.py", line 365, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ImportError: No module named _sysconfigdata_x86_64_conda_linux_gnu

This appears to be because the analysis3 modulefile from 25.06 is setting an environment variable

_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_linux_gnu

which breaks older conda environments

3 Likes
2

Ping @bethanwhite

1 Like
3

Thank you @Scott!

Is there a recommended analysis environment for cylc7?

And specifying an exact (compatible) environment should fix things?

4

conda/analysis3-25.05 works fine from what I’ve tested just running Cylc, I think @am1625 is checking the suite itself

1 Like
5

Fab, I’ll hold off until I hear from Anderson.

Fast problem-solving on a Thursday morning, hats off!

1 Like
7

Thanks for flagging this issue @Scott. I just encountered it while trying to run an ACCESS-CM2 suite.

@rbeucher do you have an update?

8

No update yet. @CharlesTurner do you think you can have a look at this?
Any reason to not use analysis3-25.06?

9

Haven’t had a look yet but I’m working on some other help stuff this morning so I’ll take a look at this too.

Off the top of my head, there are have been no major package changes/upgrades in the past few months, so sticking to the old version of the environment for now shouldn’t be an issue.

1 Like
11

Okay: it looks like _CONDA_PYTHON_SYSCONFIGDATA_NAME changed from _sysconfigdata_x86_64_conda_cos6_linux_gnu to _sysconfigdata_x86_64_conda_linux_gnu in 25.06.

I’ve turfed up some old stuff related to conda-forge migrating default compiler toolchains - apparently the _cos6 part related to a CentOS6 compatible compiler toolchain (ie. 2020-07-01 | conda-forge | community-driven packaging for conda), which seems to have some conflicts with the standard - ie. sysroot_linux-64 and libdrm-cos6-x86_64 conflict over path /x86_64-conda-linux-gnu/sysroot/lib · Issue #12899 · ContinuumIO/anaconda-issues · GitHub.

The analysis3-25.06 environment uses conda 25.5.0, upgraded from analysis3-25.05 using conda 24.11.3. There is also some stuff on the conda-forge site regarding changes, again regarding CentOS system pacakges, from late November last year: Updating our default docker images | conda-forge | community-driven packaging for conda.

I’m not particularly sure that we can resolve this (without downgrading the version of conda we are using). The _CONDA_PYTHON_SYSCONFIGDATA_NAME variable appears to be set by micromamba as part of the environment creation process. My guess is that that cyclc7/23.09 is expecting an older version of conda, which we lost compatibility with sometime between the 24.11.3 and 25.5.0 versions.

12

Is there a reason for the module file to set this variable? It should only be required inside the container so maybe all the variables the module sets can be moved to the wrapper script instead.

The variables were needed in the modulefile before containers as we were effectively doing a ‘conda activate’ inside the module file, but that ‘conda activate’ should be runnable in the container wrapper script (or in the container runscript, not sure exactly how it’s been set up) directly. Then the container has the environment that it needs but it’s not affecting other applications.

13

I’m not sure we can fix this without aligning conda versions - the environment variable is set by micromamba when it creates the conda environment, and so looks like it would require either an upgrade to the cylc conda version or a downgrade to the analysis3 version.

Unless there are packages in newer versions of the analysis3 environment you need, I think it’s going to be easier to just stick to using conda/analysis3-25.05