Build-esm-datastore failing in conda/analysis3-26.03

build-esm-datastore is failing for me with conda/analysis3-26.03:

$ module use /g/data/xp65/public/modules
$ module load conda/analysis3
Loading conda/analysis3-26.03
  Loading requirement: singularity
$ cd /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88
$ build-esm-datastore --builder AccessOm3Builder --datastore-name datastore
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/intake/catalog/utils.py:173: UserWarning: Shell command not executed due to getshell=False
  warnings.warn("Shell command not executed due to getshell=False")
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/intake/catalog/utils.py:182: UserWarning: Shell command not executed due to getshell=False
  warnings.warn("Shell command not executed due to getshell=False")
Generating esm-datastore for .
Building esm-datastore...
<frozen importlib._bootstrap>:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/wavespectra/output/ww3.py:25: ResourceWarning: unclosed file <_io.TextIOWrapper name='/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/wavespectra/output/ww3.yml' mode='r' encoding='utf-8'>
  VAR_ATTRIBUTES = yaml.load(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/utils.py:279: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
  warnings.warn(
Sucessfully built esm-datastore!
Saving esm-datastore to /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/bin/build-esm-datastore", line 8, in <module>
    sys.exit(use_esm_datastore())
             ^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/cli.py", line 1097, in use_esm_datastore
    use_datastore(
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/experiment/main.py", line 127, in use_datastore
    builder_instance.save(
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/builders.py", line 199, in save
    self._save(name, description, directory, use_parquet)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/access_nri_intake/source/builders.py", line 157, in _save
    super().save(
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/pydantic/deprecated/decorator.py", line 56, in wrapper_function
    return vd.call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/pydantic/deprecated/decorator.py", line 151, in call
    return self.execute(m)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/pydantic/deprecated/decorator.py", line 227, in execute
    return self.raw_function(**d, **var_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/ecgtools/builder.py", line 317, in save
    cat.save(
TypeError: ESMCatalogModel.save() got an unexpected keyword argument 'file_format'

It works with conda/analysis3-26.02

Hi, @aekiss ,I’m the triager this week. It looks like this issue is related to the analysis3 module and Intake. As it’s a bit late today, I’ll triage your request tomorrow if no one has responded by then.

1 Like

Thanks @RhaegarZeng, no hurry - I have a workaround

1 Like

This is really weird:

$ module load conda/analysis3-26.02
Loading conda/analysis3-26.03
$ python -c "import ecgtools; print(ecgtools.__version__)"
2025.10.7
$ module unload conda/analysis3-26.02 && module unload conda/analysis3-26.03 
Unloading conda/analysis3-26.02
  Unloading useless requirement: singularity
Loading conda/analysis3-26.03
$ python -c "import ecgtools; print(ecgtools.__version__)"
2025.10.7

So both environments contain the same version of ecgtools, the package that is apparently causing the error.

Weirder still, I’m able to build it fine in 26.03:

$ module unload conda/analysis3-26.04 && module load conda/analysis3-26.03
Unloading conda/analysis3-26.04
  Unloading useless requirement: singularity
Loading conda/analysis3-26.03
  Loading requirement: singularity

$ build-esm-datastore --builder AccessOm3Builder --datastore-name datastore --expt-dir /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/intake/catalog/utils.py:173: UserWarning: Shell command not executed due to getshell=False
  warnings.warn("Shell command not executed due to getshell=False")
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.03/lib/python3.12/site-packages/intake/catalog/utils.py:182: UserWarning: Shell command not executed due to getshell=False
  warnings.warn("Shell command not executed due to getshell=False")
Datastore found in /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88, verifying datastore integrity...
Parsing experiment dir...
...
Datastore integrity verified!
Datastore found in /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88/datastore.json!
Please note that this has not added the datastore to the access-nri-intake catalog.
To add to catalog, please run 'scaffold-catalog-entry' for help on how to do so.
To open the datastore, run `intake.open_esm_datastore('/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88/datastore.json', columns_with_iterables=['variable'])` in a Python session.

Can you run

$ python -c "import ecgtools; print(ecgtools.__version__)"
$ ls ~/.local/lib/python3.11/site-packages/

,

$ ls ~/.local/lib/python3.12/site-packages/

and

python -c "import sys; print(sys.executable)"

for me in 26.03 and let me know what they return? My suspicion is that something has somehow mangled the ecgtools version. I suspect we’ll see ecgtools in ~/.local/lib/python3.12/site-packages/ but not ~/.local/lib/python3.11/site-packages/.

Cheers!

Thanks @CharlesTurner, I get

$ python -c "import ecgtools; print(ecgtools.__version__)"
2025.10.7
$ ls ~/.local/lib/python3.11/site-packages/
ls: cannot access '/home/156/aek156/.local/lib/python3.11/site-packages/': No such file or directory
$ ls ~/.local/lib/python3.12/site-packages/
comm                      ipykernel                  jupyter_client-8.8.0.dist-info  nci_ipynb                        psutil                  pyzmq-27.1.0.dist-info   _yaml
comm-0.2.3.dist-info      ipykernel-7.1.0.dist-info  jupyter_core                    nci_ipynb-2023.11.0.3.dist-info  psutil-7.2.1.dist-info  pyzmq.libs               yaml
debugpy                   ipykernel_launcher.py      jupyter_core-5.9.1.dist-info    nest_asyncio-1.6.0.dist-info     __pycache__             tornado                  zmq
debugpy-1.8.19.dist-info  jupyter_client             jupyter.py                      nest_asyncio.py                  pyyaml-6.0.3.dist-info  tornado-6.5.4.dist-info
$ python -c "import sys; print(sys.executable)"
/g/data/xp65/public/apps/med_conda_scripts/analysis3-26.03.d/bin/python

I’m fairly close to stumped right now… there’s nothing there that indicates that there should be any issues.

I’ve double checked and confirmed that:

  1. The version of ecgtools you’re using has the file_format argument, ie. the source of the error doesn’t make sense.
  2. Nothing about the access_nri_intake package changed in a meaningful way between 26.02 and 26.03, so the correct arguments should be being passed through.
  3. It builds correctly in the 26.03 environment for me.

You do have some packages in your ~/.local/lib/python3.12/site-packages, which I’d recommend removing. It miiiight be the source of the error if it’s doing something very subtle and weird, but I can’t see anything obvious in there. It will have come in via a pip install ..., and is generally one of our main sources of bugs - having it there at all is generally inadvisable.

If removing that folder doesn’t do anything, can you post your .bash_profile and .bashrc in here for me?

Cheers

Thanks @CharlesTurner

It gets weirder. Now analysis3-26.02 doesn’t work for me either (see below). I’ve renamed ~/.local so it’s not in my path. My home dir is ~aek156 if you want to have a look around.

$ module use /g/data/xp65/public/modules
$ module load conda/analysis3-26.02
$ cd /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88
$ build-esm-datastore --builder AccessOm3Builder --datastore-name datastore
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.02/lib/python3.11/site-packages/intake/catalog/utils.py:173: UserWarning: Shell command not executed due to getshell=False
  warnings.warn("Shell command not executed due to getshell=False")
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.02/lib/python3.11/site-packages/intake/catalog/utils.py:182: UserWarning: Shell command not executed due to getshell=False
  warnings.warn("Shell command not executed due to getshell=False")
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.02/lib/python3.11/site-packages/access_nri_intake/cli.py:1097: DataStoreWarning: esm-datastore broken due to path in JSON does not match csv.gz. Regenerating datastore...
  use_datastore(
Building esm-datastore...
<frozen importlib._bootstrap>:241: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.02/lib/python3.11/site-packages/wavespectra/output/ww3.py:25: ResourceWarning: unclosed file <_io.TextIOWrapper name='/g/data/xp65/public/apps/med_conda/envs/analysis3-26.02/lib/python3.11/site-packages/wavespectra/output/ww3.yml' mode='r' encoding='UTF-8'>
  VAR_ATTRIBUTES = yaml.load(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/g/data/xp65/public/apps/med_conda_scripts/analysis3-26.02.d/bin/build-esm-datastore: line 165: 1789138 Killed                  "$SINGULARITY_BINARY_PATH" -s exec --bind "${bind_str}" ${overlay_args} "${CONTAINER_PATH}" "${cmd_to_run[@]}"
[aek156@gadi-login-04 MC_25km_jra_iaf+wombatlite-test3v2-00532b88]$ <jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at /rustc/b03b3a7ec92682be2917540b679478d41c95a30c/library/std/src/sync/lazy_lock.rs:209:19:
Once instance has previously been poisoned

thread '<unnamed>' panicked at /rustc/b03b3a7ec92682be2917540b679478d41c95a30c/library/std/src/sync/lazy_lock.rs:209:19:
Once instance has previously been poisoned

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)
<jemalloc>: background thread creation failed (11)

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at crates/polars-core/src/lib.rs:65:10:
could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at /rustc/b03b3a7ec92682be2917540b679478d41c95a30c/library/std/src/sync/lazy_lock.rs:209:19:
Once instance has previously been poisoned

If you can successfully generate a datastore for /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test3v2-00532b88 (which I updated with new data this morning), could you tell me the location please so I can use it for analysis?

That is really weird… I’ll recreate a datastore for it and put it in a location you can grab in a minute.

That other error you’re now getting is related to threading (rayon/rust in Polars) - I assume you’ve run that from a login node? I often find Rust binaries get blocked on a login node, I’ve yet to investigate the cause though - running them in an ARE session always seems to work fine.

I’ll update this comment once I’ve rebuilt a datastore - but we might want to do some digging to work out why you’re getting all these strange issues anyway…

I’ve put the built datastore in /scratch/tm70/ct1163/andrew-file-format, as esm_datastore.{json,csv}. Should be globally readable/writable, so I don’t forsee any issues with moving it somewhere. I’ve updated the internal file handle too so that it won’t break upon moving.

1 Like

Thanks @CharlesTurner. Yep, the above was from a login node. I just tried in an ARE jupyter terminal session with analysis3-26.03 and it failed as in the first post. But it worked with analysis3-26.02 on ARE.

Actually it strikes me that the rust on a login node issues are something that @sam.green might have come across and figured out a solution to?

Also, I’m now reproducing the error, so I’ll track it down and get that fixed.

1 Like

Okay, the root cause of this is that esmvalcore=2.14 now has a hard dependency on intake-esm. This causes the mainline version of intake-esm - not our fork which allows writing out parquet serialised catalogues - into the analysis3 environment:

conda list | rg 'intake|ecgtools'
access-intake-utils                     0.1.7               py_0                          accessnri
access-nri-intake                       1.6.0               py_0                          accessnri
ecgtools-access                         2025.10.7           py_0                          accessnri
intake                                  2.0.9               pyhd8ed1ab_0                  conda-forge
intake-dataframe-catalog                0.5.1               pyhd8ed1ab_0                  conda-forge
intake-esm                              2025.12.12          pyhd8ed1ab_0                  conda-forge
intake-esm-access                       2025.9.18           py_0                          accessnri

2 last lines being the most important.

From solved.json in the analysis3 repo/definition :

 {
   "name": "esmvalcore",
   "version": "2.14.0",
   "build": "pyhc364b38_0",
   "build_number": 0,
   "size_bytes": 6546992,
   "kind": "conda",
   "source": "https://conda.anaconda.org/conda-forge",
   "license": "Apache-2.0",
   "license_family": "APACHE",
   "is_explicit": true,
   "md5": "e4d7acc47b23802da419a8de8a49bbea",
   "sha256": "c4cc1a0c84900421add513684c79a80467ebfdf3d3c5a8a4f7859f48779a2cc3",
   "arch": null,
   "platform": null,
   "subdir": "noarch",
   "timestamp": 1773221341802,
   "noarch": "python",
   "file_name": "esmvalcore-2.14.0-pyhc364b38_0.conda",
   "url": "https://conda.anaconda.org/conda-forge/noarch/esmvalcore-2.14.0-pyhc364b38_0.conda",
   "requested_spec": "2.14.*",
   "constrains": [],
   "depends": [
     "python >=3.12",
     ...
     "intake-esm",
     ...
     "python"
   ],
   "track_features": []
}

@rbeucher we might need to have a bit of a think about this - as things stand, the esmvalcore/intake-esm integration will make this issue totally insoluble…