Error when loading an intake variable in new conda env 25.09

Technical
,
1

In new conda env:analysis3-25.09, I get an error when simply load a variable from intake. The same script works fine in older conda envs but I need to use 25.09 because of the newly added package available in it. Help plz

 iaf_cycle3 = ‘01deg_jra55v140_iaf_cycle3’

var=‘ht’
ht = catalog[iaf_cycle3].search(variable=var).to_dask(
    threaded=False,
    xarray_open_kwargs = {“decode_timedelta” : True})
ht = ht[‘ht’]

Error message:

KeyError: '01deg_jra55v140_iaf_cycle3'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Cell In[7], line 2
      1 var='ht'
----> 2 ht = catalog[iaf_cycle3].search(
      3     variable=var
      4 ).to_dask()
      5 ht = ht['ht']
      9 land_mask = np.squeeze(ht.values)

File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/intake_dataframe_catalog/core.py:191, in DfFileCatalog.__getitem__(self, key)
    186     if len(yamls) > 1:
    187         assert all(y == yaml_text for y in yamls)
    189     self._entries[key] = LocalCatalogEntry(
    190         name=key, **yaml.safe_load(yaml_text)["sources"][key]
--> 191     ).get()
    192     return self._entries[key]
    193 raise KeyError(
    194     f"key='{key}' not found in catalog. You can access the list of valid source keys via the .keys() method."
    195 ) from e

File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/intake/catalog/local.py:313, in LocalCatalogEntry.get(self, **user_parameters)
    310     return self._default_source
    312 plugin, open_args = self._create_open_args(user_parameters)
--> 313 data_source = plugin(**open_args)
    314 data_source.catalog_object = self._catalog
    315 data_source.name = self.name

File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/intake_esm/core.py:124, in esm_datastore.__init__(self, obj, progressbar, sep, registry, read_csv_kwargs, columns_with_iterables, storage_options, threaded, **intake_kwargs)
    122     self.esmcat = ESMCatalogModel.from_dict(obj)
    123 else:
--> 124     self.esmcat = ESMCatalogModel.load(
    125         obj, storage_options=self.storage_options, read_csv_kwargs=read_csv_kwargs
    126     )
    128 self.derivedcat = registry or default_registry
    129 self._entries = {}

File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/intake_esm/cat.py:257, in ESMCatalogModel.load(cls, json_file, storage_options, read_csv_kwargs)
    255 cat = cls.model_validate(data)
    256 if cat.catalog_file:
--> 257     cat._frames = cat._df_from_file(cat, _mapper, storage_options, read_csv_kwargs)
    258 else:
    259     cat._frames = FramesModel(
    260         lf=pl.LazyFrame(cat.catalog_dict),
    261         pl_df=pl.DataFrame(cat.catalog_dict),
    262         df=pl.DataFrame(cat.catalog_dict).to_pandas(),
    263     )

File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/intake_esm/cat.py:318, in ESMCatalogModel._df_from_file(self, cat, _mapper, storage_options, read_csv_kwargs)
    310     converters = read_csv_kwargs.pop('converters', {})  # Hack
    311     # See https://github.com/pola-rs/polars/issues/13040 - can't use read_csv.
    312     lf = pl.scan_csv(
    313         cat.catalog_file,  # type: ignore[arg-type]
    314         storage_options=storage_options,
    315         infer_schema=False,
    316         **read_csv_kwargs,
    317     ).with_columns(
--> 318         [
    319             pl.col(colname)
    320             .str.replace('^.', '[')  # Replace first/last chars with [ or ].
    321             .str.replace('.$', ']')  # set/tuple => list
    322             .str.replace_all("'", '"')
    323             .str.json_decode()  # This is to do with the way polars reads json - single versus double quotes
    324             for colname in converters.keys()
    325         ]
    326     )
    327     return FramesModel(lf=lf)
    328 else:

File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/intake_esm/cat.py:323, in <listcomp>(.0)
    310     converters = read_csv_kwargs.pop('converters', {})  # Hack
    311     # See https://github.com/pola-rs/polars/issues/13040 - can't use read_csv.
    312     lf = pl.scan_csv(
    313         cat.catalog_file,  # type: ignore[arg-type]
    314         storage_options=storage_options,
    315         infer_schema=False,
    316         **read_csv_kwargs,
    317     ).with_columns(
    318         [
    319             pl.col(colname)
    320             .str.replace('^.', '[')  # Replace first/last chars with [ or ].
    321             .str.replace('.$', ']')  # set/tuple => list
    322             .str.replace_all("'", '"')
--> 323             .str.json_decode()  # This is to do with the way polars reads json - single versus double quotes
    324             for colname in converters.keys()
    325         ]
    326     )
    327     return FramesModel(lf=lf)
    328 else:

TypeError: ExprStringNameSpace.json_decode() missing 1 required positional argument: 'dtype'
1 Like
2

Thanks for your request @polinash! I’ll work on finding someone to help address this and we will get back to you as soon as we can.

1 Like
4

I have very temporarily assigned myself to help out, but I’ll note that I’m just relaying some information from others in an effort to try and get you a workaround quickly.

It sounds like the issue is likely that the polars version needs to be downgraded in the current environment.

Could you please try to downgrade your polars version from your conda/analysis3 environment by running from a notebook:

!pip install polars=1.32.0

Please let me know if this works for you.

If this works, note that this is a temporary fix, and will need to be undone once we are able to update the conda/analysis3 environment (potentially tomorrow, or likely sometime next week). We can help you with this step if/once it becomes relevant.

5

The error should have been fixed in the new conda env, which stemmed from an optional argument being made mandatory in polars 1.33.0.

If you did follow Paige’s temporary fix, you will need to do

rm -r $HOME/.locals/lib/python3.11/site-packages to remove anything your pip install put in there.

1 Like