Xarray to_zarr causing errors in new xp65 env that weren't present in hh5

General
, , ,
1

Dear Hive,

I am using xarray to_zarr utility.

Here is the fragment of code:

data_full = xr.open_mfdataset(file_list, preprocess = preprocess_ds, parallel = True).persist()

data_stacked = data_full[all_vars].to_array(dim = "s_vars")
data_ds = xr.Dataset({"data": data_stacked}).chunk(time = -1, s_vars = -1, lat = chunk_size, lon = chunk_size).persist()

data_ds.to_zarr(chunkdir + f'{year_start}-{year_end}-{"-".join(all_vars)}', consolidated = True, mode = 'w')

The error I am getting is:

Traceback (most recent call last):
  File "/g/data/mn51/users/jp0715/acs/bias_correction/mrnbc/code/pymrnbc/mrnbc-python-wrapper/acs_chunk_mrnbc.py", line 153, in <module>
    data_ds.to_zarr(chunkdir + f'{year_start}-{year_end}-{"-".join(all_vars)}', consolidated = True, mode = 'w')
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.04/lib/python3.11/site-packages/xarray/core/dataset.py", line 2270, in to_zarr
    return to_zarr(  # type: ignore[call-overload,misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

and this:

  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.04/lib/python3.11/site-packages/dask/array/core.py", line 4614, in load_store_chunk
    if x is not None and x.size != 0:
  ^^^^^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'size'

We were using hh5 analysis-24.12. I have tried a few different xp65 envs including: 24.12, 25.04 and 25.05. The error messages were slightly different (from the second fragment I sent), however all occurred in the to_zarr call.

2

Thanks for posting @jpeter. Are you hoping to get help from ACCESS-NRI with this? Adding the help tag to your post will mean it gets triaged by our support team.

See Support FAQ (Frequently Asked Questions)

If you are after support, it would be very helpful if you could provided a minimal reproducible example.

See How to create a minimal complete reproducible example

1 Like