Warnings in xp65 conda env

COSIMA Ocean
1

I am adapting my past scripts for new conda env (transition hh5→xp65) and noticed that my jobs sometimes get thousands line of warnings. I’ll copy a few most common ones and would like to get advice how to avoid them:

/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/intake_esm/core.py:259: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass (name,) instead of name to silence this warning.
records = grouped.get_group(internal_key).to_dict(orient=‘records’)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/intake_esm/source.py:280: ConcatenationWarning: Attempting to concatenate datasets without valid dimension coordinates: retaining only first dataset. Request valid dimension coordinate to silence this warning.
warnings.warn(

Also, it got a hundred of the same warning for one specific line: /g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/intake_esm/source.py:82: FutureWarning: In a future version, xarray will not decode timedelta values based on the presence of a timedelta-like units attribute by default. Instead it will rely on the presence of a timedelta64 dtype attribute, which is now xarray’s default way of encoding timedelta64 values. To continue decoding timedeltas based on the presence of a timedelta-like units attribute, users will need to explicitly opt-in by passing True or CFTimedeltaCoder(decode_via_units=True) to decode_timedelta. To silence this warning, set decode_timedelta to True, False, or a ‘CFTimedeltaCoder’ instance.
ds = xr.open_dataset(url, **xarray_open_kwargs)

My guess that the warnings are related to loading variables from intake catalog. Here’s how I load them:

        variable = 'salt'
        tracer = catalog[iaf_cycle3].search(
            variable=variable,
            frequency='1mon'
        ).to_dask()
2

After giving all these warnings, the job finished unsuccessfully with the following error:

2025-08-27 12:29:09,548 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.Traceback (most recent call last):File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/comm/tcp.py”, line 226, in readframes_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/worker.py”, line 1267, in heartbeatresponse = await retry_operation(^^^^^^^^^^^^^^^^^^^^^^File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/utils_comm.py”, line 416, in retry_operationreturn await retry(^^^^^^^^^^^^File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/utils_comm.py”, line 395, in retryreturn await coro()^^^^^^^^^^^^File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/core.py”, line 1259, in send_recv_from_rpcreturn await send_recv(comm=comm, op=key, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/core.py”, line 1018, in send_recvresponse = await comm.read(deserializers=deserializers)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/comm/tcp.py”, line 237, in readconvert_stream_closed_error(self, e)File “/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/distributed/comm/tcp.py”, line 137, in convert_stream_closed_errorraise CommClosedError(f"in {obj}: {exc}") from excdistributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:38990 remote=tcp://127.0.0.1:35201>: Stream is closed
3

Hi Polina,

Thanks for posting this - these are all coming out of intake-esm, and one in particular I’ve been meaning to fix for a while, but couldn’t remember how to recreate it :sweat_smile:.

  1. /g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/intake_esm/core.py:259: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass (name,) instead of name to silence this warning.
    records = grouped.get_group(internal_key).to_dict(orient=‘records’)

Very much on my todo list, I’ll fix it now but it’ll be a few weeks before the fix is in the xp65 environment. Unfortunately it can’t be suppressed right now. I’ll update this comment when it is fixed.

  1. /g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/intake_esm/source.py:280: ConcatenationWarning: Attempting to concatenate datasets without valid dimension coordinates: retaining only first dataset. Request valid dimension coordinate to silence this warning.
    warnings.warn(

I think that this should only happen when you load coordinate variables. I think you can work around it with something like:

# Generates warnings
esm_datastore = esm_datastore.search(variable='geolon_t').to_dask() 

esm_datastore = esm_datastore.search(variable='geolon_t')
esm_datastore = esm_datastore.search(path = esm_datastore.unique().path[0]).to_dask()
# Does not generate warning

This effectively limits us to the first result, only opens a single file, and so shouldn’t generate the warning.

  1. /g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/lib/python3.11/site-packages/intake_esm/source.py:82: FutureWarning: In a future version, xarray will not decode timedelta values based on the presence of a timedelta-like units attribute by default. Instead it will rely on the presence of a timedelta64 dtype attribute, which is now xarray’s default way of encoding timedelta64 values. To continue decoding timedeltas based on the presence of a timedelta-like units attribute, users will need to explicitly opt-in by passing True or CFTimedeltaCoder(decode_via_units=True) to decode_timedelta. To silence this warning, set decode_timedelta to True, False, or a ‘CFTimedeltaCoder’ instance.
    ds = xr.open_dataset(url, **xarray_open_kwargs)

I think this is related to a change in defaults in xarray (annoyingly, we fixed a closely related one a couple of months ago). To get around it, you can do:

variable = 'salt'
        tracer = catalog[iaf_cycle3].search(
            variable=variable,
            frequency='1mon'
        ).to_dask(
    xarray_open_kwargs = {
        "decode_timedelta" : True, # or False, or a 'CFTimedeltaCoder' instance 
)

as with 1, this can be fixed in intake-esm, so I’ll do that. Not sure how quick it’ll be though as I don’t fully understand the implications of changing the default for other users yet.

Cheers!

1 Like
4

I think this is just a dask communication error - can you confirm if it fail with the same error if you retry the script verbatim?

EDIT: I’ve just tried the code snippet you passed me, and I get a segmentation fault. This is related to another, much more complicated problem, but can be avoided by passing threaded=False in the to_dask() call: eg.

variable = 'salt'
        tracer = catalog[iaf_cycle3].search(
            variable=variable,
            frequency='1mon'
        ).to_dask(
    threaded=False,
    xarray_open_kwargs = {
        "decode_timedelta" : True, # or False, or a 'CFTimedeltaCoder' instance 
)

If running the script again fails in the same way, could you let me know if this fixes it (N.B If so, this is a really handy test case that I know I can reliably trigger a segfault with, so I’d be really grateful to know the result either way).

1 Like