Dask is crashing: ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found

raslekar · 20 January 2026 06:14

Description of request:

I am trying to use a local dask cluster for data analysis and working with my own intake catalog. But whenever I am actually trying to open any files or do some computation, the workers are crashing before even being registered with the client. The most glaring error I have repeatedly seen during this is ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29’ not found.

How can I go about this?

Environment:

Details of my session on ARE:

Cluster: ncigadi
Walltime (hours): 3
Queue: normalbw
Compute Size: xxlargemem
Project: zn44
Storage: scratch/zn44+gdata/zn44+gdata/xp65+gdata/jk72+scratch/jk72
Module directories: /g/data/xp65/public/modules/
Modules: conda/analysis3
Jobfs size: 10GB

What executed:

from dask.distributed import Client, LocalCluster
 
cluster = LocalCluster(
    n_workers=7,
    threads_per_worker=4
)

client = Client(cluster)
client


spinup = intake.open_esm_datastore("caco3_shallow_diss_spinup.json",
                                   columns_with_iterables=["variable"])

spinup_ds = spinup.search(variable=["alk"], frequency='1mon', realm="ocean").to_dask(xarray_open_kwargs={"use_cftime": True})

Actual results:

2026-01-20 16:27:39,032 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,043 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1273, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 416, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 395, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1018, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:41388 remote=tcp://127.0.0.1:39397>: Stream is closed
2026-01-20 16:27:39,045 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x1545d794bc50>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,046 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,057 - distributed.worker - WARNING - Scheduler was unaware of this worker; shutting down.
2026-01-20 16:27:39,062 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x152cd2b1b7d0>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,067 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,072 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,072 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,072 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,080 - distributed.worker - WARNING - Scheduler was unaware of this worker; shutting down.
2026-01-20 16:27:39,083 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1273, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 416, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 395, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1018, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:41368 remote=tcp://127.0.0.1:39397>: Stream is closed
2026-01-20 16:27:39,083 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1273, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 416, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 395, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1018, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:41344 remote=tcp://127.0.0.1:39397>: Stream is closed
2026-01-20 16:27:39,083 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1273, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 416, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/utils_comm.py", line 395, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 1018, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:41358 remote=tcp://127.0.0.1:39397>: Stream is closed
2026-01-20 16:27:39,086 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x1534aa456610>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,086 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x14e1d1c87ad0>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,086 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x14d854a13890>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,085 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,093 - distributed.worker - WARNING - Scheduler was unaware of this worker; shutting down.
2026-01-20 16:27:39,092 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x152727f8eb10>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:39,096 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x14a3aa367510>>, <Task finished name='Task-4' coro=<Worker.handle_scheduler() done, defined at /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py:218> exception=ImportError("/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)")>)
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/worker.py", line 1323, in handle_scheduler
    await self.handle_stream(comm)
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
          ^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ^^^^^^^^^^^^^^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/__init__.py", line 9, in <module>
    from intake_esm.core import esm_datastore
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/intake_esm/core.py", line 10, in <module>
    import xarray as xr
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/__init__.py", line 3, in <module>
    from xarray import coders, groupers, indexes, testing, tutorial, ufuncs
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coders.py", line 6, in <module>
    from xarray.coding.times import CFDatetimeCoder, CFTimedeltaCoder
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/xarray/coding/times.py", line 12, in <module>
    import pandas as pd
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 68, in <module>
    from pandas.core.frame import DataFrame
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/frame.py", line 153, in <module>
    from pandas.core.generic import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/generic.py", line 196, in <module>
    from pandas.core.window import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
2026-01-20 16:27:41,049 - distributed.nanny - ERROR - Worker process died unexpectedly
2026-01-20 16:27:41,100 - distributed.nanny - ERROR - Worker process died unexpectedly

This is updated in the client output after executing the last command mentioned above.

2026-01-20 16:27:39,042 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:35571'.
2026-01-20 16:27:39,056 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:34933'.
2026-01-20 16:27:39,079 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:39341'.
2026-01-20 16:27:39,083 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:40377'.
2026-01-20 16:27:39,084 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:37233'.
2026-01-20 16:27:39,085 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:33041'.
2026-01-20 16:27:39,092 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:44939'.

Expected results:

Simply the variable being loaded

Additional info:

I faced the same issue when using a PBS Cluster rather than a local cluster.

dougiesquire · 20 January 2026 21:58

Hi @raslekar

I haven’t been able to reproduce your issue exactly (using conda/analysis3-26.01), but as a general tip you should use threads_per_worker=1 when opening netcdf data in parallel:

- cluster = LocalCluster(
-     n_workers=7,
-     threads_per_worker=4
- )
+ cluster = LocalCluster(threads_per_worker=1)

See here for details.

If you’re still having issues after changing this, could you please provide the exact version of the conda/analysis3 environment you are using? You can find it in the top right corner of the notebook:

raslekar · 21 January 2026 01:30

Hi @dougiesquire

I tried the change you suggested, but still the same issue. I am using the same latest version as you conda/analysis3-26.01. I also tried other versions from 25.07 to 26.01, but same issue.

dougiesquire · 21 January 2026 01:49

Are you loading any modules in your .bashrc or .bash_profile?

raslekar · 21 January 2026 01:52

cluster = LocalCluster(
    processes=False
    threads_per_worker=1
)

Adding processes=False seems to have solved the issue.

raslekar · 21 January 2026 01:53

I’m not loading any modules in .bashrc or .bash_profile

dougiesquire · 21 January 2026 03:06

Hmmm, interesting. That cluster set-up may not be well suited to any CPU-bound work you do subsequently. My guess is that you may be ending up with different environments across your scheduler and workers, but I’m not sure why this is happening for you.

I found your Intake-ESM datastore on gdata/zn44 and I am still unable to reproduce the error using your code and conda/analysis3-26.01.

What about any code from conda init? See here

raslekar · 21 January 2026 03:43

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/etc/profile.d/conda.sh" ]; then
        . "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/etc/profile.d/conda.sh"
    else
        export PATH="/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

This is the code from conda init

dougiesquire · 21 January 2026 22:30

Thanks @raslekar. Could you try what Charles suggested in the topic I linked above - see here (alternatively just comment out that code in your .bashrc). Then start a new ARE session.

If that doesn’t help, it could be helpful to see your environment variables:

$ printenv

Feel free to send the output in a message to the help team if you’d rather not share publically. Alternatively, just seeing your PYTHONPATH , PATH and LIBPATH could be helpful:

echo $PYTHONPATH
echo $PATH
echo $LIBPATH

raslekar · 23 January 2026 01:45

Changing the conda init part did not work, it is giving the same error.
I have shared my environment variables with you in the message.
Alternatively, I am getting an output only 1 of the 3 paths mentioned above.

echo $LIBPATH

echo $PATH
/opt/conda/bin:/g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/bin:/home/581/ra7595/.local/bin:/home/581/ra7595/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/default/bin

echo $PYTHONPATH

This is after implementing the changes in conda init

dougiesquire · 23 January 2026 04:35

Thanks @raslekar. I still see /g/data/xp65/public/apps/med_conda/envs/analysis3-25.07/bin on your PATH. Can you please try these steps:

Comment out the code added by conda init in your .bashrc and save.
Start a new ARE session.

In your Jupyter notebook, print the following environment variables:

import os
print(f"LIBPATH: {os.environ.get('LIBPATH')}")
print(f"PATH: {os.environ.get('PATH')}")
print(f"PYTHONPATH: {os.environ.get('PYTHONPATH')}")

Confirm that the notebook fails and send through the output from the previous step.

raslekar · 23 January 2026 05:22

It failed even after commenting out the conda init part.

LIBPATH: None
PATH: /opt/conda/analysis3-26.01/bin:/g/data/xp65/public/apps/med_conda/envs/analysis3-25.11/condabin:/g/data/xp65/public/apps/openmpi/4.1.6/bin:/g/data/xp65/public/apps/nci_scripts:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/local/pbs/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH: :/home/581/ra7595/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/ncigadi/output/8daeedc3-9699-495d-9f78-004051431c14/lib/python3

dougiesquire · 30 January 2026 04:46

Thanks for sending through your environment in a private message @raslekar.

There’s very little difference between your environment and my working environment. One point of difference is that you have some GDAL/PROJ-related variables set. Have you previously tried to install GDAL by any chance? GDAL is notorious for causing problems and this issue (found by @CharlesTurner) looks very similar to what you are experiencing.

I’m rapidly running out of ideas…

Scott · 1 February 2026 23:01

It’s finding the wrong glibc from the error message - are the dask worker processes using the same environment as the master process? You may be able to see how they get launched in the logs

raslekar · 2 February 2026 00:55

I didn’t install GDAL but I tried installing jupyter-ai previously. Maybe that has something to do with this? I did uninstall it later though.

dougiesquire · 4 February 2026 00:54

The dask docs suggest it is unlikely they’d be different using a LocalCluster but worth checking.

@raslekar could you please run the following from your notebook and let us know if any output is printed:

import os
import sys
import hashlib
import json
import importlib.metadata as im
from dask.distributed import Client, LocalCluster
 
def hash_env():
    """
    Hash the current environment
    """
    env = dict(os.environ)
    env |= {
        "python_executable": sys.executable,
        "python_version": sys.version,
        "packages": {
            dist.metadata["Name"]: dist.version for dist in im.distributions()
        }
    }
    blob = json.dumps(env, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()

client = Client(threads_per_worker=1)

worker_env = client.run(hash_env)
master_env = hash_env()

for worker, env in worker_env.items():
    if env != master_env:
        print(f"Environment mismatch on {worker}")

If not, I thought I’d check that you still think this is something that is worth your time to resolve? It seems like you have a workaround and this is proving difficult to debug at our end without being able to reproduce your issue.

raslekar · 4 February 2026 03:26

No output is printed on execution of the cell, so the worker and master are using the same environment.
I understand it’s difficult to debug without being able to reproduce the issue. The workaround is working well for me, so I’ll stick with it unless that changes.

Thanks for taking the time to try to debug this.

dougiesquire · 4 February 2026 22:41

Thanks @raslekar. In that case I’ll go ahead and tag this topic as unresolved. Do let us know if anything changes.

Topic		Replies	Views
Using dask jobqueue with xp65 conda env Technical python , help , conda , dask	19	166	1 July 2025
Dask bag crashing in xp65 env (but wasn't in hh5) Infrastructure help , conda	22	212	22 May 2025
Analysis3 dask_jobqueue storage flags query Technical help , dask , climate-conda-enviro	22	121	25 September 2025
Using dask_jobqueue in the new xp65 environment Technical help , climate-conda-enviro	10	148	4 June 2025
'NetCDF: Not a valid ID' errors Technical	26	3397	29 July 2024