Dear NRI help, I am using dask bag to read multiple files in parallel. This code was working on the hh5 environment, however, crashes on the xp65 env. Here are the pertinent fragments of code: from dask import compute, delayed import dask.bag as db #Create key:value pairs of keys and directory p…

Hi Anton, It appears that was a problem, however, the problem still occurred when I reran it. I found out that the correct kernel was not being loaded by my notebook. It prompted me for the kernel I wished to use from a pop-up menu. I loaded it, but it appears that it wasn’t loaded appropriately. …

Hi Justin Which module exactly are you loading (e.g. conda/analysis3-25.03 ?) ? [image] jpeter: on 4 different workers, but all those workers died while running it. This error message is a flow-on effect from the operation failing. Are their other error messages? e.g. maybe there is a log …

I ran this test code : import xarray as xr import glob from dask.distributed import Client client = Client(threads_per_worker = 1) client infiles = sorted(glob.glob('/g/data/ik11/outputs/access-om2/1deg_era5_ryf/output00*/ice/OUTPUT/iceh*-daily.nc')) ds = xr.open_mfdataset(infiles, parallel=Tru…

Oh - never mind my comments about starting the dask cluster - i see you have that handled ( Using dask_jobqueue in the new xp65 environment )

Not sure I can add much useful in the way of debugging steps beyond what @anton has already suggested, but I would also caution against doing anything like this: @dask.delayed def foo(): xr.open_mfdataset() Whilst xarray passes around locks to (attempt to) ensure that files are opened in a thr…

@ag0498 , I hope you don’t mind I’ve move your question into it’s own topic here: Using dask jobqueue with xp65 conda env .

[image] anton: e.g. can you make a minimal-example of what fails. I tried using the analysis-25.02 env, however, the same thing happened. I use the dask decorator on top of xarray.open_mfdataset as it enables me to read all the combinations of input files quickly and associate key:value pair…

Hi Anton, Note that this is using analysis-25.02 as suggested by @CharlesTurner Even if I use the following: test_dir = 'path_to_nc_files' infiles = sorted(glob.glob(test_dir+"/*.nc")) value = xr.open_mfdataset(infiles,chunks={'time':-1},parallel=True) I get the same error message. However, if I…

Thanks Justin I assume you are using PBSCluster ? There should be a dask-worker.e* file in the directory you are running python from. Could you have a look and see what it says please

Hi Anton, I have included portions of the dask-worker.e* file below. Note that this is using analysis-25.02 and using the minimal example above (i.e. only using xr.open_mfdatasest without dask bag, so only reading the files in one directory. Lots of lines starting the workers: 2025-05-09 11:24:08…

Signal 11 is normally a system termination, e.g. because too much memory or jobfs was used. What is the PBS footer in the dask-worker.o* file?

Dask bag crashing in xp65 env (but wasn't in hh5)

Technical Infrastructure

Topic		Replies	Views
Using dask jobqueue with xp65 conda env Technical python , help , conda , dask	17	232	19 May 2025
Dask is crashing: ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found Technical help , dask , are , jupyterlab , unresolved	17	158	4 February 2026
Analysis3 dask_jobqueue storage flags query Technical help , dask , climate-conda-enviro	22	183	25 September 2025
Cannot print to file - possible Dask issue in xp65 General help	16	136	14 May 2025
'NetCDF: Not a valid ID' errors Technical	26	3544	29 July 2024

Dask bag crashing in xp65 env (but wasn't in hh5)

Related topics