Using dask_jobqueue in the new xp65 environment

Dear NRI,

I use dask_jobqueue to set up bespoke clusters within a jupyter lab notebook. I don’t use the ARE but a script to start the notebook, which seems to work fine with the new xp65 env.

My problem occurs when submitting the dask_jobqueue to set up a cluster, which I do like this:

from dask.distributed import Client,LocalCluster
from dask_jobqueue import PBSCluster

cluster=PBSCluster(cores=28, #normalbw
processes=28,
memory=‘256GB’, #normalbw
queue=‘normalbw’,
walltime=‘08:00:00’,
local_directory=‘$TMPDIR’,
resource_spec=‘ncpus=28,mem=256GB,jobfs=200GB’, #normalbw

               job_extra_directives=['-P mn51','-lstorage=gdata/mn51+gdata/xp65+gdata/er4+gdata/eg3+gdata/ia39'])

cluster.scale(jobs=1)
client=Client(cluster)

The jobs get submitted but immediately end on connection.

Has anyone got any ideas?

Cheers,
Justin

You need to manually set the path to python when creating the cluster

import dask.config
from dask.distributed import Client,LocalCluster
from dask_jobqueue import PBSCluster
walltime = '01:00:00'
cores = 48
memory = '192GB'
 
cluster = PBSCluster(walltime=str(walltime), cores=cores, memory=str(memory), processes=cores,
                     job_extra=['-l ncpus='+str(cores),
                                '-l mem='+str(memory),
                                '-l storage=gdata/xp65'],
                     local_directory='$TMPDIR',
                     header_skip=["select"],
                     python="/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python")
cluster.scale(jobs=2)

client = Client(cluster)
display(client)

@access-nri folks, the path to python would be handy to have as an environment variable from the module, rather than needing to set the version number here explicitly

1 Like

@jpeter Can you please confirm @Scott 's answer solve your issue so we can show this topic as solved and we can close it?

@Scott :

I’ll pass along.

1 Like

Hi Claire,

Just confirming that Scott’s solution worked.

Cheers,
Justin

1 Like

It may be possible to configure the python to launch by setting up config files inside the container? maybe (not tested)

# $CONDA_PREFIX/etc/dask/jobqueue.yaml
jobqueue:
  pbs:
    python: "/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python"

Configure Dask-Jobqueue — Dask-jobqueue 0.9.0+8.g6dc463d documentation
Configuration — Dask documentation