Using dask_jobqueue in the new xp65 environment

Technical
,
1

Dear NRI,

I use dask_jobqueue to set up bespoke clusters within a jupyter lab notebook. I don’t use the ARE but a script to start the notebook, which seems to work fine with the new xp65 env.

My problem occurs when submitting the dask_jobqueue to set up a cluster, which I do like this:

from dask.distributed import Client,LocalCluster
from dask_jobqueue import PBSCluster

cluster=PBSCluster(cores=28, #normalbw
                   processes=28,
                   memory='256GB', #normalbw
                   queue='normalbw',
                   walltime='08:00:00',
                   local_directory='$TMPDIR',
                   resource_spec='ncpus=28,mem=256GB,jobfs=200GB', #normalbw

                   job_extra_directives=['-P mn51','-lstorage=gdata/mn51+gdata/xp65+gdata/er4+gdata/eg3+gdata/ia39'])

cluster.scale(jobs=1)
client=Client(cluster)

The jobs get submitted but immediately end on connection.

Has anyone got any ideas?

Cheers,
Justin

Dask bag crashing in xp65 env (but wasn't in hh5)
Analysis3 dask_jobqueue storage flags query
2

You need to manually set the path to python when creating the cluster

import dask.config
from dask.distributed import Client,LocalCluster
from dask_jobqueue import PBSCluster
walltime = '01:00:00'
cores = 48
memory = '192GB'
 
cluster = PBSCluster(walltime=str(walltime), cores=cores, memory=str(memory), processes=cores,
                     job_extra=['-l ncpus='+str(cores),
                                '-l mem='+str(memory),
                                '-l storage=gdata/xp65'],
                     local_directory='$TMPDIR',
                     header_skip=["select"],
                     python="/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python")
cluster.scale(jobs=2)

client = Client(cluster)
display(client)

@access-nri folks, the path to python would be handy to have as an environment variable from the module, rather than needing to set the version number here explicitly

1 Like
3

@jpeter Can you please confirm @Scott 's answer solve your issue so we can show this topic as solved and we can close it?

@Scott :

I’ll pass along.

1 Like
Xp65 on ARE VDI - unable to call on /opt/conda/*
4

Hi Claire,

Just confirming that Scott’s solution worked.

Cheers,
Justin

1 Like
6

It may be possible to configure the python to launch by setting up config files inside the container? maybe (not tested)

# $CONDA_PREFIX/etc/dask/jobqueue.yaml
jobqueue:
  pbs:
    python: "/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python"

Configure Dask-Jobqueue — Dask-jobqueue 0.9.0+8.g6dc463d documentation
Configuration — Dask documentation

1 Like
8

Hi all,

Based on @Scott suggestion, I have configured the DASK_JOBQUEUE__PBS__PYTHON environment variable to point to the Python interpreter in the container. That should fix your problem @jpeter. It is available for all versions of the conda/analysis3 environment

2 Likes
10

The change seems to have not gotten picked up

$ which python # Should be used for jobqueue.pbs.python
/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.04.d/bin/python
$ python -c 'import dask; import dask_jobqueue; print(dask.config.get("jobqueue.pbs.python"))'
None
11

I overwrote the changes by mistake. Try now

Loading conda/analysis3-25.04
  Loading requirement: singularity
[rb5533@gadi-login-08 ~]$ python -c 'import dask; import dask_jobqueue; print(dask.config.get("jobqueue.pbs.python"))'
/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.04.d/bin/python
2 Likes