# Using dask\_jobqueue in the new xp65 environment
**URL:** https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451
**Category:** Technical
**Tags:** help, climate-conda-enviro
**Created:** [30 April 2025 02:39 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451 "2025-04-30T02:39:03Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@jpeter](https://forum.access-hive.org.au/u/jpeter)
#### Post date: [30 April 2025 02:39 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/1 "2025-04-30T02:39:03Z")
Dear NRI,
I use dask\_jobqueue to set up bespoke clusters within a jupyter lab notebook. I don’t use the ARE but a script to start the notebook, which seems to work fine with the new xp65 env.
My problem occurs when submitting the dask\_jobqueue to set up a cluster, which I do like this:
```python
from dask.distributed import Client,LocalCluster
from dask_jobqueue import PBSCluster
cluster=PBSCluster(cores=28, #normalbw
processes=28,
memory='256GB', #normalbw
queue='normalbw',
walltime='08:00:00',
local_directory='$TMPDIR',
resource_spec='ncpus=28,mem=256GB,jobfs=200GB', #normalbw
job_extra_directives=['-P mn51','-lstorage=gdata/mn51+gdata/xp65+gdata/er4+gdata/eg3+gdata/ia39'])
cluster.scale(jobs=1)
client=Client(cluster)
```
The jobs get submitted but immediately end on connection.
Has anyone got any ideas?
Cheers,
Justin
---
### Author:  [@Scott](https://forum.access-hive.org.au/u/Scott)
#### Post date: [30 April 2025 05:00 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/2 "2025-04-30T05:00:36Z")
You need to manually set the path to python when creating the cluster
```auto
import dask.config
from dask.distributed import Client,LocalCluster
from dask_jobqueue import PBSCluster
walltime = '01:00:00'
cores = 48
memory = '192GB'
cluster = PBSCluster(walltime=str(walltime), cores=cores, memory=str(memory), processes=cores,
job_extra=['-l ncpus='+str(cores),
'-l mem='+str(memory),
'-l storage=gdata/xp65'],
local_directory='$TMPDIR',
header_skip=["select"],
python="/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python")
cluster.scale(jobs=2)
client = Client(cluster)
display(client)
```
[@access-nri](https://forum.access-hive.org.au/groups/access-nri) folks, the path to python would be handy to have as an environment variable from the module, rather than needing to set the version number here explicitly
---
### Author:  [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)
#### Post date: [30 April 2025 06:12 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/3 "2025-04-30T06:12:17Z")
@jpeter Can you please confirm @Scott 's answer solve your issue so we can show this topic as solved and we can close it?
@Scott :
> [@Scott](#):
>
> [@access-nri](https://forum.access-hive.org.au/groups/access-nri) folks, the path to python would be handy to have as an environment variable from the module, rather than needing to set the version number here explicitly
I’ll pass along.
---
### Author:  [@jpeter](https://forum.access-hive.org.au/u/jpeter)
#### Post date: [1 May 2025 06:25 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/4 "2025-05-01T06:25:05Z")
> [@Scott](#):
>
> ```auto
> header_skip=["select"],
> python="/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python")
>
> ```
Hi Claire,
Just confirming that Scott’s solution worked.
Cheers,
Justin
---
### Author:  [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)
#### Post date: [1 May 2025 06:27 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/5 "2025-05-01T06:27:05Z")
---
### Author:  [@Scott](https://forum.access-hive.org.au/u/Scott)
#### Post date: [2 May 2025 00:52 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/6 "2025-05-02T00:52:28Z")
It may be possible to configure the python to launch by setting up config files inside the container? maybe (not tested)
```auto
# $CONDA_PREFIX/etc/dask/jobqueue.yaml
jobqueue:
pbs:
python: "/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.03.d/bin/python"
```
[Configure Dask-Jobqueue — Dask-jobqueue 0.9.0+8.g6dc463d documentation](https://jobqueue.dask.org/en/latest/clusters-configuration-setup.html)
[Configuration — Dask documentation](https://docs.dask.org/en/latest/configuration.html)
---
### Author:  [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)
#### Post date: [4 June 2025 22:26 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/7 "2025-06-04T22:26:07Z")
---
### Author:  [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)
#### Post date: [4 June 2025 22:28 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/8 "2025-06-04T22:28:06Z")
Hi all,
Based on @Scott suggestion, I have configured the DASK\_JOBQUEUE\_\_PBS\_\_PYTHON environment variable to point to the Python interpreter in the container. That should fix your problem @jpeter. It is available for all versions of the conda/analysis3 environment
---
### Author:  [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)
#### Post date: [4 June 2025 22:28 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/9 "2025-06-04T22:28:17Z")
---
### Author:  [@Scott](https://forum.access-hive.org.au/u/Scott)
#### Post date: [4 June 2025 23:41 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/10 "2025-06-04T23:41:23Z")
The change seems to have not gotten picked up
```auto
$ which python # Should be used for jobqueue.pbs.python
/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.04.d/bin/python
$ python -c 'import dask; import dask_jobqueue; print(dask.config.get("jobqueue.pbs.python"))'
None
```
---
### Author:  [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)
#### Post date: [4 June 2025 23:57 UTC](https://forum.access-hive.org.au/t/using-dask-jobqueue-in-the-new-xp65-environment/4451/11 "2025-06-04T23:57:53Z")
I overwrote the changes by mistake. Try now
```bash
Loading conda/analysis3-25.04
Loading requirement: singularity
[rb5533@gadi-login-08 ~]$ python -c 'import dask; import dask_jobqueue; print(dask.config.get("jobqueue.pbs.python"))'
/g/data/xp65/public/apps/med_conda_scripts/analysis3-25.04.d/bin/python
```