Issue with custom conda env (built on top of xp65) in PBS job

Technical
, , ,
1

Hi,

I have a (likely simple) question that someone may be able to assist with. I have created my own custom python environment built on top of xp65 following the CLEX blog, but having issues with this working in a PBS job.

My custom env was built at the time of analysis3-25.10 being the default:

cd ~/conda_envs/
module use /g/data/xp65/public/modules
module load conda/analysis3
python3 -m venv heatwatch --system-site-packages
source heatwatch/bin/activate
pip install git+https://github.com/CenterForTheBuiltEnvironment/pythermalcomfort.git
pip install earthkit-meteo

My PBS script:

#!/bin/bash
#PBS -N heatwatch
#PBS -P mn51
#PBS -q normalbw
#PBS -l walltime=1:00:00
#PBS -l mem=256GB
#PBS -l ncpus=28
#PBS -l storage=gdata/mn51+gdata/xp65
#PBS -l jobfs=400GB

module use /g/data/xp65/public/modules
module load conda/analysis3-25.10
source /home/563/mtb563/conda_envs/heatwatch/bin/activate

which python
~/conda_envs/heatwatch/bin/python /g/data/mn51/users/mtb563/repos_collab/heatwatch/analysis/calc_heatwatch.py

This returns:

(which python) : /g/data/xp65/public/apps/med_conda_scripts/analysis3-25.10.d/bin/python

and the error message:

Loading conda/analysis3-25.10
  Loading requirement: singularity
/local/spool/pbs/mom_priv/jobs/157583170.gadi-pbs.SC: line 18: /home/563/mtb563/conda_envs/heatwatch/bin/python: No such file or directory

It cannot find the version of python within my heatwatch conda environment (which ultimately points to:

image

I am guessing this is a container related issue. Any thoughts on what I am doing wrong or need to do in addition? I feel like I am missing something simple.

Thanks for you help.

2

Strange, that’s supposed to just work - for the NRI folks I believe you want Python’s sys.executable to point to the python wrapper script rather than the executable in the container so that venvs function properly.

As a workaround to get you going, prepend a command with launcher.sh to explicitly run it in the container, e.g.

launcher.sh ~/conda_envs/heatwatch/bin/python /g/data/mn51/users/mtb563/repos_collab/heatwatch/analysis/calc_heatwatch.py
2 Likes
3

Thanks very much @Scott, this worked! I will hold off marking this as solved until the NRI team are back and had a chance to see your suggestion.