Hh5 scratch and analysis3 environment

Technical Infrastructure
, ,
1

I am trying to pip install packages on my user in the analysis3 environment on hh5. However, I get errors relating to disk quota. When looking at the at the hh5 scratch, I noticed that it seems to be full. Would this be the reason for this? If so, understanding that ACCESS-NRI is taking over the analysis3 environment, would it be possible to clear out some of this space?

--------------------SUMMARIES--------------------
FILESYSTEM GROUP SPACE USED TOTAL SIZE COUNT
scratch hh5 44.8T 44.8T 129454

2

Hi @Michael_Barnes,
Thanks for your message, this is good to highlight to the rest of hh5 users and a message should probably sent through nci.
However, ACCESS-NRI will be managing the conda environments, but not the project itself, data, storage, this is noted in project description here: hh5.

3

You shouldn’t need to create anything in /scratch/hh5.

The central conda environment isn’t writable, you need to make sure you add the --user flag when doing a pip install.

(for whoever’s taking over the modules, it may be worth having the module set environment variable PIP_TARGET which should change where pip installs things to)

1 Like
4

Hi Scott and Felicity.
All good to know, thanks! @Scott,I do pip with --user, but the default for some reason when pulling the files is to hh5. I have also tried using --target but no such luck. How does one change their PIP_TARGET location?

5

Make sure your home directory has space (quota -s)

Something like this would create separate pip targets for each conda environment to avoid conflicts

export PIP_TARGET=/scratch/$PROJECT/$USER/tmp/pip/$CONDA_DEFAULT_ENV
export PYTHONPATH=$PIP_TARGET:$PYTHONPATH

python -m pip install -e .
6

Thanks! I will give this a go.