Using gadi_jupyter.sh script with new xp65 conda environment

Technical
, , ,
1

I know we’re supposed to use ARE, but… many of us still prefer and are using the old gadi_jupyter script.

I’m struggling to get it to work with the xp65 conda environment instead of the hh5 one. I naively tried replacing this line:
eval "\\\$(/g/data/hh5/public/apps/miniconda3/bin/conda shell.bash hook)"
with
eval "\\\$(/g/data/xp65/public/apps/med_conda/bin/conda shell.bash hook)"
and it did not work.

Any tips?

2 Likes
2

Let’s just say that I’m one of the gadi_jupyter.sh script users. It’s not just “a preference” as in the spirit eg “I’m old school and like how things use to be”-type of preference. For me, my preference stems from the fact that I find gadi_jupyter.sh to be much much less flaky than ARE sessions.

Although I do not expect official support for gadi_jupyter.sh, if anybody knows a 1-2 line edits so that gadi_jupyter.sh works with the conda environment on xp65 it would be so greatly appreciated!

1 Like
3

Try removing the lines

eval "\\\$(/g/data/hh5/public/apps/miniconda3/bin/conda shell.bash hook)"
conda activate "${CONDA_ENV}"

and change

$STRACE jupyter lab --NotebookApp.token="\\\$TOKEN" --no-browser --ip="\\\$HOSTNAME" --port "\\\$PORT" --port-retries=0

to use the full path to jupyter (e.g. by running which jupyter after loading the xp65 module)

1 Like
4

so just remove the

eval "\\\$(/g/data/hh5/public/apps/miniconda3/bin/conda shell.bash hook)"
conda activate "${CONDA_ENV}"

and not replace with something related to the g/data/x65/public etc?

5

Yeah, instead the jupyter line will be /g/data/xp65/whatever/jupyter - hopefully the command is set up to load the environment properly

6

Or maybe you could change the conda activate to a module load? that also might be worth trying

7

I agree with ARE being very flaky and clunky for many reasons, including:

  • Substantial overhead in starting a session (compared to running a bash script)
  • Having to start a session from a web server (instead of Gadi)
  • The terminal on the ARE VDI instance is painful to use

In addition to trying to use the old gadi_jupyter.sh (as a temporary solution), I think there might be good ground for requesting a better alternative to starting ARE from the web server.
For example, a way to start an ARE instance directly from Gadi would already solve most of the issues in my opinion.

3 Likes
8

@atteggiani
not to negate any of those points, but the terminal on an ARE jupyter-lab session is reasonable, so possibly an alternative to VDI if you’re just after a terminal session

9

There are some instances in which you cannot use a Jupyterlab (instead of a VDI) instance because some GUI are needed.

Also, in general, I think it would be beneficial to everyone having an officially maintained tool that enables desired workflows, without having to apply custom patches and workarounds.

1 Like
10

I’ve tried implementing @Scott’s suggestions above, but no luck so far.

11

I’m not currently a project member so am a bit blind unfortunately. There should be a log file at /scratch/$PROJECT/$USER/tmp/runjp/pbs.log which could have error messages?

12

I needed to implement both of @Scott’s suggestions (module load and full path to jupyter).
PR here: Change hh5 to xp65 conda by adele-morrison · Pull Request #44 · coecms/nci_scripts · GitHub

Thanks @Scott!!

@Aidan I don’t seem to be able to list 2 comments as solutions, it will only let me list one?

2 Likes
13

@adele-morrison
I marked your comment as a solution, as it outlines the two solutions that you applied and worked.

14

Yeah you can only choose one solution.

If it’s important quote both parts in a reply and mark that as the solution.

I’m afraid that is as good as it gets.