I am trying to use my own virtual environment built on top of analysis3 following this guide. I can use the venv ok in the terminal or vscode but I cannot get it to appear in my kernel list in ARE’s JupyterLab.
I’ve done all the ipykernel install stuff, and the kernel appears when I open a jupyter session on the terminal and open it via ssh tunnel, it’s just ARE that refuses to work. Any guidance on how to troubleshoot this issue?
How are you launching jupyter? You need the jupyter that ARE launches to be the version in your virtual environment. Are you setting virtual environment base?
I am loading conda/analysis3 in the modules section, which is the same base I use for my environment. If I do `jupyter kernelspec list` on a terminal in JupyterLab, the kernel is listed, but it doesn’t appear in the list when launching a new file or when selecting the kernel of an existing notebook.
Are the kernels you’ve created showing up in ~/.local/share/jupyter/kernels/?
Yep! They’re all there.
Weird…
Can you post the output of tree -d ~/.local for me? I have a feeling that you might have nb_conda_kernels configured in a way that is overwriting your config.
Alternatively, depending how brave reckless you’re feeling, you could delete the entirety of ~/.local and try again. I ran into this issue once and that fixed it for me - again, I think nb_conda_kernels related.
I’ve got like a million things in .local. I limited it to tree levels deep; I hope it’s helpful. I wouldn’t like to nuke .local for the same reasons, there’s a bunch of other tools there.
local_tree.txt (17.2 KB)
Can you move it to eg. ~/.local_backup temporarily? You have site packages for python 3.{6,10,11,12} in there (~/.local/lib/python3.XX/site-packages) . Your ~/.local/share/jupyter is also full of conda environments, specifying a bunch of the conda environments that should be brought in ‘for free’ by the analysis3 module configuration.
It’s a little harder for me to say any more than that because of the amount of stuff in there. If you want to preserve the nvim/uv/zed configs, you could probably just delete and/or rename ~/.local/lib/share/python3.*/site-packages and ~/.local/share/jupyter which I suspect will fix the problem - although you’ll need to rebuild the env (following the CLEX instructions should work fine).
Fair enough. I nuked .local after backup and installed the venv again. Now my .local looks like this
/home/565/ec0044/.local
└── share
├── direnv
│ └── allow
└── jupyter
└── kernels
└── venv_models
(After I open ARE for the first time all the other analysis3 modules appear there).
JupyterLab in ARE still is not finding venv_models
even though it appears in jupyter kernelspec list. 
Weird… I’m surprised that didn’t fix it. Can you post your ARE session config here? I wonder if there’s something in that causing problems.
For reference, mine is:
Cluster: ncigadi
Walltime (hours): 4
Queue: normalbw
Compute Size: large
Project: tm70
Storage: gdata/xp65+gdata/fs38+gdata/oi10+gdata/al33+gdata/rr3+gdata/rt52+gdata/zz93+gdata/ct11+gdata/zv30+gdata/ik11+gdata/zz63+gdata/ig45+gdata/py18+gdata/hq89+gdata/cj50+gdata/lg87+gdata/tm70+gdata/ai05+gdata/ia39+gdata/kj66+gdata/mn51+gdata/qv56+gdata/ol01+gdata/e14+gdata/p73+gdata/x77+gdata/av17+gdata/g40
Software:
Settings:
Show advanced settings: 1
Extra arguments:
Module directories: /g/data/xp65/public/modules
Modules: openmpi conda/analysis3
Python or Conda virtual environment base:
Conda environment:
Environment variables:
Jobfs size: 10GB
PBS flags:
Pre-script:
I would like to receive an email when the session starts: 0
Yes, it’s strange. It’s also strange that with a clean .local, the first time I start an ARE session, it adds all those conda environments to my .local folder.
This is my ARE configuration:
{
"cluster": "ncigadi",
"bc_num_hours": "4",
"queue": "normalbw",
"compute_size": "small",
"project": "nf33",
"storage": "gdata/xp65+gdata/nf33+gdata/gb02",
"software": "",
"settings": "",
"adv_checkbox": "1",
"extra_jupyter_args_adv": "",
"data_modules_use_adv": "/g/data/xp65/public/modules",
"data_modules_adv": "conda/analysis3-26.04 cdo/2.4.3",
"python_env_adv": "",
"conda_env_adv": "",
"envvars_adv": "",
"jobfs_adv": "",
"pbsflags_adv": "",
"prescript_adv": "",
"bc_email_on_started": "0"
}
Can you post your .bash_profile and .bashrc? I remember we had some strange issues a while ago where conda had written something into a bash config file, I wonder if it might be a similar problem here.
I have a couple of utilities and aliases, but nothing outlandish, I don’t think.
# .bashrc
# Source global definitions (Required for modules)
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
prepend_path PATH ${HOME}/bin
prepend_path PATH ${HOME}/.local/bin
prepend_path PATH ${HOME}/local/go/bin
if in_interactive_shell; then
# This is where you put settings that you'd like in
# interactive shells. E.g. prompts, or aliases
# The 'module' command offers path manipulation that
# will only modify the path if the entry to be added
# is not already present. Use these functions instead of e.g.
# PATH=${HOME}/bin:$PATH
export EDITOR="vim"
export PS1='${debian_chroot:+($debian_chroot)}:\[\033[01;34m\]\w\[\033[00m\] \n ↳ '
flag=$((1 + $RANDOM % 3))
if [ $flag -eq 1 ]; then
export PS1="$(prideprompt -f agender) $PS1"
elif [ $flag -eq 2 ]; then
export PS1="$(prideprompt -f nonbinary) $PS1"
elif [ $flag -eq 3 ]; then
export PS1="$(prideprompt -f rainbow) $PS1"
fi
export PS1="\t $PS1"
export PS1="\`if [ \$? = 0 ]; then echo \[\e[33m\]^_^\[\e[0m\]; else echo \[\e[31m\]O_O\[\e[0m\]; fi\` $PS1"
alias use_cylc="module use /g/data/hr22/modulefiles && module load cylc7/23.09"
alias use_xconv="module use /g/data/access/modules && module load xconv"
alias use_payu="module use /g/data/vk83/modules && module load payu"
module use /g/data/vk83/modules
module load system-tools/gh
nfiles () {
find "$1" -type f -printf "."| wc -c
}
alias files="spf"
alias cdroot='cd $(root)'
function golast () {
wd=$(head -n 1 ~/.last_project)
cd $wd
}
if in_login_shell; then
# This is where you place things that should only
# run when you login. If you'd like to run a
# command that displays the status of something, or
# load a module, or change directory, this is the
# place to put it
# module load pbs
# module load intel-mkl intel-compiler
# module restore R_4.3.1
# module load cdo/2.4.3
module load ncview
# cd /scratch/${PROJECT}/${USER}
fi
fi
prepend_path PATH /home/565/ec0044/bin/lib
prepend_path LD_LIBRARY_PATH /home/565/ec0044/bin/lib
prepend_path LD_RUN_PATH /home/565/ec0044/bin/lib
prepend_path PKG_CONFIG_PATH /home/565/ec0044/bin/lib/pkgconfig/
prepend_path INCLUDE_DIR /home/565/ec0044/bin/include
prepend_path LIB_DIR /home/565/ec0044/bin/lib
eval "$(direnv hook bash)"
# Anything here will run whenever a new shell is launched, which
# includes when running commands like 'less'. Commands that
# produce output should not be placed in this section.
#
# If you need different behaviour depending on what machine you're
# using to connect to Gadi, you can use the following test:
#
# if [[ $SSH_CLIENT =~ 11.22.33.44 ]]; then
# Do something when I connect from the IP 11.22.33.44
# fi
#
# If you want different behaviour when entering a PBS job (e.g.
# a default set of modules), test on the $in_pbs_job variable.
# This will run when any new shell is launched in a PBS job,
# so it should not produce output
#
# if in_pbs_job; then
# module load openmpi/4.0.1
# fi
. "$HOME/.cargo/env"
RENV_PATHS_CACHE=/g/data/lo70/ec0044/renv_cache/
# BEGIN pbs-workbench
export PATH="/home/565/ec0044/.local/bin/pbs-workbench:$PATH"
[ -f "/home/565/ec0044/.local/bin/pbs-workbench/completion.sh" ] && source "/home/565/ec0044/.local/bin/pbs-workbench/completion.sh"
# END pbs-workbench
source acd_func.sh
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
. "$HOME/.cargo/env"
I’m running out of ideas here to be honest…
Can you try jupyter --paths post the results in here? Hopefully that turfs up something weird.
If jupyter kernelspec list can see the environments, but jupyter lab can’t, then it must be the case that it’s somehow not pointing at the same configuration. In practice, I’m not really sure how we’d get into that state.
Thanks for taking the time anyway! Very much appreciated.
This is jupyter --paths
config:
/home/565/ec0044/.jupyter
/home/565/ec0044/.local/etc/jupyter
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.04/etc/jupyter
/usr/local/etc/jupyter
/etc/jupyter
data:
/home/565/ec0044/.local/share/jupyter
/g/data/xp65/public/apps/med_conda/envs/analysis3-26.04/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/home/565/ec0044/.local/share/jupyter/runtime
Okay, I’m now totally stumped, because that’s exactly the same as mine (save for being your homedir, not mine).
Let’s try something a bit different:
curl -fsSL https://pixi.sh/install.sh | sh cat 'export PIXI_CACHE_DIR="/scratch/tm70/ct1163/pixi_cache"' >> .bashrc$ echo "detached-environments = true" >> ~/.pixi/config.tomlcd $DIR && pixi init && pixi add jupyterlabpixi shell && python -m ipykernel install --user --name pixi-demo --display-name "pixi-demo"Hopefully, this will show up in your kernels at this point.
This will create a lot of inodes. Run
pixi clean cachewhen you’re done with it to delete them all. Because of the lockfile, you can come back and reinstall all the exact same package versions later.
Let me know if this works!
At this point, I would seriously consider removing any customisations from your config and reintroduce them one by one - there must be something that’s introduced a weird and subtle bug.
Sorry I can’t be any more help than that!
Which config do you mean?
Literally anything nonstandard in your bash profile & bashrc, as well as moving .local somewhere it won’t get pulled in.
So eg.
$ mv .bash_profile ._bash_profile
$ mv .bashrc ._bashrc
$ mv .local ._local
and then trying again. You’ll need to get the default bash profile & rc from somewhere I guess (I don’t know where, sorry!). You could also just comment anything you added to those files out temporarily I guess…