Evening all.
The World Climate Research Program 2025 Hackathon is trying to build a temporary python environment to get around the current disk issues on gadi.
We want to build a python environment on /scratch/nf33
which can then be used in ARE sessions.
The .yaml
file is available from tools/python_envs/environment.yaml at main · digital-earths-global-hackathon/tools · GitHub
So I downloaded and installed miniconda (because I can’t rely on existing hh5/xp65 /g/data
conda installs) at this location
/scratch/nf33/public/miniconda
and ran
conda env create --name digital_earths_env --file environment.yaml
Which created an environment in
/scratch/nf33/public/miniconda/envs/digital_earths_env
This environment can be activated from the command line and the requisite modules imported.
Now, the trick part to loading it into ARE.
The NRI ARE docs state:
“Python or Conda virtual environment base
Path to a Python or conda base environment to be activated for the JupyterLab session.
It is the equivalent to running source <path/to/environment/bin/activate>
on the command line.”
The problem is, this <environment>/bin/activate
file doesn’t exist. So my ARE sessions can’t find any kernels associated with the environment.
This is a known issue with new conda, see
opened 02:47PM - 04 Mar 18 UTC
closed 03:25PM - 04 Mar 18 UTC
cli
locked
**I'm submitting a...**
- [X] bug report
- [ ] feature request
## Cur… rent Behavior
`conda create` no longer creates a symlink to `$_CONDA_ROOT/bin/activate` for the new environment.
It seems to have been some changes to activation in `conda` which may be the reason for this. Causes jorgenschaefer/pyvenv#67
### Steps to Reproduce
```
conda create -n venv python==3.6
ls $_CONDA_ROOT/envs/venv/bin/activate
```
Output
```
ls: cannot access '/home/kevin/anaconda/envs/venv/bin/activate': No such file or directory
```
## Expected Behavior
A symlink to `$_CONDA_ROOT/bin/activate` to be created
## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
```
active environment : None
shell level : 0
user config file : /home/kevin/.condarc
populated config files :
conda version : 4.4.10
conda-build version : 3.0.23
python version : 3.6.2.final.0
base environment : /home/kevin/anaconda (writable)
channel URLs : https://repo.continuum.io/pkgs/main/linux-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/linux-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/linux-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/linux-64
https://repo.continuum.io/pkgs/pro/noarch
package cache : /home/kevin/anaconda/pkgs
/home/kevin/.conda/pkgs
envs directories : /home/kevin/anaconda/envs
/home/kevin/.conda/envs
platform : linux-64
user-agent : conda/4.4.10 requests/2.18.4 CPython/3.6.2 Linux/4.15.6-1-ARCH arch/ glibc/2.26
UID:GID : 1000:1000
netrc file : /home/kevin/.netrc
offline mode : False
```
</p></details>
<details open><summary><code>`conda config --show-sources`</code></summary><p>
Nothing is returned
```
```
</p></details>
<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
```
# packages in environment at /home/kevin/anaconda:
#
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py36he11e457_0 defaults
_nb_ext_conf 0.4.0 py36_1 defaults
airflow 1.8.0 <pip>
alabaster 0.7.10 py36h306e16b_0 defaults
alembic 0.8.10 <pip>
anaconda custom py36hbbc8b67_0 defaults
anaconda-client 1.6.5 py36h19c0dcd_0 defaults
anaconda-navigator 1.6.8 py36h672ccc7_0 defaults
anaconda-project 0.8.0 py36h29abdf5_0 defaults
asn1crypto 0.22.0 py36h265ca7c_1 defaults
astroid 1.5.3 py36hbdb9df2_0 defaults
astropy 2.0.2 py36ha51211e_4 defaults
babel 2.5.0 py36h7d14adf_0 defaults
backports 1.0 py36hfa02d7e_1 defaults
backports.shutil_get_terminal_size 1.0.0 py36hfea85ff_2 defaults
basemap 1.0.7 np113py36_0 defaults
beautifulsoup4 4.6.0 py36h49b8c8c_1 defaults
bitarray 0.8.1 py36h5834eb8_0 defaults
bkcharts 0.2 py36h735825a_0 defaults
blaze 0.11.3 py36h4e06776_0 defaults
bleach 2.0.0 py36h688b259_0 defaults
bokeh 0.12.7 py36h169c5fd_1 defaults
boto 2.48.0 py36h6e4cd66_1 defaults
bottleneck 1.2.1 py36haac1ea0_0 defaults
ca-certificates 2017.08.26 h1d4fec5_0 defaults
cairo 1.14.10 h58b644b_4 defaults
certifi 2018.1.18 py36_0 defaults
cffi 1.10.0 py36had8d393_1 defaults
chardet 3.0.4 py36h0f667ec_1 defaults
click 6.7 py36h5253387_0 defaults
click-plugins 1.0.3 <pip>
cligj 0.4.0 <pip>
cloudpickle 0.4.0 py36h30f8c20_0 defaults
clyent 1.2.2 py36h7e57e65_1 defaults
colorama 0.3.9 py36h489cec4_0 defaults
conda 4.4.10 py36_0 defaults
conda-build 3.0.23 py36h1da9bb0_0 defaults
conda-env 2.6.0 h36134e3_1 defaults
conda-verify 2.0.0 py36h98955d8_0 defaults
contextlib2 0.5.5 py36h6c84a62_0 defaults
croniter 0.3.19 <pip>
cryptography 2.0.3 py36ha225213_1 defaults
curl 7.55.1 hcb0b314_2 defaults
cycler 0.10.0 py36h93f1223_0 defaults
cython 0.26.1 py36h21c49d0_0 defaults
cytoolz 0.8.2 py36h708bfd4_0 defaults
dask 0.15.2 py36h9b48dc4_0 defaults
dask-core 0.15.2 py36h0f988a8_0 defaults
datashape 0.5.4 py36h3ad6b5c_0 defaults
dbus 1.10.22 h3b5a359_0 defaults
decorator 4.1.2 py36hd076ac8_0 defaults
descartes 1.1.0 <pip>
dill 0.2.7.1 <pip>
distributed 1.18.3 py36h73cd4ae_0 defaults
docutils 0.14 py36hb0f60f5_0 defaults
entrypoints 0.2.3 py36h1aec115_2 defaults
et_xmlfile 1.0.1 py36hd6bccc3_0 defaults
expat 2.2.4 hc00ebd1_1 defaults
fastcache 1.0.2 py36h5b0c431_0 defaults
fastkml 0.11 <pip>
filelock 2.0.12 py36hacfa1f5_0 defaults
Fiona 1.7.11 <pip>
flake8 3.5.0 <pip>
flask 0.12.2 py36hb24657c_0 defaults
Flask 0.11.1 <pip>
Flask-Admin 1.4.1 <pip>
Flask-Cache 0.13.1 <pip>
flask-cors 3.0.3 py36h2d857d3_0 defaults
Flask-Login 0.2.11 <pip>
flask-swagger 0.2.13 <pip>
Flask-WTF 0.12 <pip>
fontconfig 2.12.4 h88586e7_1 defaults
freetype 2.8 h52ed37b_0 defaults
funcsigs 1.0.0 <pip>
future 0.15.2 <pip>
geojson 2.3.0 <pip>
geopandas 0.3.0 <pip>
geos 3.5.0 0 defaults
get_terminal_size 1.0.0 haa9412d_0 defaults
gevent 1.2.2 py36h2fe25dc_0 defaults
gitdb2 2.0.3 <pip>
GitPython 2.1.7 <pip>
glib 2.53.6 hc861d11_1 defaults
glob2 0.5 py36h2c1b292_1 defaults
gmp 6.1.2 hb3b607b_0 defaults
gmpy2 2.0.8 py36h55090d7_1 defaults
graphite2 1.3.10 hc526e54_0 defaults
greenlet 0.4.12 py36h2d503a6_0 defaults
gst-plugins-base 1.12.2 he3457e5_0 defaults
gstreamer 1.12.2 h4f93127_0 defaults
gunicorn 19.3.0 <pip>
h5py 2.7.0 py36he81ebca_1 defaults
harfbuzz 1.5.0 h2545bd6_0 defaults
hdf5 1.10.1 hb0523eb_0 defaults
heapdict 1.0.0 py36h79797d7_0 defaults
html5lib 0.999999999 py36h2cfc398_0 defaults
icu 58.2 h211956c_0 defaults
idna 2.6 py36h82fb2a8_1 defaults
imageio 2.2.0 py36he555465_0 defaults
imagesize 0.7.1 py36h52d8127_0 defaults
importmagic 0.1.7 <pip>
intel-openmp 2018.0.0 h15fc484_7 defaults
ipykernel 4.6.1 py36hbf841aa_0 defaults
ipython 6.1.0 py36hc72a948_1 defaults
ipython_genutils 0.2.0 py36hb52b0d5_0 defaults
ipywidgets 7.0.0 py36h7b55c3a_0 defaults
isort 4.2.15 py36had401c0_0 defaults
itsdangerous 0.24 py36h93cc618_1 defaults
jbig 2.1 hdba287a_0 defaults
jdcal 1.3 py36h4c697fb_0 defaults
jedi 0.11.1 <pip>
jedi 0.10.2 py36h552def0_0 defaults
Jinja2 2.8.1 <pip>
jinja2 2.9.6 py36h489bce4_1 defaults
jpeg 9b habf39ab_1 defaults
jsonschema 2.6.0 py36h006f8b5_0 defaults
jupyter 1.0.0 py36h9896ce5_0 defaults
jupyter_client 5.1.0 py36h614e9ea_0 defaults
jupyter_console 5.2.0 py36he59e554_1 defaults
jupyter_core 4.3.0 py36h357a921_0 defaults
jupyterlab 0.27.0 py36h86377d0_2 defaults
jupyterlab_launcher 0.4.0 py36h4d8058d_0 defaults
lazy-object-proxy 1.3.1 py36h10fcdad_0 defaults
libedit 3.1 heed3624_0 defaults
libffi 3.2.1 h4deb6c0_3 defaults
libgcc-ng 7.2.0 hcbc56d2_1 defaults
libgfortran-ng 7.2.0 h6fcbd8e_1 defaults
libpng 1.6.32 hda9c8bc_2 defaults
libsodium 1.0.13 h31c71d8_2 defaults
libspatialindex 1.8.5 1 conda-forge
libssh2 1.8.0 h8c220ad_2 defaults
libstdcxx-ng 7.2.0 h24385c6_1 defaults
libtiff 4.0.8 h90200ff_9 defaults
libtool 2.4.6 hd50d1a6_0 defaults
libxcb 1.12 he6ee5dd_2 defaults
libxml2 2.9.4 h6b072ca_5 defaults
libxslt 1.1.29 hcf9102b_5 defaults
llvmlite 0.20.0 py36_0 defaults
locket 0.2.0 py36h787c0ad_1 defaults
lockfile 0.12.2 <pip>
lxml 3.8.0 py36h6c6e760_0 defaults
lxml 4.1.1 <pip>
lzo 2.10 hc0eb8fc_0 defaults
Mako 1.0.7 <pip>
Markdown 2.6.9 <pip>
markupsafe 1.0 py36hd9260cd_1 defaults
matplotlib 2.0.2 py36h2acb4ad_1 defaults
mccabe 0.6.1 py36h5ad9710_1 defaults
mistune 0.7.4 py36hbab8784_0 defaults
mkl 2018.0.0 hb491cac_4 defaults
mkl-service 1.1.2 py36h17a0993_4 defaults
mpc 1.0.3 hf803216_4 defaults
mpfr 3.1.5 h12ff648_1 defaults
mpmath 0.19 py36h8cc018b_2 defaults
msgpack-python 0.4.8 py36hec4c5d1_0 defaults
multipledispatch 0.4.9 py36h41da3fb_0 defaults
munch 2.2.0 <pip>
navigator-updater 0.1.0 py36h14770f7_0 defaults
nb_anacondacloud 1.4.0 py36_0 defaults
nb_conda 2.2.1 py36h8118bb2_0 defaults
nb_conda_kernels 2.1.0 py36_0 defaults
nbconvert 5.3.1 py36hb41ffb7_0 defaults
nbformat 4.4.0 py36h31c9010_0 defaults
nbpresent 3.0.2 py36h5f95a39_1 defaults
ncurses 6.0 h06874d7_1 defaults
networkx 1.11 py36hfb3574a_0 defaults
nltk 3.2.4 py36h1a0979f_0 defaults
nose 1.3.7 py36hcdf7029_2 defaults
notebook 5.0.0 py36h0b20546_2 defaults
numba 0.35.0 np113py36_10 defaults
numexpr 2.6.2 py36hdd3393f_1 defaults
numpy 1.13.1 py36h5bc529a_2 defaults
numpydoc 0.7.0 py36h18f165f_0 defaults
odo 0.5.1 py36h90ed295_0 defaults
olefile 0.44 py36h79f9f78_0 defaults
openpyxl 2.4.8 py36h41dd2a8_1 defaults
openssl 1.0.2n hb7f436b_0 defaults
ordereddict 1.1 <pip>
packaging 16.8 py36ha668100_1 defaults
pandas 0.20.3 py36h842e28d_2 defaults
pandoc 1.19.2.1 hea2e7c5_1 defaults
pandocfilters 1.4.2 py36ha6701b7_1 defaults
pango 1.40.11 hedb6d6b_0 defaults
parso 0.1.1 <pip>
partd 0.3.8 py36h36fd896_0 defaults
patchelf 0.9 hf79760b_2 defaults
path.py 10.3.1 py36he0c6f6d_0 defaults
pathlib2 2.3.0 py36h49efa8e_0 defaults
patsy 0.4.1 py36ha3be15e_0 defaults
pcre 8.41 hc71a17e_0 defaults
pep8 1.7.0 py36h26ade29_0 defaults
pew 1.1.0 <pip>
pexpect 4.2.1 py36h3b9d41b_0 defaults
pickleshare 0.7.4 py36h63277f8_0 defaults
pillow 4.2.1 py36h9119f52_0 defaults
pip 9.0.1 py36h30f8307_2 defaults
pipenv 8.3.1 <pip>
pixman 0.34.0 ha72d70b_1 defaults
pkginfo 1.4.1 py36h215d178_1 defaults
ply 3.10 py36hed35086_0 defaults
prettytable 0.7.2 <pip>
prompt_toolkit 1.0.15 py36h17d85b1_0 defaults
psutil 5.2.2 py36h74c8701_0 defaults
psutil 4.4.2 <pip>
ptyprocess 0.5.2 py36h69acd42_0 defaults
py 1.4.34 py36h0712aa3_1 defaults
pycodestyle 2.3.1 py36hf609f19_0 defaults
pycosat 0.6.3 py36h0a5515d_0 defaults
pycparser 2.18 py36hf9f622e_1 defaults
pycrypto 2.6.1 py36h6998063_1 defaults
pycurl 7.43.0 py36h5e72054_3 defaults
pyflakes 1.5.0 py36h5510808_1 defaults
pygeoif 0.7 <pip>
pygments 2.2.0 py36h0d3125c_0 defaults
pylint 1.7.2 py36h484ab97_0 defaults
PyMySQL 0.8.0 <pip>
pymysql 0.7.11 py36he04d08f_0 defaults
pyodbc 4.0.17 py36h999153c_0 defaults
pyopenssl 17.2.0 py36h5cc804b_0 defaults
pyparsing 2.2.0 py36hee85983_1 defaults
pyproj 1.9.5.1 <pip>
pyqt 5.6.0 py36h0386399_5 defaults
pysocks 1.6.7 py36hd97a5b1_1 defaults
pytables 3.4.2 py36hdce54c9_1 defaults
pytest 3.2.1 py36h11ad3bb_1 defaults
python 3.6.2 hdfe5801_15 defaults
python-daemon 2.1.2 <pip>
python-dateutil 2.6.1 py36h88d3b88_1 defaults
python-editor 1.0.3 <pip>
python-nvd3 0.14.2 <pip>
python-slugify 1.1.4 <pip>
pytz 2017.2 py36hc2ccc2a_1 defaults
pywavelets 0.5.2 py36he602eb0_0 defaults
pyyaml 3.12 py36hafb9ca4_1 defaults
pyzmq 16.0.2 py36h3b0cf96_2 defaults
qt 5.6.2 h974d657_12 defaults
qtawesome 0.4.4 py36h609ed8c_0 defaults
qtconsole 4.3.1 py36h8f73b5b_0 defaults
qtpy 1.3.1 py36h3691cc8_0 defaults
readline 7.0 hac23ff0_3 defaults
requests 2.18.4 py36he2e5f8d_1 defaults
rope 0.10.5 py36h1f8c17e_0 defaults
rtree 0.8.3 py36_0 conda-forge
Rtree 0.8.3 <pip>
ruamel_yaml 0.11.14 py36ha2fb22d_2 defaults
scikit-image 0.13.0 py36had3c07a_1 defaults
scikit-learn 0.19.0 py36h97ac459_2 defaults
scipy 0.19.1 py36h9976243_3 defaults
seaborn 0.8.0 py36h197244f_0 defaults
setproctitle 1.1.10 <pip>
setuptools 36.5.0 py36he42e2e1_0 defaults
Shapely 1.6.3 <pip>
simplegeneric 0.8.1 py36h2cb9092_0 defaults
singledispatch 3.4.0.3 py36h7a266c3_0 defaults
sip 4.18.1 py36h51ed4ed_2 defaults
six 1.10.0 py36hcac75e4_1 defaults
smmap2 2.0.3 <pip>
snowballstemmer 1.2.1 py36h6febd40_0 defaults
SoftLayer 5.2.15 <pip>
sortedcollections 0.5.3 py36h3c761f9_0 defaults
sortedcontainers 1.5.7 py36hdf89491_0 defaults
sphinx 1.6.3 py36he5f0bdb_0 defaults
sphinxcontrib 1.0 py36h6d0f590_1 defaults
sphinxcontrib-websupport 1.0.1 py36hb5cb234_1 defaults
spyder 3.2.3 py36he38cbf7_1 defaults
sqlalchemy 1.2.1 py36h14c3975_0 defaults
sqlite 3.20.1 h6d8b0f3_1 defaults
statsmodels 0.8.0 py36h8533d0b_0 defaults
sympy 1.1.1 py36hc6d1c1c_0 defaults
tabulate 0.7.7 <pip>
tblib 1.3.2 py36h34cf8b6_0 defaults
terminado 0.6 py36ha25a19f_0 defaults
testpath 0.3.1 py36h8cadb63_0 defaults
thrift 0.9.3 <pip>
tk 8.6.7 h5979e9b_1 defaults
toolz 0.8.2 py36h81f2dff_0 defaults
tornado 4.5.2 py36h1283b2a_0 defaults
traitlets 4.3.2 py36h674d592_0 defaults
typing 3.6.2 py36h7da032a_0 defaults
unicodecsv 0.14.1 py36ha668878_0 defaults
Unidecode 0.04.21 <pip>
unixodbc 2.3.4 hc36303a_1 defaults
urllib3 1.22 py36hbe7ace6_0 defaults
virtualenv 15.1.0 <pip>
virtualenv-clone 0.2.6 <pip>
wcwidth 0.1.7 py36hdf4376a_0 defaults
webencodings 0.5.1 py36h800622e_1 defaults
werkzeug 0.12.2 py36hc703753_0 defaults
wheel 0.29.0 py36he7f4e38_1 defaults
widgetsnbextension 3.0.2 py36hd01bb71_1 defaults
wrapt 1.10.11 py36h28b7045_0 defaults
WTForms 2.1 <pip>
xlrd 1.1.0 py36h1db9f0c_1 defaults
xlsxwriter 0.9.8 py36hf41c223_0 defaults
xlwt 1.3.0 py36h7b00a1f_0 defaults
xz 5.2.3 h2bcbf08_1 defaults
yaml 0.1.7 h96e3832_1 defaults
yapf 0.20.0 <pip>
zeromq 4.2.2 hb0b69da_1 defaults
zict 0.1.2 py36ha0d441b_0 defaults
zlib 1.2.11 hfbfcf68_1 defaults
zope.deprecation 4.3.0 <pip>
```
</p></details>
I’ve read the suggested reply at
opened 10:07AM - 29 Jan 18 UTC
closed 11:32AM - 23 Feb 18 UTC
type::feature
source::governance
cli
locked
**I'm submitting a...** (I don't know whether it is a bug or feature)
- [x] b… ug report
- [x] feature request
## Current Behavior
### Steps to Reproduce
Create an environment, 'test' using Conda 4.4, open a new tab and directly activate that environment. Now try to invoke conda commands using any non shell program (lua/python/java/....) it will error out saying `conda: command not found` or `No such file or directory` since they can't `exec()` that.
## Expected Behavior
The program invoking conda should not error out, or should have a way to figure out where the conda program resides, so that it can exec the full `/path/to/conda`. Currently the only thing close to this is `$_CONDA_EXE`, but it is not documented, and also not exported, so not available to child processes and no guarantee whether it will be available in future releases.
The current work around is to first activate the base environment and then activate the non-base environment in the same shell session, which for whatever reason, retains the root environment's bin directory in the PATH.
but can’t make sense of it.
I understand that we prefer to containerise conda installs these days, from Conda hh5 environment setup — CLEX CMS Wiki (and Scott Wales’ template) but that’s isn’t going to solve our immediate problem.
The only activate script I can find in this new environment is
/scratch/nf33/public/miniconda/envs/digital_earths_env/lib/python3.12/venv/scripts/common/activate
Now - before I create a symlink to /bin/activate
to that file (and I’m afraid of unintended consequences), does anyone have a work-around?
Or is that the solution? I’ve noted the hh5 bin/activate
file is very different to envs/digital_earths_env/lib/python3.12/venv/scripts/common/activate
.
Currently- if I specify this to ARE
ARE can find the bin/jupyter
directory
But no kernels are available.
EDIT : A temporary work-around was to build a python virtual environment on top of the conda environment.
python -m venv digital_earths_venv
source /scratch/nf33/public/digital_earths_venv/bin/activate
python -m ipykernel install --prefix /scratch/nf33/public/digital_earths_venv --name digital_earths_venv --display-name "digital_earths_kernel"
cd ~/.local/share/jupyter/kernels/
ln -s /scratch/nf33/public/digital_earths_venv/share/jupyter/kernels/digital_earths_venv
This allows the digital earth conda environment to be loaded into ARE.
But surely there must be an easier way?
Mods - this topic is smilier to How to create a personal Conda environment on Gadi when you really need one? so feel free to merge if required.