Recommended ways to install python library basemap on gadi (if at all?) for dynlib

Technical Infrastructure
, ,
1

Hi all.

Michael Barnes from Monash would like to get the dynlib package installed on gadi: Clemens.Spensberger / dynlib · GitLab

This package still uses basemap for plotting.

Based on some older threads, e.g.:

I thought this would be a good place to ask for recommended strategies. I’m aware that basemap is deprecated and hence shouldn’t be included in the maintained conda environments.

What I’ve tried so far.

  1. Using Scott’s repo to build a new conda in a container: https://git.nci.org.au/bom/ngm/conda-container . This failed with /local/spool/pbs/mom_priv/jobs/124860541.gadi-pbs.SC: line 20: build.log: Read-only file system

  2. I’m not that familiar with containers so I followed NCI suggestions to install extra modules on top of an existing python environment, e.g. Installing custom Python packages - Specialised Environments - Opus - NCI Confluence So I tried:
    pip install -v --no-binary :all: --upgrade-strategy only-if-needed --prefix /g/data/gb02/pag548/EXTRA_PYTHON_LIBS basemap , which produced:

Successfully built basemap basemap-data matplotlib packaging
Installing collected packages: packaging, basemap-data, matplotlib, basemap
  Attempting uninstall: packaging
    Found existing installation: packaging 24.0
    Uninstalling packaging-24.0:
ERROR: Could not install packages due to an OSError.
Consider using the `--user` option or check the permissions.
OSError: [Errno 18] Invalid cross-device link: '/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/packaging-24.0.dist-info/' -> '/scratch/dx2/pag548/tmp/pip-uninstall-y81rzpso'
  1. I’m not sure why it couldn’t link to my own /scratch/dx2/pag548/tmp directory so I repeated the pip command using defaults:
    pip install -v --no-binary :all: --upgrade-strategy only-if-needed basemap This builds and installs with the following warnings:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nctoolkit 1.1.11 requires multiprocess, which is not installed.
spectrum 0.8.1 requires easydev, which is not installed.
arm-pyart 1.18.6 requires netCDF4>=1.7.0, but you have netcdf4 1.6.5 which is incompatible.
dask-xgboost 0.1.11 requires xgboost<=0.90, but you have xgboost 2.0.3 which is incompatible.
icclim 6.5.0 requires xclim<=0.47,>=0.45, but you have xclim 0.51.0 which is incompatible.
xmip 0.7.2 requires xgcm<0.7.0, but you have xgcm 0.8.1 which is incompatible.
  1. It installed the new packages to:
    /home/548/pag548/.local/lib/python3.10/site-packages/ . But, this basemap is incompatible with matplotlib installed in analysis3:
$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/home/548/pag548/.local/lib/python3.10/site-packages')
>>> from  mpl_toolkits import basemap
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/548/pag548/.local/lib/python3.10/site-packages/mpl_toolkits/basemap/__init__.py", line 33, in <module>
    import matplotlib as mpl
  File "/home/548/pag548/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 259, in <module>
    _check_versions()
  File "/home/548/pag548/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 244, in _check_versions
    from . import ft2font
ImportError: /home/548/pag548/.local/lib/python3.10/site-packages/matplotlib/ft2font.cpython-310-x86_64-linux-gnu.so: undefined symbol: FT_Load_Glyph

So now I’m wondering if I should

I have built fresh conda environments on gadi before, but I’m aware that this practise is frowned upon.

As an aside, I recently had to compile an ancient version of basemap (v1.0.7) on a python2.7 platform. This required fiddling with macros in the GEOS header file platform.h but I got it working.

I’m confident I can find a solution eventually, I’m just wondering if someone has done this already.

I note the COSIMA team has already migrated away from basemap : Shifting longitude domain from [-280, 80] to [0, 360], so perhaps another option is replacing all the dynlib calls to basemap with cartopy?

2

Basemap which was the map package for matplotlib? Hasn’t that been deprecated looooong ago?

3

oh wow… it’s been undeprecated!

2 Likes
4

I may be naïve here (meaning I didn’t read through all the log errors above @Paul.Gregory), but there are a few conda analysis environments on NCI that are curated by the CLEx CMS team (cc @Paola-CMS). Why not add dynlib on that environment?

5

Scott apparently had a gist specific to this:
Install dynlib at NCI · GitHub
NB it’s 4 years old

6

Thanks for finding that Paola. I’ll have a look at that.

7

Hi Paul, do you have the environment.yaml that failed in the container? I’ll see if I can reproduce

8

Hi Scott. I had deleted it so I cloned your repo again and this time I kept environment.yaml as supplied.

It generated the same error as before : /local/spool/pbs/mom_priv/jobs/124909747.gadi-pbs.SC: line 20: build.log: Read-only file system

Now, I was checking out the repo in /g/data/gb02/pag548 so I cloned it again in ~pag548/code and this time it worked fine.

Within this environment, what’s the recommended way to install numpy/scipy etc?

Edit environment.yaml file and run the script again?

9

Could be that it needs a storage flag if you’re running in gdata.

Yeah just add numpy etc to environment.yaml

1 Like