Unexpected Anaconda environment behavior using `pip` in certain circumstances

Overview

We're seeing a number of issues surrounding pip.

1. If pip install $package is used from a bare terminal (no Anaconda), packages are installed using system pip (/usr/bin/pip) to ~/.local.
2. If used from an Anaconda environment where pip is not installed, then see #1.
3. If used from an Anaconda environment where pip is installed, then the Anaconda pip package takes precedence over system pip, and packages are installed to .conda.

Issues arise if a researcher unwittingly hits case #1 or #2 (closed), inadvertently installing a package into .local. Python (even with Anaconda) makes use of packages in .local before .conda, which can lead to confusing version conflicts and Python errors.

Symptoms

• Unreproducible errors when installing a conda environment from a YAML file, despite conda --clean.
• Disappearing job cards and intermittent job start failures in OOD Jupyter.

Minimal Working Example

• module load Anaconda3
• pip install mkdocs-table-reader-plugin
• python
• import sys; print('\n'.join(sys.path))

You should see the following. Note that .local comes before the conda site-packages.

/share/apps/rc/software/Anaconda3/2023.07-2/lib/python311.zip
/share/apps/rc/software/Anaconda3/2023.07-2/lib/python3.11
/share/apps/rc/software/Anaconda3/2023.07-2/lib/python3.11/lib-dynload
/home/wwarr/.local/lib/python3.11/site-packages
/share/apps/rc/software/Anaconda3/2023.07-2/lib/python3.11/site-packages

Possible solutions

• Change the order of $PYTHONPATH to put .local after .conda. This can be done with export PYTHONPATH= then listing the expected Anaconda directories. There are four and the prefix is dynamic using $CONDA_PREFIX. We can possibly do this in the module file.
• Restrict read access for /usr/bin/pip to a specific group (chmod o-rx). Researchers don't need this and should be redirected to Anaconda pip. May break existing workflows for some researchers (who should be using Anaconda anyway).

added Backlog label

changed the description

python -s excludes user-specific site-packages directories, which means excluding the offending one here. This could be an option to set as a default.

(test-pythonpath) [wwarr@login004 ~]$ python
>>> import sys; print('\n'.join(sys.path))
/home/wwarr/.conda/envs/test-pythonpath/lib/python311.zip
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11/lib-dynload
/home/wwarr/.local/lib/python3.11/site-packages
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11/site-packages


(test-pythonpath) [wwarr@login004 ~]$ python -s
>>> import sys; print('\n'.join(sys.path))
/home/wwarr/.conda/envs/test-pythonpath/lib/python311.zip
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11/lib-dynload
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11/site-packages

export PYTHONNOUSERSITE=1 has the same effect

(test-pythonpath) [wwarr@login004 ~]$ export PYTHONNOUSERSITE=1 
(test-pythonpath) [wwarr@login004 ~]$ python
>>> import sys; print('\n'.join(sys.path))
/home/wwarr/.conda/envs/test-pythonpath/lib/python311.zip
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11/lib-dynload
/home/wwarr/.conda/envs/test-pythonpath/lib/python3.11/site-packages

So if we put export PYTHONNOUSERSITE=1 into the Anaconda module definition we should be able to avoid this error. It isn't clear yet what other side-effects this might have.