Unexpected Anaconda environment behavior using `pip` in certain circumstances

Overview

We're seeing a number of issues surrounding pip.

1. If pip install $package is used from a bare terminal (no Anaconda), packages are installed using system pip (/usr/bin/pip) to ~/.local.
2. If used from an Anaconda environment where pip is not installed, then see #1.
3. If used from an Anaconda environment where pip is installed, then the Anaconda pip package takes precedence over system pip, and packages are installed to .conda.

Issues arise if a researcher unwittingly hits case #1 or #2 (closed), inadvertently installing a package into .local. Python (even with Anaconda) makes use of packages in .local before .conda, which can lead to confusing version conflicts and Python errors.

Symptoms

• Unreproducible errors when installing a conda environment from a YAML file, despite conda --clean.
• Disappearing job cards and intermittent job start failures in OOD Jupyter.

Minimal Working Example

• module load Anaconda3
• pip install mkdocs-table-reader-plugin
• python
• import sys; print('\n'.join(sys.path))

You should see the following. Note that .local comes before the conda site-packages.

/share/apps/rc/software/Anaconda3/2023.07-2/lib/python311.zip
/share/apps/rc/software/Anaconda3/2023.07-2/lib/python3.11
/share/apps/rc/software/Anaconda3/2023.07-2/lib/python3.11/lib-dynload
/home/wwarr/.local/lib/python3.11/site-packages
/share/apps/rc/software/Anaconda3/2023.07-2/lib/python3.11/site-packages

Possible solutions

• Change the order of $PYTHONPATH to put .local after .conda. This can be done with export PYTHONPATH= then listing the expected Anaconda directories. There are four and the prefix is dynamic using $CONDA_PREFIX. We can possibly do this in the module file.
• Restrict read access for /usr/bin/pip to a specific group (chmod o-rx). Researchers don't need this and should be redirected to Anaconda pip. May break existing workflows for some researchers (who should be using Anaconda anyway).