User Story: As a researcher using Singularity on Cheaha, having only a single version installed directly on nodes prevents me from falling back to a known stable version if my pipelines break on the new version.

User Story

As a researcher using Singularity on Cheaha, having only a single version installed directly on nodes prevents me from falling back to a known stable version if my pipelines break on the new version.

Proposed Solution

Extend the module system to local directories to get the best of both worlds. Singularity will be installed on nodes, and multiple versions are support using Lmod with the usual module interface.

1. Prepare a local directory where all node-local modules can be installed (in this case, Singularity). This directory must be universal across all nodes.
2. Install node-local modules there when building/updating the node
3. Add the universal local directory to the Lmod MODULEPATH variable by the recommended method. This would be added in the same place we add the currently existing path, ideally. It isn't clear what happens if the directory doesn't exist (i.e., on the login node).

Notes

• More information about how Lmod picks modules: https://lmod.readthedocs.io/en/latest/060_locating.html
  • Note that order of MODULEPATH matters if there are multiple copies of the same module name and no named default. In that case, Lmod picks the first module it finds with the non-unique name.
• Our specific case is discussed here: https://lmod.readthedocs.io/en/latest/060_locating.html#n-v-picking-modules-when-there-are-multiple-directories-in-modulepath