The modules that are being loaded via DefaultModules.lua are different from production. This was a result from testing it in CoD. We should match the production DefaultModules.lua which loads rc-base by default that contains the paths for /share/apps modules. Without it the $MODULEPATH env variable is missing the /share/apps and Lmod throws error that it could see the modules but not load them.
We have a solution for the issue. In order to incorporate the fix for above problem in our build we need to build the image for compute/gpu but currently we are facing issue while trying to install the packages from the pkg list in group_vars/compute where we install pkgs from cm8. The base image we use is CentOS but compute nodes on cheaha use Bright provided base image.
We cleaned the pkgs causing problem packer-openstack-hpc-image#71 because they are not a requirement for compute image to function. After which the build was successful.