Skip to content
Snippets Groups Projects

LMOD log with context

    • Closed Issue created by Bo-Chun Chen @louistw

    Currently, the LMOD log looks like this:

    Mar 31 09:53:18 c0163 ModuleUsageTracking: user=louistw module=slurm/18.08.9 path=/cm/shared/modulefiles/slurm/18.08.9 host=c0163 time=1648738398.009698

    If a module has a bunch of dependencies, for example, R/4.2.0-foss-2021a has 84 dependencies, it will show as 84 entries in the log, and there's no context to associate all of them into a single module load event.

    Edited

    Designs

      Child items ...

      2. Activity

      • Bo-Chun Chen added Backlog label

        added Backlog label

      • Bo-Chun Chen changed the description

        changed the description

      • Ravi Tripathi assigned to @louistw

        assigned to @louistw

      • Bo-Chun Chen
        Bo-Chun Chen @louistw ·
        Author Owner

        PR created: https://github.com/jprorama/CRI_XCBC/pull/400

        Example message with two module load consecutively:

        $ ml R/4.0.3-foss-2018a-X11-20180131-bare
user=louistw pid=144933 module=GCCcore/6.4.0 path=/share/apps/rc/modules/all/GCCcore/6.4.0 host=login005 time=1661823378.823391
user=louistw pid=144933 module=binutils/2.28-GCCcore-6.4.0 path=/share/apps/rc/modules/all/binutils/2.28-GCCcore-6.4.0 host=login005 time=1661823378.832581
user=louistw pid=144933 module=GCC/6.4.0-2.28 path=/share/apps/rc/modules/all/GCC/6.4.0-2.28 host=login005 time=1661823378.832829
user=louistw pid=144933 module=numactl/2.0.11-GCCcore-6.4.0 path=/share/apps/rc/modules/all/numactl/2.0.11-GCCcore-6.4.0 host=login005 time=1661823378.852223
user=louistw pid=144933 module=hwloc/1.11.8-GCCcore-6.4.0 path=/share/apps/rc/modules/all/hwloc/1.11.8-GCCcore-6.4.0 host=login005 time=1661823378.853487
user=louistw pid=144933 module=OpenMPI/2.1.2-GCC-6.4.0-2.28 path=/share/apps/rc/modules/all/OpenMPI/2.1.2-GCC-6.4.0-2.28.lua host=login005 time=1661823378.854696
user=louistw pid=144933 module=OpenBLAS/0.2.20-GCC-6.4.0-2.28 path=/share/apps/rc/modules/all/OpenBLAS/0.2.20-GCC-6.4.0-2.28 host=login005 time=1661823378.871597
user=louistw pid=144933 module=gompi/2018a path=/share/apps/rc/modules/all/gompi/2018a host=login005 time=1661823378.886968
user=louistw pid=144933 module=FFTW/3.3.7-gompi-2018a path=/share/apps/rc/modules/all/FFTW/3.3.7-gompi-2018a host=login005 time=1661823378.888361
user=louistw pid=144933 module=ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20 path=/share/apps/rc/modules/all/ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20 host=login005 time=1661823378.898733
user=louistw pid=144933 module=foss/2018a path=/share/apps/rc/modules/all/foss/2018a host=login005 time=1661823378.899101
user=louistw pid=144933 module=bzip2/1.0.6-GCCcore-6.4.0 path=/share/apps/rc/modules/all/bzip2/1.0.6-GCCcore-6.4.0 host=login005 time=1661823378.915733
user=louistw pid=144933 module=expat/2.2.5-GCCcore-6.4.0 path=/share/apps/rc/modules/all/expat/2.2.5-GCCcore-6.4.0 host=login005 time=1661823378.934389
user=louistw pid=144933 module=zlib/1.2.11-GCCcore-6.4.0 path=/share/apps/rc/modules/all/zlib/1.2.11-GCCcore-6.4.0 host=login005 time=1661823378.960966
user=louistw pid=144933 module=libpng/1.6.34-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libpng/1.6.34-GCCcore-6.4.0 host=login005 time=1661823378.962909
user=louistw pid=144933 module=zlib/1.2.11-GCCcore-6.4.0 path=/share/apps/rc/modules/all/zlib/1.2.11-GCCcore-6.4.0 host=login005 time=1661823378.984952
user=louistw pid=144933 module=freetype/2.9-GCCcore-6.4.0 path=/share/apps/rc/modules/all/freetype/2.9-GCCcore-6.4.0 host=login005 time=1661823378.987113
user=louistw pid=144933 module=fontconfig/2.12.6-GCCcore-6.4.0 path=/share/apps/rc/modules/all/fontconfig/2.12.6-GCCcore-6.4.0 host=login005 time=1661823378.988821
user=louistw pid=144933 module=freetype/2.9-GCCcore-6.4.0 path=/share/apps/rc/modules/all/freetype/2.9-GCCcore-6.4.0 host=login005 time=1661823379.016496
user=louistw pid=144933 module=zlib/1.2.11-GCCcore-6.4.0 path=/share/apps/rc/modules/all/zlib/1.2.11-GCCcore-6.4.0 host=login005 time=1661823379.035650
user=louistw pid=144933 module=X11/20180131-GCCcore-6.4.0 path=/share/apps/rc/modules/all/X11/20180131-GCCcore-6.4.0 host=login005 time=1661823379.038101
user=louistw pid=144933 module=GMP/6.1.2-GCCcore-6.4.0 path=/share/apps/rc/modules/all/GMP/6.1.2-GCCcore-6.4.0 host=login005 time=1661823379.066589
user=louistw pid=144933 module=nettle/3.4-foss-2018a path=/share/apps/rc/modules/all/nettle/3.4-foss-2018a host=login005 time=1661823379.068489
user=louistw pid=144933 module=libdrm/2.4.91-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libdrm/2.4.91-GCCcore-6.4.0 host=login005 time=1661823379.082978
user=louistw pid=144933 module=ncurses/6.0-GCCcore-6.4.0 path=/share/apps/rc/modules/all/ncurses/6.0-GCCcore-6.4.0 host=login005 time=1661823379.101123
user=louistw pid=144933 module=LLVM/5.0.1-GCCcore-6.4.0 path=/share/apps/rc/modules/all/LLVM/5.0.1-GCCcore-6.4.0 host=login005 time=1661823379.102882
user=louistw pid=144933 module=Mesa/17.3.6-foss-2018a path=/share/apps/rc/modules/all/Mesa/17.3.6-foss-2018a host=login005 time=1661823379.104494
user=louistw pid=144933 module=libGLU/9.0.0-foss-2018a path=/share/apps/rc/modules/all/libGLU/9.0.0-foss-2018a host=login005 time=1661823379.118543
user=louistw pid=144933 module=pixman/0.34.0-GCCcore-6.4.0 path=/share/apps/rc/modules/all/pixman/0.34.0-GCCcore-6.4.0 host=login005 time=1661823379.136619
user=louistw pid=144933 module=libffi/3.2.1-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libffi/3.2.1-GCCcore-6.4.0 host=login005 time=1661823379.156243
user=louistw pid=144933 module=XZ/5.2.3-GCCcore-6.4.0 path=/share/apps/rc/modules/all/XZ/5.2.3-GCCcore-6.4.0 host=login005 time=1661823379.187265
user=louistw pid=144933 module=libxml2/2.9.7-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libxml2/2.9.7-GCCcore-6.4.0 host=login005 time=1661823379.190818
user=louistw pid=144933 module=gettext/0.19.8.1-GCCcore-6.4.0-libxml2-2.9.7 path=/share/apps/rc/modules/all/gettext/0.19.8.1-GCCcore-6.4.0-libxml2-2.9.7 host=login005 time=1661823379.193459
user=louistw pid=144933 module=libxml2/2.9.7-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libxml2/2.9.7-GCCcore-6.4.0 host=login005 time=1661823379.228171
user=louistw pid=144933 module=PCRE/8.41-GCCcore-6.4.0 path=/share/apps/rc/modules/all/PCRE/8.41-GCCcore-6.4.0 host=login005 time=1661823379.239710
user=louistw pid=144933 module=util-linux/2.31.1-GCCcore-6.4.0 path=/share/apps/rc/modules/all/util-linux/2.31.1-GCCcore-6.4.0 host=login005 time=1661823379.254563
user=louistw pid=144933 module=GLib/2.54.3-GCCcore-6.4.0 path=/share/apps/rc/modules/all/GLib/2.54.3-GCCcore-6.4.0 host=login005 time=1661823379.257561
user=louistw pid=144933 module=cairo/1.14.12-GCCcore-6.4.0 path=/share/apps/rc/modules/all/cairo/1.14.12-GCCcore-6.4.0 host=login005 time=1661823379.260304
user=louistw pid=144933 module=libreadline/7.0-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libreadline/7.0-GCCcore-6.4.0 host=login005 time=1661823379.280540
user=louistw pid=144933 module=Tcl/8.6.8-GCCcore-6.4.0 path=/share/apps/rc/modules/all/Tcl/8.6.8-GCCcore-6.4.0 host=login005 time=1661823379.304630
user=louistw pid=144933 module=SQLite/3.21.0-GCCcore-6.4.0 path=/share/apps/rc/modules/all/SQLite/3.21.0-GCCcore-6.4.0 host=login005 time=1661823379.307951
user=louistw pid=144933 module=NASM/2.13.03-GCCcore-6.4.0 path=/share/apps/rc/modules/all/NASM/2.13.03-GCCcore-6.4.0 host=login005 time=1661823379.332122
user=louistw pid=144933 module=libjpeg-turbo/1.5.3-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libjpeg-turbo/1.5.3-GCCcore-6.4.0 host=login005 time=1661823379.335533
user=louistw pid=144933 module=LibTIFF/4.0.9-GCCcore-6.4.0 path=/share/apps/rc/modules/all/LibTIFF/4.0.9-GCCcore-6.4.0 host=login005 time=1661823379.354537
user=louistw pid=144933 module=Java/1.8.0_162 path=/share/apps/rc/modules/all/Java/1.8.0_162 host=login005 time=1661823379.369688
user=louistw pid=144933 module=Tk/8.6.8-foss-2018a path=/share/apps/rc/modules/all/Tk/8.6.8-foss-2018a host=login005 time=1661823379.386020
user=louistw pid=144933 module=cURL/7.58.0-GCCcore-6.4.0 path=/share/apps/rc/modules/all/cURL/7.58.0-GCCcore-6.4.0 host=login005 time=1661823379.407693
user=louistw pid=144933 module=Python/3.6.4-foss-2018a path=/share/apps/rc/modules/all/Python/3.6.4-foss-2018a host=login005 time=1661823379.435799
user=louistw pid=144933 module=Szip/2.1.1-GCCcore-6.4.0 path=/share/apps/rc/modules/all/Szip/2.1.1-GCCcore-6.4.0 host=login005 time=1661823379.470120
user=louistw pid=144933 module=HDF5/1.10.1-foss-2018a path=/share/apps/rc/modules/all/HDF5/1.10.1-foss-2018a host=login005 time=1661823379.473473
user=louistw pid=144933 module=Szip/2.1.1-GCCcore-6.4.0 path=/share/apps/rc/modules/all/Szip/2.1.1-GCCcore-6.4.0 host=login005 time=1661823379.499032
user=louistw pid=144933 module=netCDF/4.6.0-foss-2018a path=/share/apps/rc/modules/all/netCDF/4.6.0-foss-2018a host=login005 time=1661823379.503122
user=louistw pid=144933 module=GEOS/3.6.2-foss-2018a-Python-3.6.4 path=/share/apps/rc/modules/all/GEOS/3.6.2-foss-2018a-Python-3.6.4 host=login005 time=1661823379.519286
user=louistw pid=144933 module=JasPer/2.0.14-GCCcore-6.4.0 path=/share/apps/rc/modules/all/JasPer/2.0.14-GCCcore-6.4.0 host=login005 time=1661823379.537406
user=louistw pid=144933 module=PROJ/5.0.0-foss-2018a path=/share/apps/rc/modules/all/PROJ/5.0.0-foss-2018a host=login005 time=1661823379.556901
user=louistw pid=144933 module=libgeotiff/1.4.2-foss-2018a path=/share/apps/rc/modules/all/libgeotiff/1.4.2-foss-2018a host=login005 time=1661823379.574461
user=louistw pid=144933 module=GDAL/2.2.3-foss-2018a-Python-3.6.4 path=/share/apps/rc/modules/all/GDAL/2.2.3-foss-2018a-Python-3.6.4 host=login005 time=1661823379.578532
user=louistw pid=144933 module=NLopt/2.4.2-foss-2018a path=/share/apps/rc/modules/all/NLopt/2.4.2-foss-2018a host=login005 time=1661823379.595349
user=louistw pid=144933 module=libsndfile/1.0.28-GCCcore-6.4.0 path=/share/apps/rc/modules/all/libsndfile/1.0.28-GCCcore-6.4.0 host=login005 time=1661823379.609882
user=louistw pid=144933 module=ICU/61.1-GCCcore-6.4.0 path=/share/apps/rc/modules/all/ICU/61.1-GCCcore-6.4.0 host=login005 time=1661823379.627795
user=louistw pid=144933 module=R/4.0.3-foss-2018a-X11-20180131-bare path=/share/apps/rc/modules/all/R/4.0.3-foss-2018a-X11-20180131-bare.lua host=login005 time=1661823379.633628
$ ml cURL
user=louistw pid=145426 module=GCCcore/10.3.0 path=/share/apps/rc/modules/all/GCCcore/10.3.0.lua host=login005 time=1661823392.542765
user=louistw pid=145426 module=GCCcore/10.3.0 path=/share/apps/rc/modules/all/GCCcore/10.3.0.lua host=login005 time=1661823392.576972
user=louistw pid=145426 module=zlib/1.2.11-GCCcore-10.3.0 path=/share/apps/rc/modules/all/zlib/1.2.11-GCCcore-10.3.0.lua host=login005 time=1661823392.581009
user=louistw pid=145426 module=OpenSSL/1.1 path=/share/apps/rc/modules/all/OpenSSL/1.1.lua host=login005 time=1661823392.587507
user=louistw pid=145426 module=cURL/7.76.0-GCCcore-10.3.0 path=/share/apps/rc/modules/all/cURL/7.76.0-GCCcore-10.3.0.lua host=login005 time=1661823392.592535
        Edited by Bo-Chun Chen
      • Bo-Chun Chen added Sprint 22-17 label and removed Backlog label

        added Sprint 22-17 label and removed Backlog label

      • Bo-Chun Chen mentioned in issue #122 (closed)

        mentioned in issue #122 (closed)

      • Bo-Chun Chen
        Bo-Chun Chen @louistw ·
        Author Owner

        To test it on Cheaha, follow the steps:

        1. Download the SitePackage.lua file from github
        cd $HOME
$ wget https://raw.githubusercontent.com/jprorama/CRI_XCBC/0355b40895970a75d9510f3a4d35973bd76d67fe/roles/lmod_db/files/SitePackage.lua
        1. Change the last few lines so it won't send the log to actual database.
        sed -i "s%\(lmod_system_execute.*\)%-- \1\nlmod_system_execute('echo ' .. msg .. '>> $(pwd)/output.log')%" SitePackage.lua
        1. Point lmod to the customized SitePackage
        export LMOD_PACKAGE_PATH=$(pwd)
        1. Load some packages to test the output
        module reset
module load GCCcore
module reset
module load foss/2021a
        1. Unset the customized SitePackage
        unset LMOD_PACKAGE_PATH
        1. Check the output
        less output.log
        1. Clean up the test SitePackage file
        rm output.log
rm SitePackage.lua
      • Ravi Tripathi
        Ravi Tripathi @ravi89 ·

        Tested out the method mentioned above by @louistw

        user=ravi89 pid=147987 module=shared path=/cm/local/modulefiles/shared host=login005 time=1662741473.121677
user=ravi89 pid=147987 module=slurm/18.08.9 path=/cm/shared/modulefiles/slurm/18.08.9 host=login005 time=1662741473.132633
user=ravi89 pid=147987 module=slurm/18.08.9 path=/cm/shared/modulefiles/slurm/18.08.9 host=login005 time=1662741475.799791
user=ravi89 pid=147987 module=rc-base path=/cm/shared/modulefiles/rc-base host=login005 time=1662741475.799938
user=ravi89 pid=147987 module=DefaultModules path=/usr/share/modulefiles/DefaultModules.lua host=login005 time=1662741475.800068
user=ravi89 pid=148367 module=GCCcore/10.3.0 path=/share/apps/rc/modules/all/GCCcore/10.3.0.lua host=login005 time=1662741479.003643
user=ravi89 pid=148584 module=shared path=/cm/local/modulefiles/shared host=login005 time=1662741482.430316
user=ravi89 pid=148584 module=slurm/18.08.9 path=/cm/shared/modulefiles/slurm/18.08.9 host=login005 time=1662741482.438778
user=ravi89 pid=148584 module=slurm/18.08.9 path=/cm/shared/modulefiles/slurm/18.08.9 host=login005 time=1662741485.215618
user=ravi89 pid=148584 module=rc-base path=/cm/shared/modulefiles/rc-base host=login005 time=1662741485.215852
user=ravi89 pid=148584 module=DefaultModules path=/usr/share/modulefiles/DefaultModules.lua host=login005 time=1662741485.216467
user=ravi89 pid=149001 module=GCCcore/10.3.0 path=/share/apps/rc/modules/all/GCCcore/10.3.0.lua host=login005 time=1662741488.254753
user=ravi89 pid=149001 module=zlib/1.2.11-GCCcore-10.3.0 path=/share/apps/rc/modules/all/zlib/1.2.11-GCCcore-10.3.0.lua host=login005 time=1662741488.256949
user=ravi89 pid=149001 module=binutils/2.36.1-GCCcore-10.3.0 path=/share/apps/rc/modules/all/binutils/2.36.1-GCCcore-10.3.0.lua host=login005 time=1662741488.257891
user=ravi89 pid=149001 module=GCC/10.3.0 path=/share/apps/rc/modules/all/GCC/10.3.0.lua host=login005 time=1662741488.258105
user=ravi89 pid=149001 module=numactl/2.0.14-GCCcore-10.3.0 path=/share/apps/rc/modules/all/numactl/2.0.14-GCCcore-10.3.0.lua host=login005 time=1662741488.261383
user=ravi89 pid=149001 module=XZ/5.2.5-GCCcore-10.3.0 path=/share/apps/rc/modules/all/XZ/5.2.5-GCCcore-10.3.0.lua host=login005 time=1662741488.264036
user=ravi89 pid=149001 module=libxml2/2.9.10-GCCcore-10.3.0 path=/share/apps/rc/modules/all/libxml2/2.9.10-GCCcore-10.3.0.lua host=login005 time=1662741488.265619
user=ravi89 pid=149001 module=libpciaccess/0.16-GCCcore-10.3.0 path=/share/apps/rc/modules/all/libpciaccess/0.16-GCCcore-10.3.0.lua host=login005 time=1662741488.267385
user=ravi89 pid=149001 module=hwloc/2.4.1-GCCcore-10.3.0 path=/share/apps/rc/modules/all/hwloc/2.4.1-GCCcore-10.3.0.lua host=login005 time=1662741488.269104
user=ravi89 pid=149001 module=OpenSSL/1.1 path=/share/apps/rc/modules/all/OpenSSL/1.1.lua host=login005 time=1662741488.272651
user=ravi89 pid=149001 module=libevent/2.1.12-GCCcore-10.3.0 path=/share/apps/rc/modules/all/libevent/2.1.12-GCCcore-10.3.0.lua host=login005 time=1662741488.274074
user=ravi89 pid=149001 module=UCX/1.10.0-GCCcore-10.3.0 path=/share/apps/rc/modules/all/UCX/1.10.0-GCCcore-10.3.0.lua host=login005 time=1662741488.276905
user=ravi89 pid=149001 module=libfabric/1.12.1-GCCcore-10.3.0 path=/share/apps/rc/modules/all/libfabric/1.12.1-GCCcore-10.3.0.lua host=login005 time=1662741488.279993
user=ravi89 pid=149001 module=PMIx/3.2.3-GCCcore-10.3.0 path=/share/apps/rc/modules/all/PMIx/3.2.3-GCCcore-10.3.0.lua host=login005 time=1662741488.283204
user=ravi89 pid=149001 module=OpenMPI/4.1.1-GCC-10.3.0 path=/share/apps/rc/modules/all/OpenMPI/4.1.1-GCC-10.3.0.lua host=login005 time=1662741488.285325
user=ravi89 pid=149001 module=OpenBLAS/0.3.15-GCC-10.3.0 path=/share/apps/rc/modules/all/OpenBLAS/0.3.15-GCC-10.3.0.lua host=login005 time=1662741488.290061
user=ravi89 pid=149001 module=FlexiBLAS/3.0.4-GCC-10.3.0 path=/share/apps/rc/modules/all/FlexiBLAS/3.0.4-GCC-10.3.0.lua host=login005 time=1662741488.293591
user=ravi89 pid=149001 module=gompi/2021a path=/share/apps/rc/modules/all/gompi/2021a.lua host=login005 time=1662741488.299415
user=ravi89 pid=149001 module=FFTW/3.3.9-gompi-2021a path=/share/apps/rc/modules/all/FFTW/3.3.9-gompi-2021a.lua host=login005 time=1662741488.301776
user=ravi89 pid=149001 module=ScaLAPACK/2.1.0-gompi-2021a-fb path=/share/apps/rc/modules/all/ScaLAPACK/2.1.0-gompi-2021a-fb.lua host=login005 time=1662741488.304533
user=ravi89 pid=149001 module=foss/2021a path=/share/apps/rc/modules/all/foss/2021a.lua host=login005 time=1662741488.305040
      • Bo-Chun Chen closed

        closed

      Please register or sign in to reply