Update main to v0.1.1

This merge request adds some minor usability improvements as well as bugfixes. Scripts used to run policies are not affected and will be updated in a separate request.

Changes:

• removed dependence on sif for convert-to-parquet.py's environment. Instead, run-convert-to-parquet will automatically source a virtual environment stored in /data/rc/gpfs-policy with the necessary packages and Python interpreter. (Fixes #23 (closed))
• Set the default path for split files to be in a directory with the same name as the log file being split. This directory will be in the same place as the log file. (Fixed #19 (closed))
  • The output directory can also now be specified by the user
• Added and improved documentation for each step in the current pipeline from running a policy to converting split logs to parquet

Bugfixes:

• tld defaults to None when the path is not in /data/user, /data/project, /home, or /scratch (Fixes #22 (closed))

src/convert-to-parquet/run-convert-to-parquet.sh

@@ -11,7 +11,6 @@ mem="16G"
 time="02:00:00"
 partition="amd-hdr100"
 outdir=""
-sif="gitlab.rc.uab.edu:4567/mdefende/gpfs-policy:latest"
 
 ############################################################
 # Help                                                     #
@@ -87,15 +86,12 @@ if [[ -z "$outdir" ]]; then
     outdir="${gpfs_logdir}/parquet"
 fi
 
-singularity pull --force gpfs.sif docker://${sif}
-
 nlogs=$(ls ${gpfs_logdir}/list-* | wc -l)
 
-cmd="singularity exec --bind /data,/scratch gpfs.sif python3 convert-to-parquet.py -o ${outdir} -f \${log}"
+cmd="python3 convert-to-parquet.py -o ${outdir} -f \${log}"
 
 >&2 cat << EOF
 --------------------------------------------------------------------------------
-sif:                 ${sif}
 output dir:          ${outdir}
 GPFS logs:           ${gpfs_logdir}
 
@@ -127,6 +123,8 @@ mkdir -p err
 #SBATCH --error=err/%A_%a.err
 #SBATCH --array=1-${nlogs}
 
+source /data/rc/gpfs-policy/venv/bin/activate
+
 log=\$(ls ${gpfs_logdir}/list-* | awk "NR==\${SLURM_ARRAY_TASK_ID} { print \$1 }")
 
 ${cmd}