Update main to v0.1.1

This merge request adds some minor usability improvements as well as bugfixes. Scripts used to run policies are not affected and will be updated in a separate request.

Changes:

• removed dependence on sif for convert-to-parquet.py's environment. Instead, run-convert-to-parquet will automatically source a virtual environment stored in /data/rc/gpfs-policy with the necessary packages and Python interpreter. (Fixes #23 (closed))
• Set the default path for split files to be in a directory with the same name as the log file being split. This directory will be in the same place as the log file. (Fixed #19 (closed))
  • The output directory can also now be specified by the user
• Added and improved documentation for each step in the current pipeline from running a policy to converting split logs to parquet

Bugfixes:

• tld defaults to None when the path is not in /data/user, /data/project, /home, or /scratch (Fixes #22 (closed))

src/split-info-file.sh

 #!/bin/bash
 
-set -euo pipefail
+set -euxo pipefail
 
 ############################################################
 # Default Values                                           #
@@ -21,8 +21,9 @@ usage()
 >&2 cat << EOF
 Usage: $0 [ -h ] [ -l | --lines ] [ -o | --outdir ]
           [ -n | --ntasks ] [ -p | --partition] [ -t | --time ] [ -m | --mem ] 
-           log
+          log
 EOF
+exit 0
 }
 
 help()
@@ -32,10 +33,10 @@ help()
 Splits a GPFS policy log into multiple parts for batch array processing
 Usage: $0 [ -h ] [ -l | --lines ] [ -o | --outdir ]
           [ -n | --ntasks ] [ -p | --partition] [ -t | --time ] [ -m | --mem ] 
-           log
+          log
 
 General:
-    -h|--help           Print this Help.
+    -h|--help           Print this help.
 
 Required:
     log                 Path to the log file to split
@@ -44,7 +45,7 @@ Split Parameters:
     -l|--lines          Max number of records to save in each split (default: 5000000)
 
 File Parameters:
-    -o|--outdir         Directory path to store split files in. Defaults to ${log}.d in ${log}'s parent directory.
+    -o|--outdir         Directory path to store split files in. Defaults to log.d in log's parent directory.
 
 Job Parameters:
     -n|--ntasks         Number of job tasks (default: 4)
@@ -105,7 +106,7 @@ fi
 >&2 cat << EOF
 --------------------------------------------------------------------------------
 GPFS log:           ${log}
-Output Directory    $(outdir)
+Output Directory    ${outdir}
 Lines per File:     ${lines}
 
 ntasks:             ${ntasks}