This merge request adds some minor usability improvements as well as bugfixes. Scripts used to run policies are not affected and will be updated in a separate request.
Changes:
removed dependence on sif for convert-to-parquet.py's environment. Instead, run-convert-to-parquet will automatically source a virtual environment stored in /data/rc/gpfs-policy with the necessary packages and Python interpreter. (Fixes #23 (closed))
Set the default path for split files to be in a directory with the same name as the log file being split. This directory will be in the same place as the log file. (Fixed #19 (closed))
The output directory can also now be specified by the user
Added and improved documentation for each step in the current pipeline from running a policy to converting split logs to parquet
Bugfixes:
tld defaults to None when the path is not in /data/user, /data/project, /home, or /scratch (Fixes #22 (closed))
