Automate conversion of GPFS policy outputs to parquet without Jupyter

Created a set of scripts to parse our standard GPFS policy outputs and save them as a parquet dataset without needing a Jupyter notebook. Iterated off of parquet-list-policy-data.ipynb.

Changes

• Simplified parsing algorithm
• Automatically extracts the "top-level directory" (tld). Can parse from /data/user, /data/user/home, /data/project, and /scratch.
  • Sets the tld as the index within each parquet file for faster aggregation later
• Variable output directory: defaults to log_dir/parquet but can be specified elsewhere
• Environment is controlled through a Singularity container (defaults to daskdev/dask:2024.8.0-py3.12) but is variable
  • If the container is not specified, the default is automatically downloaded and used
• Parallelization: Each part of a policy output is processed individually in an array task. Processing logs parts of 5 million lines apiece takes ~ 3 minutes
• Controlled via command line run-convert-to-parquet.sh

README.md

@@ -65,6 +65,24 @@ The ouput file is an unsorted list of files in uncompressed ASCII.  Further proc
 
 ### Pre-parse output for Python
 
+Processing GPFS log outputs is controlled by the `run-convert-to-parquet.sh` script and assumes the GPFS log has been split into a number of files of the form `list-XXX.gz` where `XXX` is an incrementing numeric index. This creates an array job where each task in the array reads the quoted text in one file, parses it into a dataframe, and exports it as a parquet file with the name `list-XXX.parquet`. 
+
+While the file is being parsed, the top-level-directory (`tld`) is extracted for each entry and added as a separate column to make common aggregations easier.
+
+This script is written to parse the `list-policy-external` policy format with quoted special characters.
+
+```
+Usage: ./run-convert-to-parquet.sh [ -h ] 
+        [ -o | --outdir ] [ -n | --ntasks ] [ -p | --partition] 
+        [ -t | --time ] [ -m | --mem ] 
+        gpfs_logdir"
+```
+
+- `outdir`: Path to save parquet outputs. Defaults to `${gpfs_logdir}/parquet`
+- `gpfs_logdir`: Directory path containing the split log files as `*.gz`
+
+All other options control the array job resources. The default resources can parse 5 million line files in approximately 3 minutes so should cover all common use cases.
+
 ## Running reports
 
 ### Disk usage by top level directies