Enhance raw GPFS log splitting and compression script

See merge request !10

Automate conversion of GPFS policy outputs to parquet without Jupyter

See merge request !8

convert to running from SIF container instead of bare metal conda env. add option to specify container

Create distinct listing that includes all object types on gpfs.
Copy of the the list-path-external with the additional option.

These nodebooks use dataframes build from parquet files to sanity check
file listings from multiple sources.

These are use to create specific reports by opening the notebook
then copying it and modifying the parameters to a specific
policy run data set.

Their utility may be limited based on current parquet pipelines.

This is intended to be run on URL encoded output lines from a
gpfs list policy run.  It creates panda structures that are
then saved as parquet format for ease of downstream processing.

Can be run in parallel across many inputs by wrapping with papermill
and have upstream split the input file.

Makes it slightly more resilient and forms the foundation for
the parquet notebook.

Parameters hard coded to split on 50k enteries per file which provides
reasonable parallel read efficiency for downstream tasks.
Put splits in a .d subdir of file name.

Provide details on running a policy script with the provided
workflow framework.
Information on the list-policy-external policy and generated output.
Outline of future docs for preprocessing and reports.

Resolve "create script wrappers to run notebooks in parallel"

Closes #5

See merge request !7

Resolve "Create report to aggregate stored data by year of last access"

Closes #4

See merge request !6

The report accepts a top-level-dir pattern and reports aggregation
stats for files below that level

Merge branch '3-create-a-pickling-script-to-parse-raw-text-into-pickled-pandas-dataframes' into 'main'

Create notebook to pickle list policy output files

Closes #3

See merge request !5

This simplifies later report running to consolidating parsing and dataframe
create operations into a single batch.

Takes a directory of atime reports generated by array job and
merges into a single per-user report.

The report atime generator reads a provided input and generates
a per-user atime report.
The wrapper sbatch allows running the script in an array job
to support scaling across large data sets split into many files.

Figure out logic to parse list files and produce atime summaries.

Feat parsable file names

Closes #2

See merge request !2

The defer option with a list generates the list file outputs
according to the format described here:

https://www.ibm.com/docs/en/spectrum-scale/4.2.0?topic=pools-file-list-format

The file and ouput args control the location of the resulting file.

Pass the output are to the sbatch via env params.

Remove the hard-coded external script both because it is not needed
and because defer mode produces list output files directly.