set the policy file to either list-path-external or list-path-dirplus, then pass as a path to submit-pol-job

add option to run any policy file so Ops is not restricted to only list-path-external and list-path-dirplus

Automate conversion of GPFS policy outputs to parquet without Jupyter

See merge request !8

convert to running from SIF container instead of bare metal conda env. add option to specify container

Create distinct listing that includes all object types on gpfs.
Copy of the the list-path-external with the additional option.

These nodebooks use dataframes build from parquet files to sanity check
file listings from multiple sources.

These are use to create specific reports by opening the notebook
then copying it and modifying the parameters to a specific
policy run data set.

Their utility may be limited based on current parquet pipelines.

This is intended to be run on URL encoded output lines from a
gpfs list policy run.  It creates panda structures that are
then saved as parquet format for ease of downstream processing.

Can be run in parallel across many inputs by wrapping with papermill
and have upstream split the input file.

Makes it slightly more resilient and forms the foundation for
the parquet notebook.

Parameters hard coded to split on 50k enteries per file which provides
reasonable parallel read efficiency for downstream tasks.
Put splits in a .d subdir of file name.

Provide details on running a policy script with the provided
workflow framework.
Information on the list-policy-external policy and generated output.
Outline of future docs for preprocessing and reports.

Resolve "create script wrappers to run notebooks in parallel"

Closes #5

See merge request !7

Resolve "Create report to aggregate stored data by year of last access"

Closes #4

See merge request !6

The report accepts a top-level-dir pattern and reports aggregation
stats for files below that level

Merge branch '3-create-a-pickling-script-to-parse-raw-text-into-pickled-pandas-dataframes' into 'main'

Create notebook to pickle list policy output files

Closes #3

See merge request !5