Reorganize code to python package and add functionality for processing and visualizing policy logs

Major Changes

• Reworked the repo layout to be a Python package
  • Moved the log preprocessing scripts (splitting and converting to parquet) to their own submodule but have not integrated them
  • Left the scripts for running the policy out of the package
• Added Poetry as the build management tool for CI/CD
• Added automated determination of compute backend based on available resources defined by the Slurm job. Closes #32 (closed), #33 (closed)
  • The different backends are transparent to the user, they will be notified of which backend was chosen if they did not choose one themselves, but all processing modules are compute-agnostic
  • Automatically adds the appropriate modules to the global namespace based on the backend. Also performs initialization hooks where appropriate such as running cudf.pandas.install() when using the cudf backend. Fixes #34 (closed)
• Converted the processing steps performed in the various notebooks to callable functions. Fixes #36 (closed), #37 (closed)
  • Processing commands are used identically regardless of backend
• Added basic plotting functions using plotly. Will be expanded on later for creating a full report. Fixes #38 (closed)
• Stubbed out other submodules for further development

src/run-policy/run-mmpol.sh

@@ -127,7 +127,7 @@ if [[ ! ${dry_run} ]]; then
     log_dir="${outdir}/${log_name}"
 
     mkdir -p ${log_dir}/raw
-    chmod 1770 ${log_dir}
+    chmod 2770 ${log_dir}
 
     # tag output file with run metadata
     raw_log_file=$(find ${outdir} -maxdepth 1 -name "list-${SLURM_JOBID}*" -type f | head -1)
@@ -136,7 +136,7 @@ if [[ ! ${dry_run} ]]; then
         gzip ${log_dir}/raw/${log_name}
 
         chmod 440 ${log_dir}/raw/${log_name}.gz
-        chmod 550 ${log_dir}/raw
+        chmod 1550 ${log_dir}/raw
     fi
 
     chown -R ${USER}:atlab ${log_dir}