Skip to content

Reorganize code to python package and add functionality for processing and visualizing policy logs

Matthew K Defenderfer requested to merge mdefende/gpfs-policy:main into main

Major Changes

  • Reworked the repo layout to be a Python package
    • Moved the log preprocessing scripts (splitting and converting to parquet) to their own submodule but have not integrated them
    • Left the scripts for running the policy out of the package
  • Added Poetry as the build management tool for CI/CD
  • Added automated determination of compute backend based on available resources defined by the Slurm job. Closes #32 (closed), #33 (closed)
    • The different backends are transparent to the user, they will be notified of which backend was chosen if they did not choose one themselves, but all processing modules are compute-agnostic
    • Automatically adds the appropriate modules to the global namespace based on the backend. Also performs initialization hooks where appropriate such as running cudf.pandas.install() when using the cudf backend. Fixes #34 (closed)
  • Converted the processing steps performed in the various notebooks to callable functions. Fixes #36 (closed), #37 (closed)
    • Processing commands are used identically regardless of backend
  • Added basic plotting functions using plotly. Will be expanded on later for creating a full report. Fixes #38 (closed)
  • Stubbed out other submodules for further development

Merge request reports