Refactoring run-mmpol scripts

This encompasses many changes to the submit-pol-job and run-mmpol.sh that were originally pushed to rel_v0.1.1-1 but that branch got messed up. Reverting was going to be more work than creating a new release version. The level of changes in this merge have the qualities of a new minor release in any case so is warranted. All of the changes over three prior merge requests are listed below.

Major changes:

• Added getopt functionality allowing the user to specify the following variables in the function call:
  • outdir: parent directory for the log to be placed in
  • output-log-prefix: the prefix to add to the policy file name to specify extra metadata
  • policy-file: path to the policy file
  • device: path or name of the target of the policy
• Removed use of environment variables not explicitly set in the run-mmpol.sh script. These were also removed from the submit-pol-job script and replaced with the corresponding values to be explicitly passed
• default policy file prefix is automatically created from the device path or name if not passed as an argument
• Set user:group ownership to $USER:atlab.
• Set permissions to rws for the containing directory and r-- for the log itself
• Automatically gzip the log file
• Implements changes suggested in #17 (closed). run-mmpol now creates a single directory to store the raw policy output, the split gz files, and the parquet files plus anything else associated with the run. The directory name matches the name of the log file.

Minor changes:

• Changed some variable names to maintain consistency with related scripts
• Full device path is used in the prefix, changing / to -. All metadata entries in the file/directory names are separated by _ for clarity
• Added dry-run option to all 3 scripts for testing. Dry runs submit no jobs and make no changes to any directory structure.
• Changed methods for finding the raw log once it's created and moving/renaming it
• Removed default device from run-mmpol.sh. Now the device must be specified regardless of which script is run.

Syntax Changes:

• Encased all called variables in "{}" for clarity in variable substitution.
• Replaced ```` syntax with $() when performing command substitution

Issues Fixed:

• Fixes #26 (closed)
• Fixes #25 (closed)
• Fixes #24 (closed)
• Fixes #17 (closed)

src/run-policy/run-mmpol.sh

@@ -12,6 +12,7 @@ outdir="/data/rc/gpfs-policy/data"
 policy_file="./policy-def/list-path-external"
 device="scratch"
 output_log_prefix=""
+dry_run=""
 
 ############################################################
 # Help                                                     #
@@ -33,7 +34,9 @@ Runs mmapplypolicy on the specified device/fileset. The policy file dictates the
 Usage: $0 [ -h ] [ -o | --outdir ] [ -f | --output-prefix ] [ -P | --policy-file ] device
 
 options:
-    -h|--help       Print this Help.
+    -h|--help           Print this Help.
+    --dry-run           Do not run the policy command or any command that modifies any files or directories such as 
+                            mkdir
 
 Required:
     device              GPFS fileset/directory apply the policy to. Can be 
@@ -53,7 +56,7 @@ EOF
 exit 0
 }
 
-args=$(getopt -a -o ho:f:P: --long help,outdir:,output-prefix:,policy-file: -- "$@")
+args=$(getopt -a -o ho:f:P: --long help,outdir:,output-prefix:,policy-file:,dry-run -- "$@")
 
 if [[ $? -gt 0 ]]; then
   usage
@@ -67,7 +70,8 @@ do
     -h | --help)            help                    ;;
     -o | --outdir)          outdir=$2               ; shift 2 ;;
     -f | --output-prefix)   output_log_prefix=$2    ; shift 2 ;;
-    -P | --policy-file)     policy_file=$2           ; shift 2 ;;
+    -P | --policy-file)     policy_file=$2          ; shift 2 ;;
+    --dry-run)              dry_run=true            ; shift 1 ;;
     --) shift; break ;;
     *) >&2 echo Unsupported option: $1
        usage ;;
@@ -86,8 +90,8 @@ if [[ -z "$device" ]]; then
     usage
 fi
 
-# create default output_log_prefix if not specified in the arguments
-if [[ -z "$output_log_prefix"]]; then
+# set default output_log_prefix if not specified in the arguments
+if [[ -z "$output_log_prefix" ]]; then
     modified_device=$(echo "$device" | sed -e 's|^/||' -e 's|/$||' -e 's|/|-|g')
     output_log_prefix="list-policy_${modified_device}"
 fi
@@ -95,7 +99,6 @@ fi
 # create temporary working directory for list aggregation
 tmpglobal=$outdir/slurm-tmp-${SLURM_JOBID}
 tmpscratch=$outdir/slurm-tmp-${SLURM_JOBID}
-mkdir -p $tmpglobal
 
 nodes=`scontrol show hostnames "${SLURM_JOB_NODELIST}" | tr '\n' ',' | sed -e 's/,$//'`
 cores="${SLURM_CPUS_PER_TASK}"
@@ -112,19 +115,23 @@ cmd="mmapplypolicy ${device} -I defer \
   -f ${outdir}/list-${SLURM_JOBID} \
   -M FILEPATH=${device} \
   -M JOBID=${SLURM_JOBID} \
-  -M LIST_OUTPUT_FILE=${output_prefix} \
+  -M LIST_OUTPUT_FILE=${output_log_prefix} \
   -N ${nodes} -n ${cores} -m ${cores}"
 
 # report final command in job log
 echo $cmd
 
-# run policy command
-$cmd
-
-# tag output file with run metadata
-outfile=`ls -t $tmpglobal | head -1`
-if [[ "$outfile" != "" ]]
-then
-   mv -n $tmpglobal/$outfile $tmpglobal/../${output_log_prefix}_$filetag
-fi
-rmdir $tmpglobal
+if [[ -z "${dry_run}" ]]; then
+    mkdir -p $tmpglobal
+    
+    # run policy command
+    $cmd
+    
+    # tag output file with run metadata
+    outfile=`ls -t $tmpglobal | head -1`
+    if [[ "$outfile" != "" ]]
+    then
+       mv -n $tmpglobal/$outfile $tmpglobal/../${output_log_prefix}_$filetag
+    fi
+    rmdir $tmpglobal
+fi
\ No newline at end of file