Skip to content
Snippets Groups Projects
Commit 69c7beda authored by Matthew K Defenderfer's avatar Matthew K Defenderfer
Browse files

no longer create spare script for sbatch

parent efec8d47
1 merge request!8Automate conversion of GPFS policy outputs to parquet without Jupyter
#!/bin/bash
#
#SBATCH --job-name=parquet-list-%a
#SBATCH --ntasks=1
#SBATCH --partition=amd-hdr100
#SBATCH --time=02:00:00
#SBATCH --mem=16G
#SBATCH --output=out/%A_%a.out
#SBATCH --error=err/%A_%a.err
#SBATCH --array=1-111
module load Anaconda3
conda activate gpfs
log=$(ls /data/rc/gpfs-policy/data/list-policy_data-user_2024-07-23/list-* | awk "NR==${SLURM_ARRAY_TASK_ID} { print $1 }")
python convert-to-parquet.py -o /home/mdefende/Desktop/test_data_user_parquet -f ${log}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment