Skip to content
Snippets Groups Projects
Commit 8713a5eb authored by John-Paul Robinson's avatar John-Paul Robinson
Browse files

Add script to split output from policy run

Parameters hard coded to split on 50k enteries per file which provides
reasonable parallel read efficiency for downstream tasks.
Put splits in a .d subdir of file name.
parent 3786cc5a
No related branches found
No related tags found
No related merge requests found
#!/bin/bash
# split an info file into individual list files to make searching with an array job fast
file=$1
dirname=${file}.d
prefix=${dirname}/list-
mkdir -p ${dirname}
srun -p amd-hdr100 --time 06:00:00 --mem 4G split -a 3 -d -l 5000000 ${file} ${prefix}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment