Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
G
gpfs-policy
Manage
Activity
Members
Labels
Plan
Issues
14
Issue boards
Milestones
Wiki
Code
Merge requests
4
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Deploy
Package Registry
Operate
Terraform modules
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
rc
gpfs-policy
Merge requests
!7
Resolve "create script wrappers to run notebooks in parallel"
Code
Review changes
Check out branch
Download
Patches
Plain diff
Merged
Resolve "create script wrappers to run notebooks in parallel"
5-create-script-wrappers-to-run-notebooks-in-parallel
into
main
Overview
0
Commits
2
Pipelines
0
Changes
2
Merged
John-Paul Robinson
requested to merge
5-create-script-wrappers-to-run-notebooks-in-parallel
into
main
2 years ago
Overview
0
Commits
2
Pipelines
0
Changes
2
Expand
Closes
#5 (closed)
👍
0
👎
0
Merge request reports
Compare
main
main (base)
and
latest version
latest version
05d7ef92
2 commits,
1 year ago
2 files
+
60
−
0
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
Files
2
Search (e.g. *.vue) (Ctrl+P)
run-list-pickle-nb.sh
0 → 100755
+
53
−
0
Options
#!/bin/bash
#
# run RS notebook with specific data set.
#
#
# if we have gpus load cuda modules
if
[
"
$CUDA_VISIBLE_DEVICES
"
!=
""
]
then
module load cuda11.3/toolkit
fi
module load Anaconda3/2021.11
#module load Singularity
#source /share/apps/rc/software/Anaconda3/2020.11/etc/profile.d/conda.sh
conda activate gpfs-policy
# active openmp if multiple CPUs
#export OMP_NUM_THREADS=${SLURM_NPROCS:-1}
#./watchme.sh ${SLURM_TASK_PID} &
#TAGNAME=slurm-${SLURM_JOBID}
# add tmp dir to path for singularity support
# this is where the interface to the kernel started in singularity is found
#PATH=$PATH:$TMPDIR
#echo $PATH
NB
=
"
$3
"
jobid
=
${
SLURM_ARRAY_JOB_ID
:-
$SLURM_JOB_ID
}
papermillcmd
=
"papermill
\
-p dirname
$1
\
-p glob_pattern
$2
\
-k
${
NBKERNEL
:-
base
}
\
${
NB
}
.ipynb
\
slurm-
${
jobid
}
_
${
SLURM_ARRAY_TASK_ID
}
_
${
NB
}
.ipynb"
echo
$papermillcmd
$papermillcmd
pmreturn
=
$?
# make notebook read only after papermill to avoid corrupting experiment record
#chmod -w ${RESULTSDIR}${NB}_${CNAME}_${DATASET}_slurm-${SLURM_JOBID}.ipynb
if
[
$pmreturn
-ne
0
]
then
echo
"papermill failed"
exit
1
fi
Loading