diff --git a/singularity_container.ipynb b/singularity_container.ipynb
index 7b35988f3c937e34474b6e46ab42512023cd7a0c..99539cb72d14d79c2b4f5ce2f48748f09433b069 100644
--- a/singularity_container.ipynb
+++ b/singularity_container.ipynb
@@ -7,6 +7,77 @@
"# SIngularity Containers"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## First container\n",
+ "\n",
+ "Let's begin by pulling our first container from [Singularity-Hub](https://singularity-hub.org/).\n",
+ "\n",
+ "This singularity image contains the tool [neurodebian](http://neuro.debian.net/).\n",
+ "\n",
+ "NeuroDebian provides a large collection of popular neuroscience research software for the Debian operating system as well as Ubuntu and other derivatives."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!singularity pull shub://neurodebian/neurodebian"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now that we have pulled the image you should be able to see that image in your directory by simply running a 'ls' command"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!ls"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, we'll try to execute a command from within the container. \n",
+ "__exec__ parameter allows you to achieve this functionality. Let's list the content of you /home/$USER directory"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!singularity exec neurodebian-neurodebian-master-latest.simg ls /home/$USER"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, that we have the container pulled, we can use it for running the actual tool that we wanted to use, the reason for actually downloading the container in the first place: [dcm2nii](https://www.nitrc.org/projects/mricron)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!singularity exec -B /data neurodebian-neurodebian-master-latest.simg dcm2nii"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -112,18 +183,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Example 1:\n",
- "\n",
- "In this example we are going to be pulling images from [Singularity-Hub](https://singularity-hub.org/). This singularity image contains the tool [neurodebian](http://neuro.debian.net/).\n",
- "\n",
- "NeuroDebian provides a large collection of popular neuroscience research software for the Debian operating system as well as Ubuntu and other derivatives."
+ "## Namespace resolution within the container:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's begin by pulling the [Neurodebian Singularity image](https://singularity-hub.org/collections/209) from Singularity Hub"
+ "Now, let's list the content of your /data/user/$USER directory by running it from within the container. We'll use __exec__ parameter for this."
]
},
{
@@ -132,14 +199,18 @@
"metadata": {},
"outputs": [],
"source": [
- "!singularity pull shub://neurodebian/neurodebian"
+ "!singularity exec neurodebian-neurodebian-master-latest.simg ls /data/user/$USER"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now that we have pulled the image you should be able to see that image in your directory by simply running a 'ls' command"
+ "Hmmm, an error. Remember your singularity container image doesn't know about the directories on your host machine. It by default (in most containers) binds your HOME and tmp directory. \n",
+ "\n",
+ "Now, all our raw data is generally in our /data/user/$USER locations, so we really need to access that location if our container has to be useful. \n",
+ "\n",
+ "Thankfully, you can explicitly tell singularity to bind a host directory to your container image. Singularity provides you with a parameter (-B) to bind path from your host machine to the container. Try the same command again, but with the bind parameter"
]
},
{
@@ -148,14 +219,14 @@
"metadata": {},
"outputs": [],
"source": [
- "!ls"
+ "!singularity exec -B /data/user/$USER neurodebian-neurodebian-master-latest.simg ls /data/user/$USER"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now, we'll try to execute a command from within the container. Remember that exec parameter allows you to achieve this functionality. Let's list the content of you /data/user/$USER directory"
+ "Now like mentioned earlier during the security considerations of Singularity in a HPC environment, all the sigularity runs adhere to the user level permissions, from the host system. So I would get a permission denied issue if I try to list William's directory content."
]
},
{
@@ -164,41 +235,62 @@
"metadata": {},
"outputs": [],
"source": [
- "!singularity exec neurodebian-neurodebian-master-latest.simg ls /data/user/$USER"
+ "!singularity exec -B /data/user/wsmonroe neurodebian-neurodebian-master-latest.simg ls /data/user/wsmonroe"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Hmmm, an error. Remember your singularity container image doesn't know about the directories on your host machine. It by default (in most containers) binds your HOME and tmp directory. So try to run the above command again but change the list path to your HOME directory.\n",
+ "Now that we know how to bind paths from the host machine to the container image, I would be able to use __dcm2nii__ tool with my raw files available in my DATA_USER location.\n",
"\n",
- "Now, all our raw data is generally in our /data/user/$USER locations, so we really need to access that location if our container has to be useful. SIngularity provides you with a parameter (-B) to bind path from your host machine to the container. Try the same command again, but with the bind parameter"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!singularity exec -B /data neurodebian-neurodebian-master-latest.simg ls /data/user/wsmonroe"
+ "\n",
+ "We'll take a look at how to use it within a job script in the next section."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now rather then using ls command from within the container we are going to see one of the softwares within the container: [dcm2nii](https://www.nitrc.org/projects/mricron)"
+ "## Job Script with containers\n",
+ "\n",
+ "Using Singularity container with [SLURM job script](https://docs.uabgrid.uab.edu/wiki/Slurm) is very easy, as the containers run as a process on the host machine. You just need to load Singularity in your job script and run the singularity process. Here's an example job script below:"
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "!singularity exec -B /data neurodebian-neurodebian-master-latest.simg dcm2nii"
+ "```\n",
+ "#!/bin/bash\n",
+ "#\n",
+ "#SBATCH --job-name=test\n",
+ "#SBATCH --output=res.out\n",
+ "#SBATCH --error=res.err\n",
+ "#\n",
+ "# Number of tasks needed for this job. Generally, used with MPI jobs\n",
+ "#SBATCH --ntasks=1\n",
+ "#SBATCH --partition=express\n",
+ "#\n",
+ "# Time format = HH:MM:SS, DD-HH:MM:SS\n",
+ "#SBATCH --time=10:00\n",
+ "#\n",
+ "# Number of CPUs allocated to each task. \n",
+ "#SBATCH --cpus-per-task=1\n",
+ "#\n",
+ "# Mimimum memory required per allocated CPU in MegaBytes. \n",
+ "#SBATCH --mem-per-cpu=100\n",
+ "#\n",
+ "# Send mail to the email address when the job fails\n",
+ "#SBATCH --mail-type=FAIL\n",
+ "#SBATCH --mail-user=$USER@uab.edu\n",
+ "\n",
+ "#Set your environment here\n",
+ "module load Singularity/2.5.2-GCC-5.4.0-2.26\n",
+ "\n",
+ "#Run your commands here\n",
+ "singularity exec -B /data/user/$USER /data/user/$USER/rc-training-sessions/neurodebian-neurodebian-master-latest.simg dcm2nii PATH_TO_YOUR_DICOM_FILES\n",
+ "```"
]
},
{
@@ -278,7 +370,7 @@
"metadata": {},
"outputs": [],
"source": [
- "!singularity pull docker://nvcr.io/hpc/vmd:cuda9-ubuntu1604-egl-1.9.4a17"
+ "!singularity build vmc_gpu.simg docker://nvcr.io/hpc/vmd:cuda9-ubuntu1604-egl-1.9.4a17"
]
},
{
@@ -287,7 +379,7 @@
"metadata": {},
"outputs": [],
"source": [
- "!singularity exec --nv vmd-cuda9-ubuntu1604-egl-1.9.4a17.simg /opt/vmd/bin/vmd"
+ "!singularity exec --nv vmc_gpu.simg /opt/vmd/bin/vmd -dispdev openglpbuffer -e hiv-simple-egloptix-test.vmd"
]
},
{