A100 node SSD configuration, align with other nodes

Let's set up the SSD configuration on the A100 nodes for performance enhancement.

This issue does NOT block A100 node release.

Original issue: #436 (closed), relevant comments copied in thread below.

added Backlog label

added Amperenodes label

From Mike

Actually, looking at the current configuration of the nodes, BrightCM provisioned one NVMe as / and other OS partitions, the remaining NVMe and the 2 x 500G drives are un-provisioned:

[root@c0237 ~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 447.1G  0 disk
sdb           8:16   0 447.1G  0 disk
├─sdb1        8:17   0   200M  0 part
└─sdb2        8:18   0     2G  0 part
nvme0n1     259:1    0   2.9T  0 disk
├─nvme0n1p1 259:2    0    40G  0 part /
├─nvme0n1p2 259:3    0     6G  0 part /var
├─nvme0n1p3 259:4    0     6G  0 part /tmp
├─nvme0n1p4 259:5    0    12G  0 part [SWAP]
└─nvme0n1p5 259:6    0   2.9T  0 part /local
nvme1n1     259:0    0   2.9T  0 disk

Note that these nodes do not have hardware RAID. We can do software RAID.

Proposal:

• 2 x 500G drives: Create a software RAID 0 (stripe) that would give the OS ~1TB
• 2 x 3TB NVMe drives: Create a software RAID 0 (stripe) and mount as /local to provide ~6TB local scratch

From JPR

Make the os raid 1 mirror.

From Mike

RAID 0 (stripe) is more performant than RAID 1 (mirror) and it also aggregates the capacity.

The drawback, if you lose one member of the stripe, you lose all the data. Since /local/scratch is ephemeral, this shouldn't be a concern.

From JPR

I'd like to move in the direction of /local as the base for locally attached disks. Then we can have things like /local/scratch and /local/home when that makes sense.

Ideally we will move towards /local/scratch/$USER and then make it so that slurm cleans that dir up at end of job.

I think we need some programming for that.

** From Mike**

@jpr Some thoughts around what will be needed to make the change from using /local as the endpoint for local scratch to /local/scratch

• Notify the users of the up coming change
• Create a Slurm prolog script that will create /local/scratch/$USER/$JOBID on assigned compute nodes if it doesn't exist (with appropriate $USER:$GROUP and 700 permissions)
• Create a Slurm epilog script that will clear out /local/scratch/$USER/$JOBID
• Update compute node images with /local/scratch and /local/home
• Update the $LOCAL_SCRATCH to point to /local/scratch/$USER (currently it points to /scratch/local)
• Update the symlink (or delete it?) /scratch/local -> /local/scratch, (currently symlink'd to /local
• Testing to ensure serial, MPI, array and other types of jobs work properly

From William

FYI for broadcast and gather operations on local mounts across multiple nodes:

• From GPFS to multiple nodes: sbcast (broadcast)
• From multiple nodes to GPFS: sgather

May be useful for multi-node GPU users.

From Mike

In case you are interested, work notes are here: https://gitlab.rc.uab.edu/rc/cheaha/-/issues/30

End result, the nodes are now partitioned as follows:

• sda, sdb: RAID1 mirror
  • /tmp - 10G
  • /var - 10G
  • Swap - 16G
  • / - max (remainder)
• nvme0n1, nvme1n1: RAID0 stripe
  • /local/scratch - max

Example from c0236

[root@c0236 ~]# df -h / /local/scratch
Filesystem      Size  Used Avail Use% Mounted on
/dev/md3        411G   19G  393G   5% /
/dev/md4        5.9T   34M  5.9T   1% /local/scratch

[root@c0236 ~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0 447.1G  0 disk
├─sda1        8:1    0    10G  0 part
│ └─md0       9:0    0    10G  0 raid1 /tmp
├─sda2        8:2    0    10G  0 part
│ └─md1       9:1    0    10G  0 raid1 /var
├─sda3        8:3    0    16G  0 part
│ └─md2       9:2    0    16G  0 raid1 [SWAP]
└─sda4        8:4    0 411.1G  0 part
  └─md3       9:3    0 411.1G  0 raid1 /
sdb           8:16   0 447.1G  0 disk
├─sdb1        8:17   0    10G  0 part
│ └─md0       9:0    0    10G  0 raid1 /tmp
├─sdb2        8:18   0    10G  0 part
│ └─md1       9:1    0    10G  0 raid1 /var
├─sdb3        8:19   0    16G  0 part
│ └─md2       9:2    0    16G  0 raid1 [SWAP]
└─sdb4        8:20   0 411.1G  0 part
  └─md3       9:3    0 411.1G  0 raid1 /
nvme0n1     259:1    0   2.9T  0 disk
└─nvme0n1p1 259:2    0   2.9T  0 part
  └─md4       9:4    0   5.8T  0 raid0 /local/scratch
nvme1n1     259:0    0   2.9T  0 disk
└─nvme1n1p1 259:3    0   2.9T  0 part
  └─md4       9:4    0   5.8T  0 raid0 /local/scratch

From Mike

I have updated the disk setup (XML) to change the mount point from /local/scratch to /local for the NVMe drives.

At some point in the future, we will revert back.

In the mean time, I have created the /local/scratch directory so that we can begin using it:

dir=/local/scratch; for node in c0{236..255}; do sudo ssh $node "mkdir $dir; chmod 1777 $dir"; done; unset dir

@mhanby @jpr @ravi89

I've separated out the conversation surrounding the nvme drives on the A100 nodes from the original issue (Epics would be nice!) into this issue. In my opinion, we shouldn't let this issue block the A100 release, but still need to work on it. I'll bring it up at the next planning meeting.

From my perspective, the main remaining task is to align all node configs in terms of User Experience when using local drives. The A100s are configured differently, but the researchers shouldn't have to care where /local points. They should be able to change partition, and nothing else, without their sbatch script breaking.