A100 `amperenodes` communication to User Community

assigned to @wwarr

I need assistance with some open questions @mhanby @jpr

ETA on release of the partitions to the public?
What steps do we still need to take before release?
- Presumably acceptance and performance testing, and integration of the local SSD into workflows.
- I'd also like to ensure we have functional, recent versions of drivers, cuda toolkit (>=11.8), cuDNN and tensorrt.
How many A100s are in amperenodes-medium partition?
Are the virtualized A100s going away?
What is the filesystem path to the local SSD?
Any other quirks about performance that we're aware of? I understand we need to test these things out.

The virtual nodes go away when the physical nodes come online.

Aren't there 3 amperenodes partition: amperenodes-debug, amperenodes and amperenodes-medium? Or are we doing away with debug?

Rich text editor

At this moment, the only nodes in the Ampere partitions are the virtual nodes. The A100 physical compute nodes are not in Slurm.

I plan to have 18 of the nodes ready to test by next Monday (2 nodes are currently experiencing provisioning / networking issues). As for when we add them to amperenodes and amperenodes-medium that's up to the team
The SSD's are not currently provisioned. I need input from the team as to what I should do with them, mount path, etc...
CUDA...
No idea about the virtual nodes
These nodes have AMD CPUs, same as c0202 - c0235
These nodes are in DCBlox, so they'll be less performant with GPFS and for MPI jobs should stay on the same IB subnet

Actually, looking at the current configuration of the nodes, BrightCM provisioned one NVMe as / and other OS partitions, the remaining NVMe and the 2 x 500G drives are un-provisioned:

[root@c0237 ~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 447.1G  0 disk
sdb           8:16   0 447.1G  0 disk
├─sdb1        8:17   0   200M  0 part
└─sdb2        8:18   0     2G  0 part
nvme0n1     259:1    0   2.9T  0 disk
├─nvme0n1p1 259:2    0    40G  0 part /
├─nvme0n1p2 259:3    0     6G  0 part /var
├─nvme0n1p3 259:4    0     6G  0 part /tmp
├─nvme0n1p4 259:5    0    12G  0 part [SWAP]
└─nvme0n1p5 259:6    0   2.9T  0 part /local
nvme1n1     259:0    0   2.9T  0 disk

Note that these nodes do not have hardware RAID. We can do software RAID.

Proposal:

2 x 500G drives: Create a software RAID 0 (stripe) that would give the OS ~1TB
2 x 3TB NVMe drives: Create a software RAID 0 (stripe) and mount as /local to provide ~6TB local scratch

Make the os raid 1 mirror.

@ravi89

Good point, thank you for reminding me it existed! I don't know our plan for that. The original intent of amperenodes-debug was for folks moving from P100 to A100. Since we intend to release A100s generally without the previous group restrictions, I don't see a need for a debug partition for that purpose. There may be other valid use cases. My opinion is to do away with it to minimize clutter and noise in partitions.

In any case, it's status needs to be communicated. I'll work to add it in and clarify.

@mhanby

Good point that they aren't available yet. I can reword to take that into consideration, and we can send a second communication when they are released.

For the drives, I am not sure what would be most performant. Would RAID 0 provide improved performance, similar to how DDR works? That is, interleaving reads/writes across two separate channels simultaneously to double the practical throughput. For the sake of completeness, I think that local drive fault tolerance is less of a concern than performance, and to maximize performance we want to ensure A100 computation is the bottleneck, rather than I/O or memory bandwidth.

Generally I agree with your proposal, barring a more performant strategy. For that I would need someone else to weigh in.

Thanks for the other performance considerations, we'll be sure to document them!

RAID 0 (stripe) is more performant than RAID 1 (mirror) and it also aggregates the capacity.

The drawback, if you lose one member of the stripe, you lose all the data. Since /local/scratch is ephemeral, this shouldn't be a concern.

changed the description

I've made changes to the communications based on feedback so far. I've noted the amperenodes-debug partition, how the amperenodes were handled up to now, and a table summarizing the changes. I've added "(complete)" to the parts that are already done to set them apart from what we have yet to do. I like the /local path so much I added it to the sample script.

I've also added a TODO list based on our discussion.

Local mount considerations

Concretely, I propose having a variable like $LOCAL_SCRATCH that points to the proposed /local mount on the A100 nodes. I think we have that variable already, so maybe we could use $LOCAL_PERFORMANCE_BUFFER or something similar, to tie the variable to its use. A bit wordy. Any thoughts?

Other thoughts

Thinking forward, ideally we'd have a consistent mechanism for local mounts to ensure consistency across partitions and over time, to present a homogeneous interface. I think it may be useful to have a shell variable that points either to the local disk (if available) or another reasonable location if not. Alternatively, I can set the example up to check for the existence of the path. Shell variables again, makes checking for unbound variables easier. Thoughts?

The "easy to do right; hard to do wrong" approach is a consistent shell variable with no checking. The second approach is harder for folks unfamiliar with the shell to understand and use, but won't result in unnecessary writing. Too bad shell scripts can't have decorator functions! Or can they?

This also raises the issue of using local mounts in interactive jobs. Researchers will need to copy data the local mount within their code for Jupyter, RStudio and MATLAB. It gets trickier for other applications.

I think in any event we're going to need to educate researchers on how to make the most of these nodes at a conceptual level.

changed the description

I'd like to move in the direction of /local as the base for locally attached disks. Then we can have things like /local/scratch and /local/home when that makes sense.

Ideally we will move towards /local/scratch/$USER and then make it so that slurm cleans that dir up at end of job.

I think we need some programming for that.

@jpr Some thoughts around what will be needed to make the change from using /local as the endpoint for local scratch to /local/scratch

Notify the users of the up coming change
Create a Slurm prolog script that will create /local/scratch/$USER/$JOBID on assigned compute nodes if it doesn't exist (with appropriate $USER:$GROUP and 700 permissions)
Create a Slurm epilog script that will clear out /local/scratch/$USER/$JOBID
Update compute node images with /local/scratch and /local/home
Update the $LOCAL_SCRATCH to point to /local/scratch/$USER (currently it points to /scratch/local)
Update the symlink (or delete it?) /scratch/local -> /local/scratch, (currently symlink'd to /local
Testing to ensure serial, MPI, array and other types of jobs work properly

That sounds like an entire project by itself. Should we turn that comment into a GitLab issue?

changed the description

FYI for broadcast and gather operations on local mounts across multiple nodes:

From GPFS to multiple nodes: sbcast (broadcast)
From multiple nodes to GPFS: sgather

May be useful for multi-node GPU users.

changed the description

closed

A100 `amperenodes` communication to User Community

TODO

For this issue

Timeline

For A100s generally

Release coordination

Designs

Child items ...

Activity

Rich text editor

Local mount considerations

Other thoughts