Tune CephFS MDS Performance

CephFS performance appears to be slow. We would like to find ways to tune the system in order to speed up transfers.

added Backlog label

assigned to @cmcclung

added Sprint 25-06 label and removed Backlog label

The process for this is to add a new ceph-fs unit with juju add-unit ceph-fs --to lxd:3. This will add an lxd container to machine 3.

However, the process is not working because the container never becomes ready. I found it is due to a bug in Netplan 106 where the container never obtains IP addresses. I can resolve this within the container by downgrading neplan, but afterward I'm not able to restart the add-unit process.

I'm currently waiting to hear back from vendor support.

changed title from Tune cephfs mds performance to Tune CephFS MDS Performance

changed the description

I performed tests using fio to get a baseline before making any changes.

#FIO Command used
for cnt in 1 2 4 8 16 24
do
    fio --name=read_throughput --directory=/mnt/cephfs-charmed/share/cmcclung --numjobs=$cnt --size=10G \
        --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
        --direct=0 --verify=0 --bs=4M --iodepth=64 --rw=read \
        --group_reporting=1 \
        >> fio-cephfs-read-blk4M-indirect.log;
done

The results were ~6GB/s read performance:

ubuntu@c0134:~/cmcclung$ cat fio-cephfs-read-blk4M-indirect.log | egrep 'Starting|READ'
Starting 1 process
   READ: bw=649MiB/s (680MB/s), 649MiB/s-649MiB/s (680MB/s-680MB/s), io=38.2GiB (41.0GB), run=60279-60279msec
Starting 2 processes
   READ: bw=1125MiB/s (1180MB/s), 1125MiB/s-1125MiB/s (1180MB/s-1180MB/s), io=65.9GiB (70.8GB), run=60001-60001msec
Starting 4 processes
   READ: bw=2296MiB/s (2407MB/s), 2296MiB/s-2296MiB/s (2407MB/s-2407MB/s), io=135GiB (145GB), run=60037-60037msec
Starting 8 processes
   READ: bw=3360MiB/s (3524MB/s), 3360MiB/s-3360MiB/s (3524MB/s-3524MB/s), io=199GiB (214GB), run=60615-60615msec
Starting 16 processes
   READ: bw=6063MiB/s (6357MB/s), 6063MiB/s-6063MiB/s (6357MB/s-6357MB/s), io=356GiB (382GB), run=60069-60069msec
Starting 24 processes
   READ: bw=6821MiB/s (7152MB/s), 6821MiB/s-6821MiB/s (7152MB/s-7152MB/s), io=401GiB (431GB), run=60260-60260msec

I was also able to observe this on the Ceph side:

  io:
    client:   6.3 GiB/s rd, 4.5 MiB/s wr, 2.86k op/s rd, 501 op/s wr

I'm currently waiting to hear back from vendor support on the LXD container deploy issue. This is a blocker to adding new ceph-fs units, which will add new MDS daemons.

Once this is complete, I plan to use a new MDS to add a rank to a filesystem, test perf, and then add more MDS in order to achieve an active-active config.

added Sprint 25-07 label and removed Sprint 25-06 label

added Sprint 25-08 label and removed Sprint 25-07 label