Load test on ssh proxy node

Perform scp or some other command that can produce some data streaming through sshpiper

added Sprint 25-04 label

changed the description

Some preliminary test laptop off-campus. The first is to the direct endpoint for login004 and the second is for the sshpiper interface. It looks like the sshpiper interface starts off at the same rate ~1.2MiB/s but then drops to about half that throughput steady. As the drop occurs it first drops very low 10-20KiB/s then picks up to the 500-600KiB/s steady state. That suggest we may have a tuning issue to address with sshpiper.

(base) jpr@laptop:~/projects/gpfs-policy$ cat /dev/zero | pv | ssh 164.111.161.27 "cat > /dev/null"
^C59MiB 0:01:58 [1.31MiB/s] [                                   <=>                                                               ]

(base) jpr@laptop:~/projects/gpfs-policy$ cat /dev/zero | pv | ssh cheaha "cat > /dev/null"
^C.2MiB 0:00:34 [ 678KiB/s] [                                                                <=>                                  ]

assigned to @krish94

To replicate the SSH throughput tests on macOS, install pv (Pipe Viewer) using Homebrew:

Install and verify pv:

brew install pv  
pv --version  

As seen on a few such tests, the proxy connection seems to range from 1.08MiB/s to 1.47MiB/s on the network i've been on while testing both login004 and ssh-proxy

➜ cat /dev/zero | pv -N "Direct" -pterab | ssh -T krish94@164.111.161.27 "exec bash --noprofile -c 'cat > /dev/null 2>/dev/null'"
^C   Direct: 19.9MiB 0:00:15 [1.27MiB/s] (1.27MiB/s) [                                     <=>                                                                                                                         ]

➜  cat /dev/zero | pv -N "SSHProxy" -pterab | ssh -T krish94@138.26.48.234 "exec bash --noprofile -c 'cat > /dev/null 2>/dev/null'"
^C   Direct: 76.4MiB 0:00:51 [1.47MiB/s] (1.47MiB/s) [                                                                                                                                                          <=>    ]

To further analyze the performance of the SSH proxy, I conducted tests comparing direct SSH connections with those routed through SSHPiper. Instead of streaming /dev/zero, I used a fixed 20MB test file (20mb-data) to ensure consistency across runs.

Direct SSH Connection:

cat 20mb-data | pv -N "Direct" --numeric --bytes --rate --timer 2>direct.txt | ssh krish94@164.111.161.27 "cat > /dev/null"

SSH Proxy Connection (SSHPiper):

cat 20mb-data | pv -N "Proxy" --numeric --bytes --rate --timer 2>proxy| ssh krish94@138.26.49.58 "cat > /dev/null"

direct.txt proxy.txt

@krish94 your direct.txt data set seems to be incorrect. It's not following format documented in the pv command line parameters.

Command Breakdown

Reads the contents of a 20MB file (20mb-data) and sends it as input to the next command.
This ensures a fixed, consistent test dataset instead of streaming random data.
2. pv -N "Direct" (or "Proxy") --numeric --bytes --rate --timer
pv (Pipe Viewer) monitors data throughput in real-time.
-N "Direct" / -N "Proxy": Labels the output as "Direct" or "Proxy" to distinguish results.
--numeric: Outputs numeric values only, making logs easier to process.
--bytes: Displays total bytes transferred.
--rate: Shows the current transfer speed.
--timer: Records the duration of the transfer.
2>direct1.txt / 2>proxy1.txt: Redirects pv's numeric output to log files for analysis.```

added Sprint 25-05 label and removed Sprint 25-04 label

Quick update on why I chose to generate a 20MB file filled with null bytes (/dev/zero) instead of random data for the SSH proxy load tests:

Short Answer - Consistency. Random data can behave unpredictable. Null bytes keep tests repeatable and fair.

Test File Creation

dd if=/dev/zero of=20mb-data bs=1M count=20  

Command Breakdown: if=/dev/zero: Input File: Reads from /dev/zero, a Linux device that generates an infinite stream of null bytes (0x00).

of=20mb-data: Output File: Saves the generated data to a file named 20mb-data.

bs=1MB: Block Size: Writes data in 1MB chunks.

count=20: Block Count: Creates 20 blocks of 1MB each, resulting in a 20MB file.

Result

20+0 records in  
20+0 records out  
20000000 bytes (20 MB, 19 MiB) copied, 0.0500297 s, 400 MB/s  

Why Use dd? Consistency: Generates identical files for all tests, eliminating content-based variability.

Key Metrics Comparison

Conn.  Test Run	Total Time (s)	Avg Rate (MB/s)
Proxy	Run 1	15.06	~66.4	
Proxy	Run 2	13.12	~76.3	
Direct	Run 1	8.53	~117.2	
Direct	Run 2	8.91	~112.3	

proxy.txt

proxy_1.txt

direct_1.txt

direct.txt

@krish94 Please create a collection of 12 runs each from the proxy and direct so that we can compute reasonable average values. Ideally I'd like these tests reproduced about 30 times so we can a very good average.

I've run it twice each for a 1gb, 5gb and 10gb file, posted the results below, writing a summary of the tests

Earlier tests with 10/20 MB files showed inconsistent results, hence the 1000mb file for testing.

Also running the tests on local machine had different results (depending on the wifi network), so we chose an ubuntu box on the cloud rc space, that ensures consistency in network speed.

pv Version on Ubuntu Box

ubuntu@ubuntu24-04-load-test:~$ pv --version  
pv 1.8.5  

proxy_10gb_1.txt

proxy_10gb_2.txt

direct_10gb_2.txt

proxy_5gb_1.txt

direct_10gb_1.txt

proxy_5gb_2.txt

direct_5gb_1.txt

direct_5gb_2.txt

proxy_1gb_1.txt

proxy_1gb_2.txt

direct_1gb_1.txt

direct_1gb_2.txt

@krish94 in addition to the login004->login005 physical-machine-only path, add results for transferring directly to your ubuntu instance. This will provide insight on the ssh performance between a client and a node on ssh without a proxy jump.

Also, please be sure to diagram the testing scenarios in excalidraw. This will highlight which network segments and machines are used in each test scenario.

added Sprint 25-06 label and removed Sprint 25-05 label

added Sprint 25-07 label and removed Sprint 25-06 label

@krish94 I've been meaning to share this reference. Here is a notebook I created to assess power use reported by bright cluster nodes.

https://gitlab.rc.uab.edu/jpr/power-study/-/blob/master/power-stats.ipynb

The specifics of the notebook aren't important but it's built on time series data tutorial that should be helpful for creating good data frame for plotting the transfer rates. The tutorial discusses creating averages over different time scales. That's less important in this issue than simply creating a single time series with data filled in at regular intervals.

https://www.dataquest.io/blog/tutorial-time-series-analysis-with-pandas/

Hope this is useful.

pushing all the code relating to this ticket in https://gitlab.rc.uab.edu/rc/load-test-for-proxy-node

here’s the results i have so far for 4 test scenarios.

@jpr could you have a look at these results

1. Direct login to login004
2. Login to proxy node
3. Login to login004 via a jumphost
4. Login from login004 to login005

This is the box plot of the average/run for 30 runs per test scenario.

Good plots.

Let's add a Direct transfer to a VM. So in your config that's likely direct from your test client to the VM that serves as your jumphost, but don't jump anywhere. This is just a single SSH connection transit test.