StarWind have always been known for their high quality products and for their interesting free software solutions and now they have added another interesting application to the free software collection. It is called rPerf and it has been written to specifically test bandwidth and latency on RDMA connections.
What is an RDMA connection?

Well, it is a lot like a dedicated replication or VMotion network, however, it is designed to allow file copies to bypass CPU’s, caches and so allow file operations to be faster and lower latency as the copy itself is handled by the network cards of the source and destination machines.

It is often called “zero copy networking”, however I feel that this gives it something of an invalid name because It is not “zero copy” from the point of few of any data moving from A to B but it is zero copy from the point of view of the CPU.
In normal copy operations, the network traffic is actually passed to the CPU and the CPU makes the decision on what the copy needs at that time and then passes that on to the CPU. Not only does this soak up some CPU cycles, it also creates latency, as the data has to be passed from the network card, across the motherboard, up to the CPU and then back down again. By offloading that decision making to the network card, it allows the CPU to free up some CPU cycles and reduces latency because the data no longer has to have that journey from the nic to the CPU. It all stays on the NIC.

As you can see, it is zero copy but only from the CPU. The network cards still have to transfer data between each other.

Mellanox, who make RDMA compatible network cards have a pretty good write up on how RDMA works here

The rPerf tool can be downloaded from https://www.starwindsoftware.com/starwind-rperf after filling out a form. The download itself is a zip file that contains two folders, one for windows and one for linux. In turn, each folder contains two files, one is the rpef test tool and the other is an RDMA ping too.

The rPerf tool supports similar parameters to iPerf which makes sense as rPerf is heavily based on iPerf but rPerf is obviously focus on just the RDMA side of things.
In order to get rPerf up and running, you will need an RDMA compatible nic and, at minimum, Windows 7, Windows 2012, CentOS or an Ubuntu operating system.
In the operating system itself, pop the files in something like %homepath% on windows or /tmp on linux. On linux, remember to do a chmod +x rperf and the same for rping otherwise, they will not run.

Using rPerf will need a source and destination machine. On one machine, you will need to setup rPerf in server mode so that it can receive the traffic and on the other server you’ll need to set up rPerf in client mode which will send the data to the server.
With Windows as an example run:

nd_rperf.exe –s –a [ip address]

On the client side, run:

nd_rperf.exe -c -a 10.1.2.12 -C 10000 -S 4096 -q 16 -o R

This test will execute 1000 read interations with a 4096 bye buffer size and queue depth of 16.

The same process works on linux – for the server side use:

./rperf -s -a [ip address]

On the client side, run:

./rperf -c -a 10.1.2.11 -C 100000 -S 4096 -q 16 -o W

There is a good PDF on the starwind site which goes into this in a bit more detail

Testing tools are always good and it is great to see StarWind add a new and useful tool like this to their portfolio.