VPP/HostStack/LDP/iperfLast updated 4 months ago

iperf3 with LD_PRELOAD

Example of how to run iperf3 via ldp and vcl on top of vpp's host stack. This was last tested with iperf 3.1.3 on Ubuntu 18.04.1.

To run the test two hosts with networking connectivity are needed for the client and server instances of iperf.

VPP configuration

In addition to the typical startup config parameters, the host stack requires the ones lower for enabling the use of memfds segments for vpp message queues and to configure the binary api socket transport. For a list of startup parameters see here. This is needed for both vpp instances.

session { evt_qs_memfd_seg  }
socksvr { socket-name /tmp/vpp-api.sock}

Run iperf

First start the two vpp instances and ensure that the network between them is functional. The simplest option would be to use the vpp builtin ping utility. Then, on both hosts, define the following variables, with the appropriate paths.

VCL_CFG=/path/to/vcl.conf
LDP_PATH=/path/to/vpp/build-root/build-vpp-native/vpp/lib/libvcl_ldpreload.so

And create two vcl.conf files:

vcl {
  rx-fifo-size 4000000
  tx-fifo-size 4000000
  app-scope-local
  app-scope-global
  api-socket-name /tmp/vpp-api.sock
}

To start the server:

sudo LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG taskset --cpu-list <core-list> iperf3 -4 -s

To start the client:

sudo LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG taskset --cpu-list <core-list> iperf3 -c

Make sure that the core-list is such selected that it does not overlap vpp's workers but it maintains the same numa.

Recommended half-duplex throughput testing configuration

• 16k mbufs
• 1 thread (main thread) since connection oriented transport protocols like TCP have sessions pinned to a core
• 256 rx/tx-descriptors
• 1 rx-queue, 1 tx-queue

To ensure that main thread runs on the same numa as the nic, first find the numa for the nic with sh hardware and then in startup.conf make sure main-core under cpu is set to a core on the same numa as the nic. To find out what numa a core pertains to use lscpu.