From fd.io
< VPP‎ | HostStack
Jump to: navigation, search

iperf3 with LD_PRELOAD

Example of how to run iperf3 via ldp and vcl on top of vpp's host stack. This was last tested with iperf 3.1.3 on Ubuntu 18.04.1.

To run the test two hosts with networking connectivity are needed for the client and server instances of iperf.

VPP configuration

In addition to the typical startup config parameters, the host stack requires the ones lower for enabling the use of memfds segments for vpp message queues and to configure the binary api socket transport. For a list of startup parameters see here. This is needed for both vpp instances.

session { evt_qs_memfd_seg  }
socksvr { socket-name /tmp/vpp-api.sock}

Run iperf

First start the two vpp instances and ensure that the network between them is functional. The simplest option would be to use the vpp builtin ping utility. Then, on both hosts, define the following variables, with the appropriate paths.


And create two vcl.conf files:

vcl {
  rx-fifo-size 4000000
  tx-fifo-size 4000000
  api-socket-name /tmp/vpp-api.sock

The above configures vcl to request 4MB receive and transmit fifo sizes and access to both local and global session scopes. Additionally, it provides the path to vpp's binary api socket.

To start the server:

sudo taskset --cpu-list <core-list> sh -c "LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG iperf3 -4 -s"

To start the client:

sudo taskset --cpu-list <core-list> sh -c "LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG iperf3 -c"

Make sure that the core-list is such selected that it does not overlap vpp's workers but it maintains the same numa.

TLS testing (not stable)

LDP can convert tcp into tls connections, transparently from an app's perspective, by means of three additional environment variables

  • LDP_TRANSPARENT_TLS must be set to 1
  • LDP_TLS_CERT_FILE must be set to the path to a certificate file
  • LDP_TLS_KEY_FILE must be set to the path to a key file

Once these are configured, iperf + ldp will actually establish and measure throughput of tls connections.

Recommended half-duplex throughput testing configuration

  • cubic as congestion control algorithm: add tcp {cc-algo cubic} to vpp's startup.conf
  • 16k mbufs
  • 1 thread (main thread) since connection oriented transport protocols like TCP have sessions pinned to a core
  • 256 rx/tx-descriptors
  • 1 rx-queue, 1 tx-queue

To ensure that main thread runs on the same numa as the nic, first find the numa for the nic with sh hardware and then in startup.conf make sure main-core under cpu is set to a core on the same numa as the nic. To find out what numa a core pertains to use lscpu.