From fd.io
Jump to: navigation, search

High Speed container communications using VPP Communications Library (VCL)

Using the VPP Communications Library and an LD_PRELOAD library for standard POSIX sockets, VPP can demonstrate a 2.75x speed improvement over traditional Linux bridge.



Environment Setup and Building

  • Clone VPP and the VPPSB projects.
  • Edit vppsb/vcl-ldpreload/env.sh
    • Set VCL_BASE to directory above VPP and VPPSB source. Leave all other env vars in this file alone.
 source ./env.sh
  • Change to VPP source directory and build
 cd $VPP_DIR
  • Modify uri.am to enable socket_test program
 perl -pi -e 's/noinst_PROGRAMS/bin_PROGRAMS/g' $VPP_DIR/src/uri.am
  • Build VPP release
 make install-dep wipe-release bootstrap dpdk-install-dev build-release
  • Build LD_PRELOAD library against VPP build above.

This does not install the LD_PRELOAD library in your system. Instead it will be referenced from the build directory set in VCL_LDPRELOAD_LIB

 cd $LDP_DIR/vcl-ldpreload/src
 autoreconf -i -f
 ./configure VPP_DIR=$VPP_DIR

Running the demo

  • Run test script without parameters to see help menu:
 cd $VPP_DIR/test/scripts

Single Host Tests

These demos launch xterms (or xfce4-terminals), and have been tested on Ubuntu16.04.

Connecting two Docker Containers

  • Docker iPerf using default Linux Bridge
 ./socket_test.sh -bi docker-kernel
  • Docker iPerf using VCL-LDPRELOAD and VPP
 ./socket_test.sh -bi docker-preload

To quit, close the xterms and run following docker kill command

WARNING: This will kill all docker containers on your system. The command below is deliberately commented out to protect the innocent, and copypasta-happy.

 #docker kill $(docker ps -q)

Multi-host Tests

VCL-Client/VCL-Server on Vagrant VMs

Create two vagrant VMs, build VPP in each, and run the VCL-Client sock_test_client on one and connect to the VCL-Server sock_test_server on the other.

 cd $WS_ROOT  # VPP workspace root directory
 cd extras/vagrant
 cp Vagrantfile.vcl_test Vagrantfile
 vagrant up

Enjoy a cup of coffee, tea, water, donut, bagel or scooby snack while the VM's are created and build VPP.

  • VCL Socket Test Client/Server on default Linux Bridge

Start the VCL-Server sock_test_server application on the vcl-server VM:

 vagrant ssh vcl-server
 cd $WS_ROOT
 $WS_ROOT/test/scripts/socket_test.sh -bm s native-kernel
 ifconfig enp0s8

In another terminal, run the VCL-Client sock_test_client application on the vcl-client VM:

 cd $WS_ROOT/extras/vagrant
 vagrant ssh vcl-client
 cd $WS_ROOT
 $WS_ROOT/test/scripts/socket_test.sh -bm c -S <server ip address> native-kernel

  • VCL Socket Test Client/Server over VPP

Start the VCL-Server vcl_test_server application on the vcl-server VM:

 vagrant ssh vcl-server
 cd $WS_ROOT
 $WS_ROOT/test/scripts/socket_test.sh -tbm s native-vcl

Note: Vagrant configures the NICs to acquire an ip address using DHCP. socket_test.sh configures VPP with the address of enp0s8 or another interface specified by the '-n' option or a default address (e.g. It prints the address that VPP uses to the terminal for use on the VCL-Client VM.

In another terminal, run the VCL-Client vcl_test_client application on the vcl-client VM:

 cd $WS_ROOT/extras/vagrant
 vagrant ssh vcl-client
 cd $WS_ROOT
 $WS_ROOT/test/scripts/socket_test.sh -tbm c -S <server ip address> native-vcl

Note: If VPP failed to exit cleanly, then enp0s8 may not have been successfully re-bound to the kernel driver. In order to fix this issue, run the following dpdk python script:

 sudo /usr/share/dpdk/usertools/dpdk-devbind.py -b e1000 0000:00:08.0

  • Post testing cleanup.

After completing testing, issue the following command to clean up the vagrant VM's:

 cd $WS_ROOT/extras/vagrant
 vagrant destroy -f
 git checkout -- Vagrantfile

Note: The vagrant VM infrastructure is stored in $WS_ROOT/extras/vagrant/.vagrant, therefore beware of the use of 'git clean -fdX' in the VPP workspace when running vagrant VM's as it will unceremoniously delete it (as you asked it to!). However, don't panic if you forget and run git clean -- to recover, execute virtualbox to shut down and remove the remnants of the VMs. Then go enjoy some more coffee, tea, water, donuts or scooby snacks :-)