High Speed container communications using VPP Communications Library (VCL)
Using the VPP Communications Library and an LD_PRELOAD library for standard POSIX sockets, VPP can demonstrate a 2.75x speed improvement over traditional Linux bridge.
- docker (Version: 17.05)
- iperf3 (3.0.11)
- VPP: git clone https://gerrit.fd.io/r/vpp
- VPPSB: git clone https://gerrit.fd.io/r/vppsb
Environment Setup and Building
- Clone VPP and the VPPSB projects.
- Edit vppsb/vcl-ldpreload/env.sh
- Set VCL_BASE to directory above VPP and VPPSB source. Leave all other env vars in this file alone.
- Change to VPP source directory and build
- Modify uri.am to enable socket_test program
perl -pi -e 's/noinst_PROGRAMS/bin_PROGRAMS/g' $VPP_DIR/src/uri.am
- Build VPP release
make install-dep wipe-release bootstrap dpdk-install-dev build-release
- Build LD_PRELOAD library against VPP build above.
This does not install the LD_PRELOAD library in your system. Instead it will be referenced from the build directory set in VCL_LDPRELOAD_LIB
cd $LDP_DIR/vcl-ldpreload/src autoreconf -i -f ./configure VPP_DIR=$VPP_DIR make
Running the demo
- Run test script without parameters to see help menu:
cd $VPP_DIR/test/scripts ./socket_test.sh
Single Host Tests
These demos launch xterms (or xfce4-terminals), and have been tested on Ubuntu16.04.
Connecting two Docker Containers
- Docker iPerf using default Linux Bridge
./socket_test.sh -bi docker-kernel
- Docker iPerf using VCL-LDPRELOAD and VPP
./socket_test.sh -bi docker-preload
To quit, close the xterms and run following docker kill command
WARNING: This will kill all docker containers on your system. The command below is deliberately commented out to protect the innocent, and copypasta-happy.
#docker kill $(docker ps -q)
VCL-Client/VCL-Server on Vagrant VMs
Create two vagrant VMs, build VPP in each, and run the VCL-Client sock_test_client on one and connect to the VCL-Server sock_test_server on the other.
cd $WS_ROOT # VPP workspace root directory cd extras/vagrant cp Vagrantfile.vcl_test Vagrantfile vagrant up
Enjoy a cup of coffee, tea, water, donut, bagel or scooby snack while the VM's are created and build VPP.
- VCL Socket Test Client/Server on default Linux Bridge
Start the VCL-Server sock_test_server application on the vcl-server VM:
vagrant ssh vcl-server cd $WS_ROOT $WS_ROOT/test/scripts/socket_test.sh -bm s native-kernel ifconfig enp0s8
In another terminal, run the VCL-Client sock_test_client application on the vcl-client VM:
cd $WS_ROOT/extras/vagrant vagrant ssh vcl-client cd $WS_ROOT $WS_ROOT/test/scripts/socket_test.sh -bm c -S <server ip address> native-kernel
- VCL Socket Test Client/Server over VPP
Start the VCL-Server vcl_test_server application on the vcl-server VM:
vagrant ssh vcl-server cd $WS_ROOT $WS_ROOT/test/scripts/socket_test.sh -tbm s native-vcl
Note: Vagrant configures the NICs to acquire an ip address using DHCP. socket_test.sh configures VPP with the address of enp0s8 or another interface specified by the '-n' option or a default address (e.g. 10.10.10.10). It prints the address that VPP uses to the terminal for use on the VCL-Client VM.
In another terminal, run the VCL-Client vcl_test_client application on the vcl-client VM:
cd $WS_ROOT/extras/vagrant vagrant ssh vcl-client cd $WS_ROOT $WS_ROOT/test/scripts/socket_test.sh -tbm c -S <server ip address> native-vcl
Note: If VPP failed to exit cleanly, then enp0s8 may not have been successfully re-bound to the kernel driver. In order to fix this issue, run the following dpdk python script:
sudo /usr/share/dpdk/usertools/dpdk-devbind.py -b e1000 0000:00:08.0
- Post testing cleanup.
After completing testing, issue the following command to clean up the vagrant VM's:
cd $WS_ROOT/extras/vagrant vagrant destroy -f git checkout -- Vagrantfile
Note: The vagrant VM infrastructure is stored in $WS_ROOT/extras/vagrant/.vagrant, therefore beware of the use of 'git clean -fdX' in the VPP workspace when running vagrant VM's as it will unceremoniously delete it (as you asked it to!). However, don't panic if you forget and run git clean -- to recover, execute virtualbox to shut down and remove the remnants of the VMs. Then go enjoy some more coffee, tea, water, donuts or scooby snacks :-)