Difference between revisions of "VPP/Tutorial DPDK and MacSwap"

Latest revision as of 14:20, 31 January 2018

This tutorial will cover several basic aspects of VPP, namely: using DPDK interfaces, connecting interfaces at layer 2, experimenting with packet traces and demonstrating VPP modularity by compiling/using a sample plugin.

It is intended to people with little experience of VPP. However, having followed e.g. the tutorial on routing and switching can be helpful to have a first insight on VPP's internals.

Prerequisites

Having a working Linux VPP environment. The easiest way to get it done is to install Vagrant and build a test image.

Using DPDK interfaces

Configuring Vagrant and start the VM

If you have not changed your Vagrantfile, Vagrant should be configured to use two NICs on your VPP virtual machine, belonging to a private network on your host OS. Make sure your file looks like the following.

host:~/vpp/build-root/vagrant$ cat Vagrantfile 
 ... snip ...

 # Define some physical ports for your VMs to be used by DPDK
 nics = 2
 if ENV.key?('VPP_VAGRANT_NICS')
   nics = ENV['VPP_VAGRANT_NICS'].to_i(10)
 end
 for i in 1..nics
   config.vm.network "private_network", type: "dhcp"
 end
 
 ... snip ...

Also, make sure promiscuous mode is enabled for your VM NICs. With Virtualbox, this can be achieved with [Your VM] -> Settings -> Network -> Adapter X -> Promiscuous Mode -> Allow All

Another option is to set the VM NICs in promiscuous mode when updating your Vagrantfile:

 --- a/build-root/vagrant/Vagrantfile
 +++ b/build-root/vagrant/Vagrantfile
 @@ -61,6 +61,8 @@ Vagrant.configure(2) do |config|
    config.vm.synced_folder "../../", "/vpp", disabled: false
    config.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--ioapic", "on"]
 +      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
 +      vb.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"]
        vb.memory = 4096
        vb.cpus = 2
    end

You can now boot up and access your virtual machine. If your machine was already running, please restart it so as to be in a clean state.

host:~/vpp/build-root/vagrant$ vagrant halt
host:~/vpp/build-root/vagrant$ vagrant up
host:~/vpp/build-root/vagrant$ vagrant ssh

Checking the host-only network

The private network is accessible from your host OS via the interface vboxnet0. On your guest, you should see three Intel e1000 NICs, the first one being the management interface (NATed to your host), and the two other ones belonging two the host-only private network.

vagrant@localhost:~$ sudo lshw -class network -businfo
Bus info          Device     Class      Description
===================================================
pci@0000:00:03.0  eth0       network    82540EM Gigabit Ethernet Controller
pci@0000:00:08.0  eth1       network    82540EM Gigabit Ethernet Controller
pci@0000:00:09.0  eth2       network    82540EM Gigabit Ethernet Controller

Make sure your host can communicate with your VPP virtual machine. For this, get the address of the first NIC of your VM and ping it from your host.

vagrant@localhost:~$ ifconfig eth1
 eth1      Link encap:Ethernet  HWaddr 08:00:27:69:dc:cc  
           inet addr:172.28.128.5  Bcast:172.28.128.255  Mask:255.255.255.0
           inet6 addr: fe80::a00:27ff:fe69:dccc/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:9 errors:0 dropped:0 overruns:0 frame:0
           TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:4284 (4.2 KB)  TX bytes:2076 (2.0 KB)
 
host:~/vpp/build-root/vagrant$ ping 172.28.128.5
PING 172.28.128.5 (172.28.128.5): 56 data bytes
64 bytes from 172.28.128.5: icmp_seq=0 ttl=64 time=0.955 ms
64 bytes from 172.28.128.5: icmp_seq=1 ttl=64 time=0.289 ms
64 bytes from 172.28.128.5: icmp_seq=2 ttl=64 time=0.248 ms
64 bytes from 172.28.128.5: icmp_seq=3 ttl=64 time=0.194 ms
^C
--- 172.28.128.5 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.194/0.421/0.955/0.310 ms

Attaching a DPDK interface to VPP

By default, VPP will not use attempt to use those interfaces that are in use by the kernel. In order for VPP to bind an interface to DPDK, you need set it down, then whitelist in the configuration file by supplying the corresponding PCI address.

Do this with eth1 (which in this case has PCI address 0000:00:08.0):

vagrant@localhost:~$ sudo ifconfig eth1 down
vagrant@localhost:~$ cat /etc/vpp/startup.conf 
unix {
  nodaemon
  log /var/log/vpp/vpp.log
  full-coredump
}

api-trace {
  on
}

dpdk {
  socket-mem 1024
  dev 0000:00:08.0
}

You can now start VPP:

vagrant@localhost:~$ sudo start vpp

From this point on, you can execute CLI commands with sudo vppctl <command>. Alternatively, sudo vppctl opens a VPP prompt in which multiple commands can be typed.

Configuring the interface in VPP

Your NIC should be identified as GigabitEthernet0/8/0 by VPP (the name being directly derived from the NIC PCI address).

vpp# show hardware
              Name                Idx   Link  Hardware
GigabitEthernet0/8/0               5    down  GigabitEthernet0/8/0
  Ethernet address 08:00:27:69:dc:cc
  Intel 82540EM (e1000)
    carrier up full duplex speed 1000 mtu 9216 

local0                             0    down  local0
  local
pg/stream-0                        1    down  pg/stream-0
  Packet generator
pg/stream-1                        2    down  pg/stream-1
  Packet generator
pg/stream-2                        3    down  pg/stream-2
  Packet generator
pg/stream-3                        4    down  pg/stream-3
  Packet generator

Set it up and assign it an IP address in the private network subnet:

vpp# set interface state GigabitEthernet0/8/0 up
vpp# set interface ip address GigabitEthernet0/8/0 172.28.128.5/24

Host to VPP communication

You should now be able to ping the VPP interface from your host OS:

host:~/vpp/build-root/vagrant$ ping 172.28.128.5
PING 172.28.128.5 (172.28.128.5): 56 data bytes
64 bytes from 172.28.128.5: icmp_seq=0 ttl=64 time=0.191 ms
64 bytes from 172.28.128.5: icmp_seq=1 ttl=64 time=0.211 ms
^C
--- 172.28.128.5 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.191/0.201/0.211/0.010 ms

From VPP, you can see that pings have indeed reached the interface and be replied to:

vpp# show interface GigabitEthernet0/8/0
              Name               Idx       State          Counter          Count     
GigabitEthernet0/8/0              5         up       rx packets                     3
                                                     rx bytes                     256
                                                     tx packets                     3
                                                     tx bytes                     256
                                                     ip4                            2
vpp# show error
   Count                    Node                  Reason
         2             ip4-icmp-input             echo replies sent
         1                arp-input               ARP replies sent

Layer-2 cross-connection

In this section, we will see how to cross-connect two interfaces, such that traffic coming to one is redirected to the other. We will create a tap interface on VPP and connect it to our DPDK interface, enabling the host OS to communicate with the guest OS while going through VPP.

Set up the tap interface

First, restart VPP to clean up your previous work.

vagrant@localhost:~$ sudo restart vpp

Create a tap interface in VPP. This will spawn an interface named tap-0 inside VPP and tap0 in the guest OS.

vpp# tap connect tap0
tap-0

In the guest OS, configure the corresponding interface.

vagrant@localhost:~$ sudo ifconfig tap0 172.28.128.42/24

If you have an eth2 interface attached to vboxnet0 that has an address in the same subnet, disable it with sudo ifconfig eth2 down, otherwise packets might go through it.

Cross-connect DPDK and tap

You can now cross-connect the DPDK interface and the newly created tap interface. Any traffic arriving on one interface will be redirected to the other.

vpp# set interface l2 xconnect tap-0 GigabitEthernet0/8/0
vpp# set interface l2 xconnect GigabitEthernet0/8/0 tap-0
vpp# set interface state GigabitEthernet0/8/0 up                             
vpp# set interface state tap-0 up

In this setup, you can now send traffic from your host OS vboxnet0 interface to your guest OS tap0 interface, going through the following path: host vboxnet0 -> VPP GigabitEthernet0/8/0 -> VPP tap-0 -> guest tap0.

Host to guest tap communication

You should now be able to ping your guest tap interface from your host.

host:~/vpp/build-root/vagrant$ ping 172.28.128.42
PING 172.28.128.42 (172.28.128.42): 56 data bytes
64 bytes from 172.28.128.42: icmp_seq=0 ttl=64 time=0.241 ms
64 bytes from 172.28.128.42: icmp_seq=1 ttl=64 time=0.223 ms
^C
--- 172.28.128.42 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.223/0.232/0.241/0.009 ms

This time, you can see that packets have flown through the L2 module of VPP:

vpp# show error
   Count                    Node                  Reason
         8                tapcli-rx               no error
         8                l2-output               L2 output packets
         8                l2-input                L2 input packets

Packet tracing

An interesting debug feature of VPP is the packet tracer, which allows a future given number of packets coming from an input node to be recorded. Let us trace the packets coming from the DPDK module in our previous setup:

vpp# trace add dpdk-input 10
host:~/vpp/build-root/vagrant$ ping 172.28.128.42
vpp# show trace
------------------- Start of thread 0 vpp_main -------------------
Packet 1 

00:07:52:290879: dpdk-input
  GigabitEthernet0/8/0 rx queue 0
  buffer 0x1039a: current data 0, length 98, free-list 0, totlen-nifb 0, trace 0x0
  PKT MBUF: port 0, nb_segs 1, pkt_len 98
    buf_len 2176, data_len 98, ol_flags 0x0,
    packet_type 0x0
  IP4: 0a:00:27:00:00:00 -> 08:00:27:ce:15:49
  ICMP: 172.28.128.1 -> 172.28.128.42
    tos 0x00, ttl 64, length 84, checksum 0x4c09
    fragment id 0xd63b
  ICMP echo_request checksum 0xf6af
00:07:52:290910: ethernet-input
  IP4: 0a:00:27:00:00:00 -> 08:00:27:ce:15:49
00:07:52:290915: l2-input
  l2-input: sw_if_index 5 dst 08:00:27:ce:15:49 src 0a:00:27:00:00:00
00:07:52:290917: l2-output
  l2-output: sw_if_index 6 dst 08:00:27:ce:15:49 src 0a:00:27:00:00:00
00:07:52:290918: tap-0-output
  tap-0
  IP4: 0a:00:27:00:00:00 -> 08:00:27:ce:15:49
  ICMP: 172.28.128.1 -> 172.28.128.42
    tos 0x00, ttl 64, length 84, checksum 0x4c09
    fragment id 0xd63b
  ICMP echo_request checksum 0xf6af 

You can see that the path taken by the first packet is dpdk-input -> ethernet-input -> l2-input -> l2-output -> tap-0-output: the two interfaces are indeed directly connected.

To trace packets coming from the tap interface, use trace add tapcli-rx 10. To clear the trace, use clear trace.

You can trace packets coming from different nodes by using several trace add commands, they will be put in their arrival order in the trace buffer.

Cleaning DPDK interfaces

If you use VPP with a DPDK interface and later decide to stop VPP and use the NIC normally through the Linux stack, you will need to bind it back to its generic PCI driver. To that purpose, you can use the dpdk_nic_bind.py Python script, which requires the driver name and the address of the PCI interface. If you use the default VirtualBox setup, the driver will be e1000 (the standard Intel Gigabit Ethernet driver).

vagrant@localhost:$ sudo stop vpp
vagrant@localhost:$ ifconfig eth1
eth1: error fetching interface information: Device not found
vagrant@localhost:$ sudo /vpp/build-root/build-vpp-native/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py -b e1000 0000:00:08.0
vagrant@localhost:$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:69:dc:cc  
          inet addr:172.28.128.5  Bcast:172.28.128.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe69:dccc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:590 (590.0 B)  TX bytes:850 (850.0 B)

MacSwap plugin

In this part, we are going to see how to compile and use a plugin for VPP. We are going to work with the sample macswap plugin, whose role is simply to reverse source and destination hardware addresses of packets arriving on an interface, before retransmitting them on the same one.

Compilation

Compile the plugin:

vagrant@localhost:~$ cd /vpp/
vagrant@localhost:/vpp$ cp -r plugins/sample-plugin ./
vagrant@localhost:/vpp$ cd build-root
vagrant@localhost:/vpp/build-root$ make V=0 PLATFORM=vpp TAG=vpp sample-plugin-install

The plugin is now located in /vpp/build-root/install-vpp-native/sample-plugin/lib64/sample_plugin.so

Loading the plugin

To load the plugin, you can use the plugin-path directive in VPP startup file, specifying the directory where the plugin lies.

vagrant@localhost:/vpp/build-root$ mkdir ~/plugins
vagrant@localhost:/vpp/build-root$ cp install-vpp-native/sample-plugin/lib64/sample_plugin.so ~/plugins/
vagrant@localhost:/vpp/build-root$ echo "plugin_path /home/vagrant/plugins/" | sudo tee -a /etc/vpp/startup.conf

Alternatively, you can copy the .so file to /usr/lib/vpp_plugins, which is VPP's default path for plugin search.

vagrant@localhost:/vpp/build-root$ sudo mkdir /usr/lib/vpp_plugins
vagrant@localhost:/vpp/build-root$ sudo cp install-vpp-native/sample-plugin/lib64/sample_plugin.so /usr/lib/vpp_plugins/

You can now start VPP:

 vagrant@localhost:~$ sudo ifconfig eth1 down
 vagrant@localhost:~$ sudo start vpp

Alternatively, if you chose to start VPP manually, you will see in the output that the first thing done by VPP is to load the plugin:

vagrant@localhost:~$ sudo vpp -c /etc/vpp/startup.conf 
vlib_plugin_early_init:201: plugin path /home/vagrant/plugins/
load_one_plugin:87: Loaded plugin: /home/vagrant/plugins//sample_plugin.so
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Support maximum 256 logical core(s) by configuration. 
...

Using the plugin

Check that the plugin is correctly loaded:

vpp# sample ?
  sample macswap                           sample macswap <interface-name> [disable]

As you can see, this plugin adds a CLI command which enables/disables the MAC swapping feature on a specified interface. Let's try this on our DPDK interface:

vpp# sample macswap GigabitEthernet0/8/0
vpp# set interface state GigabitEthernet0/8/0 up

Now, let's trace what's arriving on our DPDK interface, while generating packets from the host. We will simply use ping to generate traffic from the host to VPP.

vpp# trace add dpdk-input 10
host:~/vpp/build-root/vagrant$ ping 172.28.128.5
vpp# show trace
------------------- Start of thread 0 vpp_main -------------------
Packet 1 

00:05:07:349249: dpdk-input
  GigabitEthernet0/8/0 rx queue 0
  buffer 0x10f07: current data 0, length 60, free-list 0, totlen-nifb 0, trace 0x0
  PKT MBUF: port 0, nb_segs 1, pkt_len 60
    buf_len 2176, data_len 60, ol_flags 0x0,
    packet_type 0x0
  ARP: 0a:00:27:00:00:00 -> ff:ff:ff:ff:ff:ff
  request, type ethernet/IP4, address size 6/4
  0a:00:27:00:00:00/172.28.128.1 -> 00:00:00:00:00:00/172.28.128.5
00:05:07:349703: sample
  SAMPLE: sw_if_index 5, next index 0
00:05:07:349724: GigabitEthernet0/8/0-output
  GigabitEthernet0/8/0
  ARP: ff:ff:ff:ff:ff:ff -> 0a:00:27:00:00:00
  request, type ethernet/IP4, address size 6/4
  0a:00:27:00:00:00/172.28.128.1 -> 00:00:00:00:00:00/172.28.128.5
00:05:07:349980: GigabitEthernet0/8/0-tx
  GigabitEthernet0/8/0 tx queue 0
  buffer 0x10f07: current data 0, length 60, free-list 0, totlen-nifb 0, trace 0x0
  ARP: ff:ff:ff:ff:ff:ff -> 0a:00:27:00:00:00
  request, type ethernet/IP4, address size 6/4
  0a:00:27:00:00:00/172.28.128.1 -> 00:00:00:00:00:00/172.28.128.5

In this example, you can see that the path taken by the packet is: dpdk-input -> sample -> GigabitEthernet0/8/0-output. This shows that the plugin has created a sample node, that traps all packets destined to the GigabitEthernet0/8/0 interface. If you run Wireshark on your host OS, you will see that the packet has been sent back on the vboxnet0 interface with hardware addresses reversed:

1	0.000000	0a:00:27:00:00:00	Broadcast		ARP	42	Who has 172.28.128.5? Tell 172.28.128.1
2	0.000097	Broadcast		0a:00:27:00:00:00	ARP	42	Who has 172.28.128.5? Tell 172.28.128.1

In addition to creating a CLI command and a graph node, the plugin also creates an error counter in order to keep track of the number of packets that it processed:

vpp# show error
   Count                    Node                  Reason
         1                 sample                 Mac swap packets processed

Cleanup

Finally, you can disable the packet interception with:

vpp# sample macswap GigabitEthernet0/8/0 disable

Note however that the plugin will remain loaded until you restart VPP.