VPP/How To Use The API Trace Tools

Introduction

The VPP platform implements a set of binary APIs that control-plane agents can use to configure the VPP data plane. The VPP engine can trace, print, and replay binary API traces. This is an extremely powerful mechanism. When the data plane fails to pass traffic, any number of factors may be involved. Often, the data plane is simply not configured correctly.

API trace replay is very fast. Replaying 100,000 control-plane operations takes a fraction of a second - even though it may have taken an hour or more to accumulate that many transactions. There's no requirement for you to replay an API trace using the exact binary which generated the trace. It's often handy to add scaffolding to a debug image to work out what's going wrong. As long as the API definitions are consistent, you can add scaffolding, test bug fixes, and so forth.

Capturing API traces

The usual method for capturing a binary API trace is to configure API tracing from the command-line:

... api-trace { on } ...

Current packaging always enables API tracing at data-plane start time. Alternatively, the debug Command-line Interface (CLI) api trace on command can be used at any time.

Using the API Test Tool to Create a Sample API Trace

Start the API test tool, located in /usr/lib/vpp_api_test_plugins . Determine which interface to use. The API test tool exec command sends debug CLI commands to the data-plane, and prints the results:

# vpp_api_test
vat# exec show interface
              Name               Idx       State          Counter          Count     
GigabitEthernet2/1/0              5        down      
local0                            0        down      
pg/stream-0                       1        down      
pg/stream-1                       2        down      
pg/stream-2                       3        down      
pg/stream-3                       4        down

Use the help command in the API test tool to see all of the APIs that it knows about. With a very few exceptions related to uploading statistics, it knows about all of the data plane binary APIs.

# help
Help is available for the following:
add_node_next
bd_ip_mac_add_del
bridge_domain_add_del
bridge_domain_dump
<snip> ## type help yourself to see the entire list...

Create and Save a Real API Trace

Let's do the following: add an interface address on GigabitEthernet2/1/0, enable the interface, and add a static route via a live gateway on the 192.168.1.1/24 network. Lastly, we'll save the resulting API trace.

Use help in the API test tool to see how to add/delete an interface address. Then add an address for GigabitEthernet2/1/0 on the 192.168.1.1/24 network:

vat# help sw_interface_add_del_address
usage: sw_interface_add_del_address <intfc> | sw_if_index <id> <ip4-address> | <ip6-address> [del] [del-all] 

vat# sw_interface_add_del_address GigabitEthernet2/1/0 192.168.1.101/24

Use help to see how to add/delete a route. Then add a route:

vat# help ip_add_del_route
usage: ip_add_del_route <addr>/<mask> via <addr> [vrf <n>]
[<intfc> | sw_if_index <id>] [resolve-attempts <n>]
[weight <n>] [drop] [local] [classify <n>] [del]
[multipath] [count <n>]

vat# ip_add_del_route 1.2.3.4/32 GigabitEthernet2/1/0 via 192.168.1.1 resolve-attempts 3

Verify that all of the above actually worked:

vat# exec sh ip arp
    Time      FIB        IP4      Stat      Ethernet              Interface       
    137.1815   0     192.168.1.1        48:f8:b3:e0:85:1a    GigabitEthernet2/1/0  

vat# exec sh ip fib
Table 0, fib_index 0, flow hash: src dst sport dport proto 
     Destination         Packets          Bytes         Adjacency   
192.168.1.0/24                     3             438 weight 1, index 3
                                                     arp GigabitEthernet2/1/0 192.168.1.101/24
192.168.1.1/32                     0               0 weight 1, index 6
                                                     GigabitEthernet2/1/0
                                                     IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a
192.168.1.101/32                   0               0 weight 1, index 4
                                                     local 192.168.1.101/24
1.2.3.4/32                         0               0 weight 1, index 5
                                                     GigabitEthernet2/1/0
                                                     IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a

Judging by the ARP and FIB tables in the above sample, the API message sequence worked.

If you are working in a development environment, use telnet to see the data plane debug CLI console, and save the API trace:

# telnet 0 5002
vpp# api trace save demo.api
API trace saved to /tmp/demo.api

Pretty-print and Replay the API Trace

Restart the data plane:

# stop vpp
# start vpp

If you are working in a development environment, use telnet to see the data plane debug CLI console, and pretty-print the API trace:

# telnet 0 5002
vpp# api trace custom-dump /tmp/demo.api
SCRIPT: memclnt_create name vpe_api_test 
SCRIPT: sw_interface_dump name_filter Ether 
SCRIPT: sw_interface_dump name_filter lo 
SCRIPT: sw_interface_dump name_filter vxlan 
SCRIPT: sw_interface_dump name_filter l2tpv3_t 
SCRIPT: control_ping 
SCRIPT: cli_request 
SCRIPT: sw_interface_add_del_address sw_if_index 5 192.168.1.101/24 
SCRIPT: sw_interface_set_flags sw_if_index 5 admin-up link-down
SCRIPT: ip_add_del_route sw_if_index 5 1.2.3.4/32 via 192.168.1.1 resolve-attempts 3 
SCRIPT: cli_request 
SCRIPT: cli_request 
vl_api_memclnt_delete_t:
index: 0
handle: 0x3059b0c0

Lines which begin with "SCRIPT: " turn into scripts. More on that in a minute.

Let's replay the API trace, and see what the ARP and IP FIB tables look like:

vpp# api trace replay /tmp/demo.api
vpp# show ip arp
    Time      FIB        IP4      Stat      Ethernet              Interface       
     60.5953   0     192.168.1.1        48:f8:b3:e0:85:1a    GigabitEthernet2/1/0  

vpp# sh ip fib
Table 0, fib_index 0, flow hash: src dst sport dport proto 
     Destination         Packets          Bytes         Adjacency   
192.168.1.0/24                     0               0 weight 1, index 3
                                                     arp GigabitEthernet2/1/0 192.168.1.101/24
192.168.1.1/32                     0               0 weight 1, index 5
                                                     GigabitEthernet2/1/0
                                                     IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a
192.168.1.101/32                   0               0 weight 1, index 4
                                                     local 192.168.1.101/24
1.2.3.4/32                         0               0 weight 1, index 6
                                                     GigabitEthernet2/1/0
                                                     IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a

Identical to the original.

Convert an API Trace Pretty-print Into a vpe_api_test Input Script

Here are the "interesting" API messages, as captured and pretty-printed:

SCRIPT: sw_interface_add_del_address sw_if_index 5 192.168.1.101/24 
SCRIPT: sw_interface_set_flags sw_if_index 5 admin-up link-down
SCRIPT: ip_add_del_route sw_if_index 5 1.2.3.4/32 via 192.168.1.1 resolve-attempts 3

Using a text editor, remove "SCRIPT: ". You are left with the text for the final vpe_api_test input script:

$ cat /tmp/api_script
sw_interface_add_del_address sw_if_index 5 192.168.1.101/24 
sw_interface_set_flags sw_if_index 5 admin-up link-down
ip_add_del_route sw_if_index 5 1.2.3.4/32 via 192.168.1.1 resolve-attempts 3

Execute the vpe_api_test Script

Restart the data plane as before:

# stop vpp
# start vpp

Execute the vpe_api_test script. Check the interface state and address:

# vpe_api_test in /tmp/api_script
vat# exec sh api interface 
              Name               Idx       State          Counter          Count     
GigabitEthernet2/1/0              5         up       rx packets                     2
                                                     rx bytes                     120
                                                     tx packets                     1
                                                     tx bytes                      42
                                                     drops                          2
local0                            0        down      
pg/stream-0                       1        down      
pg/stream-1                       2        down      
pg/stream-2                       3        down      
pg/stream-3                       4        down      

vat# exec show interface addr
GigabitEthernet2/1/0 (up):
  192.168.1.101/24
local0 (dn):
pg/stream-0 (dn):
pg/stream-1 (dn):
pg/stream-2 (dn):
pg/stream-3 (dn):

The interface address has been configured. Check the FIB:

vat# exec show ip fib
Table 0, fib_index 0, flow hash: src dst sport dport proto 
    Destination         Packets          Bytes         Adjacency   
192.168.1.0/24                     1             146 weight 1, index 3
                                                     arp GigabitEthernet2/1/0 192.168.1.101/24
192.168.1.1/32                     0               0 weight 1, index 7
                                                     GigabitEthernet2/1/0
                                                     IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a
192.168.1.101/32                   0               0 weight 1, index 4
                                                     local 192.168.1.101/24
1.2.3.4/32                         0               0 weight 1, index 6
                                                     GigabitEthernet2/1/0
                                                     IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a

The above sample shows what was expected.

Additional Things to Know About API Tracing

The API trace replay command takes additional arguments, including "first <NNN> and "last <NNN>". When trying to figure out precisely which API message caused an issue, rapid binary and/or linear searches may be performed.

vpe_api_test includes a recursive macro expander, of the form: "set INTERFACE1_ADDR 192.168.1.1/24".

It's not hard to take an API trace and convert it to a form where e.g. testbed addressing details can be moved to a host-specific file. Whereupon, a trace-derived script element can be rendered as "sw_interface_add_del_address GigabitEthernet2/1/0 $(INTERFACE1_ADDR)"

vpe_api_test takes multiple input filenames from the command line, allowing the following:

# vpe_api_test script in testbed_details in actual_script_with_macros 

The "script" argument causes vpe_api_test to exit upon script completion, instead of emitting a user prompt and waiting for additional commands.