View source for VPP/How To Use The API Trace Tools

=== Introduction ===

The VPP platform implements a set of binary APIs that control-plane agents can use to configure the VPP data plane. The VPP engine can trace, print, and replay binary API traces. This is an extremely powerful mechanism. When the data plane fails to pass traffic, any number of factors may be involved. Often, the data plane is simply not configured correctly.

API trace replay is very fast. Replaying 100,000 control-plane operations takes a fraction of a second - even though it may have taken an hour or more to accumulate that many transactions. There's no requirement for you to replay an API trace using the exact binary which generated the trace. It's often handy to add scaffolding to a debug image to work out what's going wrong. As long as the API definitions are consistent, you can add scaffolding, test bug fixes, and so forth.  

=== Capturing API traces ===

The usual method for capturing a binary API trace is to configure API tracing from the command-line:

 ... api-trace { on } ...

Current packaging always enables API tracing at data-plane start time. Alternatively, the debug Command-line Interface (CLI) <code>api trace on</code> command can be used at any time.

=== Using the API Test Tool to Create a Sample API Trace ===

Start the API test tool, located in /usr/lib/vpp_api_test_plugins . Determine which interface to use. The API test tool <code>exec</code> command sends debug CLI commands to the data-plane, and prints the results:

 # vpp_api_test
 vat# exec show interface
               Name               Idx       State          Counter          Count     
 GigabitEthernet2/1/0              5        down      
 local0                            0        down      
 pg/stream-0                       1        down      
 pg/stream-1                       2        down      
 pg/stream-2                       3        down      
 pg/stream-3                       4        down

Use the <code>help</code> command in the API test tool to see all of the APIs that it knows about. With a very few exceptions related to uploading statistics, it knows about all of the data plane binary APIs.

 # help
 Help is available for the following:
 add_node_next
 bd_ip_mac_add_del
 bridge_domain_add_del
 bridge_domain_dump
 <snip> ## type help yourself to see the entire list...

=== Create and Save a Real API Trace ===

Let's do the following: add an interface address on GigabitEthernet2/1/0, enable the interface, and add a static route via a live gateway on the 192.168.1.1/24 network. Lastly, we'll save the resulting API trace.

Use <code>help</code> in the API test tool to see how to add/delete an interface address. Then add an address for GigabitEthernet2/1/0 on the 192.168.1.1/24 network:

 vat# help sw_interface_add_del_address
 usage: sw_interface_add_del_address <intfc> | sw_if_index <id> <ip4-address> | <ip6-address> [del] [del-all] 
 
 vat# sw_interface_add_del_address GigabitEthernet2/1/0 192.168.1.101/24


Use <code>help</code> to see how to add/delete a route. Then add a route:

 vat# help ip_add_del_route
 usage: ip_add_del_route <addr>/<mask> via <addr> [vrf <n>]
 [<intfc> | sw_if_index <id>] [resolve-attempts <n>]
 [weight <n>] [drop] [local] [classify <n>] [del]
 [multipath] [count <n>]
 
 vat# ip_add_del_route 1.2.3.4/32 GigabitEthernet2/1/0 via 192.168.1.1 resolve-attempts 3

Verify that all of the above actually worked: 

 vat# exec sh ip arp
     Time      FIB        IP4      Stat      Ethernet              Interface       
     137.1815   0     192.168.1.1        48:f8:b3:e0:85:1a    GigabitEthernet2/1/0  

 vat# exec sh ip fib
 Table 0, fib_index 0, flow hash: src dst sport dport proto 
      Destination         Packets          Bytes         Adjacency   
 192.168.1.0/24                     3             438 weight 1, index 3
                                                      arp GigabitEthernet2/1/0 192.168.1.101/24
 192.168.1.1/32                     0               0 weight 1, index 6
                                                      GigabitEthernet2/1/0
                                                      IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a
 192.168.1.101/32                   0               0 weight 1, index 4
                                                      local 192.168.1.101/24
 1.2.3.4/32                         0               0 weight 1, index 5
                                                      GigabitEthernet2/1/0
                                                      IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a

Judging by the ARP and FIB tables in the above sample, the API message sequence worked.

If you are working in a development environment, use telnet to see the data plane debug CLI console, and save the API trace:

 # telnet 0 5002
 vpp# api trace save demo.api
 API trace saved to /tmp/demo.api

=== Pretty-print and Replay the API Trace ===

Restart the data plane:

 # stop vpp
 # start vpp

If you are working in a development environment, use telnet to see the data plane debug CLI console, and pretty-print the API trace:

 # telnet 0 5002
 vpp# api trace custom-dump /tmp/demo.api
 SCRIPT: memclnt_create name vpe_api_test 
 SCRIPT: sw_interface_dump name_filter Ether 
 SCRIPT: sw_interface_dump name_filter lo 
 SCRIPT: sw_interface_dump name_filter vxlan 
 SCRIPT: sw_interface_dump name_filter l2tpv3_t 
 SCRIPT: control_ping 
 SCRIPT: cli_request 
 SCRIPT: sw_interface_add_del_address sw_if_index 5 192.168.1.101/24 
 SCRIPT: sw_interface_set_flags sw_if_index 5 admin-up link-down
 SCRIPT: ip_add_del_route sw_if_index 5 1.2.3.4/32 via 192.168.1.1 resolve-attempts 3 
 SCRIPT: cli_request 
 SCRIPT: cli_request 
 vl_api_memclnt_delete_t:
 index: 0
 handle: 0x3059b0c0

Lines which begin with "SCRIPT: " turn into scripts. More on that in a minute.

Let's replay the API trace, and see what the ARP and IP FIB tables look like:

 vpp# api trace replay /tmp/demo.api
 vpp# show ip arp
     Time      FIB        IP4      Stat      Ethernet              Interface       
      60.5953   0     192.168.1.1        48:f8:b3:e0:85:1a    GigabitEthernet2/1/0  
 
 vpp# sh ip fib
 Table 0, fib_index 0, flow hash: src dst sport dport proto 
      Destination         Packets          Bytes         Adjacency   
 192.168.1.0/24                     0               0 weight 1, index 3
                                                      arp GigabitEthernet2/1/0 192.168.1.101/24
 192.168.1.1/32                     0               0 weight 1, index 5
                                                      GigabitEthernet2/1/0
                                                      IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a
 192.168.1.101/32                   0               0 weight 1, index 4
                                                      local 192.168.1.101/24
 1.2.3.4/32                         0               0 weight 1, index 6
                                                      GigabitEthernet2/1/0
                                                      IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a

Identical to the original.

=== Convert an API Trace Pretty-print Into a vpe_api_test Input Script ===

Here are the "interesting" API messages, as captured and pretty-printed: 

 SCRIPT: sw_interface_add_del_address sw_if_index 5 192.168.1.101/24 
 SCRIPT: sw_interface_set_flags sw_if_index 5 admin-up link-down
 SCRIPT: ip_add_del_route sw_if_index 5 1.2.3.4/32 via 192.168.1.1 resolve-attempts 3

Using a text editor, remove "SCRIPT: ". You are left with the text for the final vpe_api_test input script:

 $ cat /tmp/api_script
 sw_interface_add_del_address sw_if_index 5 192.168.1.101/24 
 sw_interface_set_flags sw_if_index 5 admin-up link-down
 ip_add_del_route sw_if_index 5 1.2.3.4/32 via 192.168.1.1 resolve-attempts 3

=== Execute the vpe_api_test Script ===

Restart the data plane as before:

 # stop vpp
 # start vpp

Execute the vpe_api_test script. Check the interface state and address:

 # vpe_api_test in /tmp/api_script
 vat# exec sh api interface 
               Name               Idx       State          Counter          Count     
 GigabitEthernet2/1/0              5         up       rx packets                     2
                                                      rx bytes                     120
                                                      tx packets                     1
                                                      tx bytes                      42
                                                      drops                          2
 local0                            0        down      
 pg/stream-0                       1        down      
 pg/stream-1                       2        down      
 pg/stream-2                       3        down      
 pg/stream-3                       4        down      
 
 vat# exec show interface addr
 GigabitEthernet2/1/0 (up):
   192.168.1.101/24
 local0 (dn):
 pg/stream-0 (dn):
 pg/stream-1 (dn):
 pg/stream-2 (dn):
 pg/stream-3 (dn):

The interface address has been configured. Check the FIB:

 vat# exec show ip fib
 Table 0, fib_index 0, flow hash: src dst sport dport proto 
     Destination         Packets          Bytes         Adjacency   
 192.168.1.0/24                     1             146 weight 1, index 3
                                                      arp GigabitEthernet2/1/0 192.168.1.101/24
 192.168.1.1/32                     0               0 weight 1, index 7
                                                      GigabitEthernet2/1/0
                                                      IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a
 192.168.1.101/32                   0               0 weight 1, index 4
                                                      local 192.168.1.101/24
 1.2.3.4/32                         0               0 weight 1, index 6
                                                      GigabitEthernet2/1/0
                                                      IP4: 00:0c:29:ec:42:5b -> 48:f8:b3:e0:85:1a

The above sample shows what was expected.

=== Additional Things to Know About API Tracing ===

The API trace replay command takes additional arguments, including "first <NNN> and "last <NNN>". When trying to figure out precisely which API message caused an issue, rapid binary and/or linear searches may be performed.

vpe_api_test includes a recursive macro expander, of the form: "set INTERFACE1_ADDR 192.168.1.1/24". 

It's not hard to take an API trace and convert it to a form where e.g. testbed addressing details can be moved to a host-specific file. Whereupon, a trace-derived script element can be rendered as "sw_interface_add_del_address GigabitEthernet2/1/0 $(INTERFACE1_ADDR)"

vpe_api_test takes multiple input filenames from the command line, allowing the following:

 # vpe_api_test script in testbed_details in actual_script_with_macros 

The "script" argument causes vpe_api_test to exit upon script completion, instead of emitting a user prompt and waiting for additional commands.