CSIT/VPP-16.06 Test Report DraftLast updated 8 years ago

FD.io CSIT Test Report for VPP-16.06

Introduction

This report aims to provide a comprehensive and self-explanatory summary of all CSIT test cases that have been executed against FD.io VPP-16.06 code release, driven by the automated test infrastructure developed within the FD.io CSIT project (Continuous System and Integration Testing).

CSIT test cases have been grouped into the following test suites: {editor: use table instead of list below}

1. <test_suite_name>: <test_case_name>, [<test_case_name>]
2. bridge_domain: bridge_domain_untagged
3. cop: cop_whitelist_blacklist, cop_whitelist_blacklist_IPv6
4. dhcp: dhcp_client
5. fds_related_tests: provider_network, tenant_network
6. gre: gre_encapsulation
7. honeycomb: interface_management, vxlan, bridge_domain, tap, interface_vhost_user, sub_interface, persistence, vxlan_gpe.
8. ipv4: ipv4_arp_untagged, ipv4_iacl_untagged, ipv4_untagged
9. ipv6: ipv6_iacl_untagged, ipv6_untagged
10. l2_xconnect: l2_xconnect_untagged
11. lisp: lisp_api_untagged, lisp_dataplane_untagged
12. performance: Long_Bridge_Domain_Intel-X520-DA2, Long_IPv4_Cop_Intel-X520-DA2, Long_IPv4_Intel-X520-DA2, Long_IPv6_Cop_Intel-X520-DA2, Long_IPv6_Intel-X520-DA2, Long_Xconnect_Dot1q_Intel-X520-DA2, Long_Xconnect_Intel-X520-DA2, Short_Bridge_Domain_Intel-X520-DA2, Short_IPv4_Cop_Intel-X520-DA2, Short_IPv4_Intel-X520-DA2, Short_IPv6_Cop_Intel-X520-DA2, Short_IPv6_Intel-X520-DA2, Short_Xconnect_Dot1q_Intel-X520-DA2, Short_Xconnect_Intel-X520-DA2
13. tagging: qinq_l2_xconnect
14. vxlan: vxlan_bd_dot1q, vxlan_bd_untagged, vxlan_xconnect_untagged

CSIT source code for test cases listed above is available in CSIT branch stable/1606 in directory ./tests/suites/<name_of_the_test_suite>. Local copy of CSIT source code can be obtained by cloning CSIT git repository ("git clone https://gerrit.fd.io/r/csit"). CSIT testing virtual environment can be reproduced using Vagrant by following the instructions in CSIT tutorials.

Following sections provide description of CSIT test cases executed against VPP-16.06 release (vpp branch stable/1606), followed by summary test results and links to more detailed test results data. LF FD.io test environment and VPP DUT configuration specifics are provided later in this report to aid anyone interested in reproducing the complete LF FD.io CSIT testing environment, in either virtual or physical test beds.

Functional tests description

   per topology
       list of executed test cases and their results PASS/FAIL/ISSUE
   description of topologies
   description of each test case
       reference spec of tested functionality i.e. IETF RFC

Performance tests description

   per topology
       list of executed test cases and their results
           long tests - graphs with discovered NDR and PDR rates
           short tests - list with PASS/FAIL/ISSUE
   description of topologies
   description of each test case, test case type

Configuration of DUT VPP technology stack

   compute HW
   OS
   VPP compilation settings
   VPP startup configs

Performance tests environment

   Config Data: VPP (DUT)
       NUMA Node Location of VPP Interfaces
       VPP Running Configuration
       VPP Startup Configuration
       VPP Version
       VPP Compile Parameters
       VPP Build Script used - pointer if default

   Config Data: Compute Host (DUT)
       Host BIOS Information
       Host OS Information

   Config Data: Traffic Generator - T-Rex
       TG Version
       TG settings

   Test Data: VPP (DUT)
       VPP Debug_cli Outputs at NDR

   Test Data: Compute Host (DUT)
       CPU Perf Counters at NDR - not collected today

   Test Data: Traffic Generator - T-Rex
       TG (T-Rex) Outputs During & After Test at 64B NDR
       TG (T-Rex) Outputs During & After Test at IMIX NDR
       TG (T-Rex) Outputs During & After Test at 1518 NDR
       TG (T-Rex) Outputs During & After Test at 9000 NDR

Functional tests environment

   Config Data: VPP (DUT)
       VPP Running Configuration
       VPP Startup Configuration
       VPP Version
       VPP Compile Parameters
       VPP Build Script used - pointer if default

   Config Data: VM (DUT)
       VM OS Information

   Config Data: Traffic Generator - Scapy
       TG Version

   Test Data: VPP (DUT)
       VPP Debug_cli Outputs at NDR

   Test Data: Traffic Generator - Scapy

CSIT release notes

   jira release notes
   dump of test cases from csit-verified branch
       functional
       performance