CSIT/VPP-16.06 Test Report Draft
Contents
[hide]Introduction
This report aims to provide a comprehensive and self-explanatory summary of all CSIT test cases that have been executed against FD.io VPP-16.06 code release, driven by the automated test infrastructure developed within the FD.io CSIT project (Continuous System and Integration Testing).
CSIT test cases have been grouped into the following test suites: {editor: use table instead of list below}
- <test_suite_name>: <test_case_name>, [<test_case_name>]
- bridge_domain: bridge_domain_untagged
- cop: cop_whitelist_blacklist, cop_whitelist_blacklist_IPv6
- dhcp: dhcp_client
- fds_related_tests: provider_network, tenant_network
- gre: gre_encapsulation
- honeycomb: interface_management, vxlan, bridge_domain, tap, interface_vhost_user, sub_interface, persistence, vxlan_gpe.
- ipv4: ipv4_arp_untagged, ipv4_iacl_untagged, ipv4_untagged
- ipv6: ipv6_iacl_untagged, ipv6_untagged
- l2_xconnect: l2_xconnect_untagged
- lisp: lisp_api_untagged, lisp_dataplane_untagged
- performance: Long_Bridge_Domain_Intel-X520-DA2, Long_IPv4_Cop_Intel-X520-DA2, Long_IPv4_Intel-X520-DA2, Long_IPv6_Cop_Intel-X520-DA2, Long_IPv6_Intel-X520-DA2, Long_Xconnect_Dot1q_Intel-X520-DA2, Long_Xconnect_Intel-X520-DA2, Short_Bridge_Domain_Intel-X520-DA2, Short_IPv4_Cop_Intel-X520-DA2, Short_IPv4_Intel-X520-DA2, Short_IPv6_Cop_Intel-X520-DA2, Short_IPv6_Intel-X520-DA2, Short_Xconnect_Dot1q_Intel-X520-DA2, Short_Xconnect_Intel-X520-DA2
- tagging: qinq_l2_xconnect
- vxlan: vxlan_bd_dot1q, vxlan_bd_untagged, vxlan_xconnect_untagged
CSIT source code for test cases listed above is available in CSIT branch stable/1606 in directory ./tests/suites/<name_of_the_test_suite>. Local copy of CSIT source code can be obtained by cloning CSIT git repository ("git clone https://gerrit.fd.io/r/csit"). CSIT testing virtual environment can be reproduced using Vagrant by following the instructions in CSIT tutorials.
Following sections provide description of CSIT test cases executed against VPP-16.06 release (vpp branch stable/1606), followed by summary test results and links to more detailed test results data. LF FD.io test environment and VPP DUT configuration specifics are provided later in this report to aid anyone interested in reproducing the complete LF FD.io CSIT testing environment, in either virtual or physical test beds.
Functional tests description
Functional tests run on virtual testbeds which are created in VIRL running on a Cisco UCS C240 server. There is currently only one testbed topology being used for functional testing - a three node topology with two links between each pair of nodes as shown in this diagram:
+--------+ +--------+ | <----------------------> | | DUT1 | | DUT2 | | <----------------------> | +--^--^--+ +--^--^--+ | | | | | | | | | | +-------+ | | | +---------> <----------+ | | | TG | | +------------> <-------------+ +-------+
Virtual testbeds are created dynamically whenever a patch is submitted to gerrit and destroyed upon completion of all functional tests. During test execution, all nodes are reachable thru the MGMT network connected to every node via dedicated NICs and links (not shown above for clarity). Each node is a Virtual Machine and each connection that is drawn on the diagram is available for use in any test case.
-dwallace notes- Should we add an overview of the test suites here? Since the generated results tables include test names and documentation fields, I don't think we need that level of detail here.
Performance tests description
description of topologies tested description of each test case, test case type taken from documentation from .robot files(?) reference specs of tested functionality i.e. IETF RFC list of executed test cases and summary results long tests - graphs with discovered NDR and PDR rates short tests - list with PASS/FAIL/ISSUE
Functional tests environment
Config Data: VPP (DUT) VPP Running Configuration VPP Startup Configuration VPP Version VPP Compile Parameters VPP Build Script used - pointer if default
Config Data: VM (DUT) VM OS Information
Config Data: Traffic Generator - Scapy TG Version
Test Data: VPP (DUT) VPP telemetry cli Outputs
Test Data: Traffic Generator - Scapy
Performance tests environment
Compute Node Environment for VPP (DUT) Host hardware: cpu, memory, nics Host BIOS Information Host OS Linux distro ver Host OS Linux kernel ver Host OS Linux additional patch references Host OS Qemu ver Config Data: VPP (DUT) NUMA Node Location of VPP Interfaces NIC types DPDK version VPP Version VPP Compile Parameters VPP Build Script used - pointer if default VPP Startup Configuration VPP Running Configuration
Compute Node Environment for Traffic Generator (TG) Host BIOS Information Host OS Linux distro ver Host OS Linux kernel ver Host OS Linux additional patch references
Config Data: Traffic Generator - T-Rex TG Version DPDK version - if applicable TG Compile Parameters TG Build Script used - pointer if default TG Startup Configuration TG Running Configuration TG common API calls?
Test Data: VPP (DUT) VPP Telemetry Outputs collected at NDR and PDR
Test Data: Compute Host (DUT) CPU Perf Counters collected at NDR - not collected today
Test Data: Traffic Generator - T-Rex TG (T-Rex) Outputs During & After Test at 64B NDR TG (T-Rex) Outputs During & After Test at IMIX NDR TG (T-Rex) Outputs During & After Test at 1518 NDR TG (T-Rex) Outputs During & After Test at 9000 NDR
Functional tests results
Dump of a vpp-csit-verify-virl-1606 job from console in ASCII.
Performance tests results
Set of graphs from vpp-csit-verify-hw-perf-1606-long job and trending from semiweekly. For more granular view, we can also link to wiki page with console output in ASCII and also links to robot log.html and report.html, if we find a way to host them somewhere.
CSIT release notes
jira release notes complete dump of test cases from stable/1606 branch functional performance