CSIT/VPP-16.06 Test Report Draft
Contents
[hide]Introduction
This report aims to provide a comprehensive and self-explanatory summary of all CSIT test cases that have been executed against FD.io VPP-16.06 code release, driven by the automated test infrastructure developed within the FD.io CSIT project (Continuous System and Integration Testing).
CSIT test cases have been grouped into the following test suites: {editor: use table instead of list below}
- <test_suite_name>: <test_case_name>, [<test_case_name>]
- bridge_domain: bridge_domain_untagged
- cop: cop_whitelist_blacklist, cop_whitelist_blacklist_IPv6
- dhcp: dhcp_client
- fds_related_tests: provider_network, tenant_network
- gre: gre_encapsulation
- honeycomb: interface_management, vxlan, bridge_domain, tap, interface_vhost_user, sub_interface, persistence, vxlan_gpe.
- ipv4: ipv4_arp_untagged, ipv4_iacl_untagged, ipv4_untagged
- ipv6: ipv6_iacl_untagged, ipv6_untagged
- l2_xconnect: l2_xconnect_untagged
- lisp: lisp_api_untagged, lisp_dataplane_untagged
- performance: Long_Bridge_Domain_Intel-X520-DA2, Long_IPv4_Cop_Intel-X520-DA2, Long_IPv4_Intel-X520-DA2, Long_IPv6_Cop_Intel-X520-DA2, Long_IPv6_Intel-X520-DA2, Long_Xconnect_Dot1q_Intel-X520-DA2, Long_Xconnect_Intel-X520-DA2, Short_Bridge_Domain_Intel-X520-DA2, Short_IPv4_Cop_Intel-X520-DA2, Short_IPv4_Intel-X520-DA2, Short_IPv6_Cop_Intel-X520-DA2, Short_IPv6_Intel-X520-DA2, Short_Xconnect_Dot1q_Intel-X520-DA2, Short_Xconnect_Intel-X520-DA2
- tagging: qinq_l2_xconnect
- vxlan: vxlan_bd_dot1q, vxlan_bd_untagged, vxlan_xconnect_untagged
CSIT source code for test cases listed above is available in CSIT branch stable/1606 in directory ./tests/suites/<name_of_the_test_suite>. Local copy of CSIT source code can be obtained by cloning CSIT git repository ("git clone https://gerrit.fd.io/r/csit"). CSIT testing virtual environment can be reproduced using Vagrant by following the instructions in CSIT tutorials.
Following sections provide description of CSIT test cases executed against VPP-16.06 release (vpp branch stable/1606), followed by summary test results and links to more detailed test results data. LF FD.io test environment and VPP DUT configuration specifics are provided later in this report to aid anyone interested in reproducing the complete LF FD.io CSIT testing environment, in either virtual or physical test beds.
Functional tests description
Functional tests run on virtual testbeds which are created in VIRL running on a Cisco UCS C240 server. There is currently only one testbed topology being used for functional testing - a three node topology with two links between each pair of nodes as shown in this diagram:
+--------+ +--------+
| <----------------------> |
| DUT1 | | DUT2 |
| <----------------------> |
+--^--^--+ +--^--^--+
| | | |
| | | |
| | +-------+ | |
| +---------> <----------+ |
| | TG | |
+------------> <-------------+
+-------+
Virtual testbeds are created dynamically whenever a patch is submitted to gerrit and destroyed upon completion of all functional tests. During test execution, all nodes are reachable thru the MGMT network connected to every node via dedicated NICs and links (not shown above for clarity). Each node is a Virtual Machine and each connection that is drawn on the diagram is available for use in any test case.
-dwallace notes- Should we add an overview of the test suites here? Since the generated results tables include test names and documentation fields, I don't think we need that level of detail here.
Performance tests description
Performance tests run on physical testbeds consisting of three Cisco UCS C240 servers. The logical testbed topology is fundamentally the same structure as the functional testbeds: three nodes with two links between each pair of nodes as shown in this diagram:
+--------+ +--------+
| <----------------------> |
| DUT1 | | DUT2 |
| <----------------------> |
+--^--^--+ +--^--^--+
| | | |
| | | |
| | +-------+ | |
| +---------> <----------+ |
| | TG | |
+------------> <-------------+
+-------+
However, at a physical level there are actually eight physical connections between each node utilizing 4 pairs of 10GbE or 40GbE NICs per node made by different vendors: Cisco 10GbE VICs, Cisco 40GbE VICs, Intel 10GbE Intel NICs, Intel 40GbE NICs. During test execution, all nodes are reachable thru the MGMT network connected to every node via dedicated NICs and links (not shown above for clarity). Currently the performance tests only utilize the Intel NICs.
-dwallace notes- Should we add an overview of the test suites here? Since the generated results tables include test names and documentation fields, I don't think we need that level of detail here.
Functional tests environment
Config Data: VPP (DUT)
VPP Running Configuration
VPP Startup Configuration
VPP Version
VPP Compile Parameters
VPP Build Script used - pointer if default
Config Data: VM (DUT)
VM OS Information
Config Data: Traffic Generator - Scapy
TG Version
Test Data: VPP (DUT)
VPP telemetry cli Outputs
Test Data: Traffic Generator - Scapy
Performance tests environment
Compute Node Environment for VPP (DUT)
Host hardware: cpu, memory, nics
Host BIOS Information
Host OS Linux distro ver
Host OS Linux kernel ver
Host OS Linux additional patch references
Host OS Qemu ver
Config Data: VPP (DUT)
NUMA Node Location of VPP Interfaces
NIC types
DPDK version
VPP Version
VPP Compile Parameters
VPP Build Script used - pointer if default
VPP Startup Configuration
VPP Running Configuration
Compute Node Environment for Traffic Generator (TG)
Host BIOS Information
Host OS Linux distro ver
Host OS Linux kernel ver
Host OS Linux additional patch references
Config Data: Traffic Generator - T-Rex
TG Version
DPDK version - if applicable
TG Compile Parameters
TG Build Script used - pointer if default
TG Startup Configuration
TG Running Configuration
TG common API calls?
Test Data: VPP (DUT)
VPP Telemetry Outputs collected at NDR and PDR
Test Data: Compute Host (DUT)
CPU Perf Counters collected at NDR - not collected today
Test Data: Traffic Generator - T-Rex
TG (T-Rex) Outputs During & After Test at 64B NDR
TG (T-Rex) Outputs During & After Test at IMIX NDR
TG (T-Rex) Outputs During & After Test at 1518 NDR
TG (T-Rex) Outputs During & After Test at 9000 NDR
Functional tests results
Dump of a vpp-csit-verify-virl-1606 job from console in ASCII.
Performance tests results
Set of graphs from vpp-csit-verify-hw-perf-1606-long job and trending from semiweekly. For more granular view, we can also link to wiki page with console output in ASCII and also links to robot log.html and report.html, if we find a way to host them somewhere.
CSIT release notes
jira release notes
complete dump of test cases from stable/1606 branch
functional
performance
