Difference between revisions of "CSIT/VPP-16.06 Test Report Draft"
Mackonstan (Talk | contribs) |
Mackonstan (Talk | contribs) |
||
Line 1: | Line 1: | ||
− | + | <span style="color:#ff0000; font-size:200%"><div style="text-align:center">DRAFT</div></span> | |
==Introduction== | ==Introduction== |
Revision as of 18:27, 10 June 2016
Contents
Introduction
This report aims to provide a comprehensive and self-explanatory summary of all CSIT test cases that have been executed against FD.io VPP-16.06 code release, driven by the automated test infrastructure developed within the FD.io CSIT project (Continuous System and Integration Testing).
CSIT test cases have been grouped into the following test suites: {editor: use table instead of list below}
- <test_suite_name>: <test_case_name>, [<test_case_name>]
- bridge_domain: bridge_domain_untagged
- cop: cop_whitelist_blacklist, cop_whitelist_blacklist_IPv6
- dhcp: dhcp_client
- fds_related_tests: provider_network, tenant_network
- gre: gre_encapsulation
- honeycomb: interface_management, vxlan, bridge_domain, tap, interface_vhost_user, sub_interface, persistence, vxlan_gpe.
- ipv4: ipv4_arp_untagged, ipv4_iacl_untagged, ipv4_untagged
- ipv6: ipv6_iacl_untagged, ipv6_untagged
- l2_xconnect: l2_xconnect_untagged
- lisp: lisp_api_untagged, lisp_dataplane_untagged
- performance: Long_Bridge_Domain_Intel-X520-DA2, Long_IPv4_Cop_Intel-X520-DA2, Long_IPv4_Intel-X520-DA2, Long_IPv6_Cop_Intel-X520-DA2, Long_IPv6_Intel-X520-DA2, Long_Xconnect_Dot1q_Intel-X520-DA2, Long_Xconnect_Intel-X520-DA2, Short_Bridge_Domain_Intel-X520-DA2, Short_IPv4_Cop_Intel-X520-DA2, Short_IPv4_Intel-X520-DA2, Short_IPv6_Cop_Intel-X520-DA2, Short_IPv6_Intel-X520-DA2, Short_Xconnect_Dot1q_Intel-X520-DA2, Short_Xconnect_Intel-X520-DA2
- tagging: qinq_l2_xconnect
- vxlan: vxlan_bd_dot1q, vxlan_bd_untagged, vxlan_xconnect_untagged
CSIT source code for test cases listed above is available in CSIT branch stable/1606 in directory ./tests/suites/<name_of_the_test_suite>. Local copy of CSIT source code can be obtained by cloning CSIT git repository ("git clone https://gerrit.fd.io/r/csit"). CSIT testing virtual environment can be reproduced using Vagrant by following the instructions in CSIT tutorials.
Following sections provide description of CSIT test cases executed against VPP-16.06 release (vpp branch stable/1606), followed by summary test results and links to more detailed test results data. LF FD.io test environment and VPP DUT configuration specifics are provided later in this report to aid anyone interested in reproducing the complete LF FD.io CSIT testing environment, in either virtual or physical test beds.
Functional tests description
description of topologies tested description of each test case taken from documentation from .robot files(?) reference specs of tested functionality i.e. IETF RFC list of executed test cases and summary of results PASS/FAIL/ISSUE
Functional tests results
Dump of a vpp-csit-verify-virl-1606 job from console in ASCII.
Performance tests description
description of topologies tested description of each test case, test case type taken from documentation from .robot files(?) reference specs of tested functionality i.e. IETF RFC list of executed test cases and summary results long tests - graphs with discovered NDR and PDR rates short tests - list with PASS/FAIL/ISSUE
Performance tests results
Set of graphs from vpp-csit-verify-hw-perf-1606-long job and trending from semiweekly. For more granular view, we can also link to wiki page with console output in ASCII and also links to robot log.html and report.html, if we find a way to host them somewhere.
Functional tests environment
Config Data: VPP (DUT) VPP Running Configuration VPP Startup Configuration VPP Version VPP Compile Parameters VPP Build Script used - pointer if default
Config Data: VM (DUT) VM OS Information
Config Data: Traffic Generator - Scapy TG Version
Test Data: VPP (DUT) VPP telemetry cli Outputs
Test Data: Traffic Generator - Scapy
Performance tests environment
Compute Node Environment for VPP (DUT) Host hardware: cpu, memory, nics Host BIOS Information Host OS Linux distro ver Host OS Linux kernel ver Host OS Linux additional patch references Host OS Qemu ver Config Data: VPP (DUT) NUMA Node Location of VPP Interfaces NIC types DPDK version VPP Version VPP Compile Parameters VPP Build Script used - pointer if default VPP Startup Configuration VPP Running Configuration
Compute Node Environment for Traffic Generator (TG) Host BIOS Information Host OS Linux distro ver Host OS Linux kernel ver Host OS Linux additional patch references
Config Data: Traffic Generator - T-Rex TG Version DPDK version - if applicable TG Compile Parameters TG Build Script used - pointer if default TG Startup Configuration TG Running Configuration TG common API calls?
Test Data: VPP (DUT) VPP Telemetry Outputs collected at NDR and PDR
Test Data: Compute Host (DUT) CPU Perf Counters collected at NDR - not collected today
Test Data: Traffic Generator - T-Rex TG (T-Rex) Outputs During & After Test at 64B NDR TG (T-Rex) Outputs During & After Test at IMIX NDR TG (T-Rex) Outputs During & After Test at 1518 NDR TG (T-Rex) Outputs During & After Test at 9000 NDR
CSIT release notes
jira release notes complete dump of test cases from stable/1606 branch functional performance