Difference between revisions of "CSIT/VPP-16.06 Test Report Sandbox"

Revision as of 12:48, 16 June 2016

Introduction

This report aims to provide a comprehensive and self-explanatory summary of all CSIT test cases that have been executed against FD.io VPP-16.06 code release, driven by the automated test infrastructure developed within the FD.io CSIT project (Continuous System and Integration Testing).

CSIT test cases have been grouped into the following test suites: {editor: use table instead of list below}

1. <test_suite_name>: <test_case_name>, [<test_case_name>]
2. bridge_domain: bridge_domain_untagged
3. cop: cop_whitelist_blacklist, cop_whitelist_blacklist_IPv6
4. dhcp: dhcp_client
5. fds_related_tests: provider_network, tenant_network
6. gre: gre_encapsulation
7. honeycomb: interface_management, vxlan, bridge_domain, tap, interface_vhost_user, sub_interface, persistence, vxlan_gpe.
8. ipv4: ipv4_arp_untagged, ipv4_iacl_untagged, ipv4_untagged
9. ipv6: ipv6_iacl_untagged, ipv6_untagged
10. l2_xconnect: l2_xconnect_untagged
11. lisp: lisp_api_untagged, lisp_dataplane_untagged
12. performance: Long_Bridge_Domain_Intel-X520-DA2, Long_IPv4_Cop_Intel-X520-DA2, Long_IPv4_Intel-X520-DA2, Long_IPv6_Cop_Intel-X520-DA2, Long_IPv6_Intel-X520-DA2, Long_Xconnect_Dot1q_Intel-X520-DA2, Long_Xconnect_Intel-X520-DA2, Short_Bridge_Domain_Intel-X520-DA2, Short_IPv4_Cop_Intel-X520-DA2, Short_IPv4_Intel-X520-DA2, Short_IPv6_Cop_Intel-X520-DA2, Short_IPv6_Intel-X520-DA2, Short_Xconnect_Dot1q_Intel-X520-DA2, Short_Xconnect_Intel-X520-DA2
13. tagging: qinq_l2_xconnect
14. vxlan: vxlan_bd_dot1q, vxlan_bd_untagged, vxlan_xconnect_untagged

CSIT source code for test cases listed above is available in CSIT branch stable/1606 in directory ./tests/suites/<name_of_the_test_suite>. Local copy of CSIT source code can be obtained by cloning CSIT git repository ("git clone https://gerrit.fd.io/r/csit"). CSIT testing virtual environment can be reproduced using Vagrant by following the instructions in CSIT tutorials.

Following sections provide description of CSIT test cases executed against VPP-16.06 release (vpp branch stable/1606), followed by summary test results and links to more detailed test results data. LF FD.io test environment and VPP DUT configuration specifics are provided later in this report to aid anyone interested in reproducing the complete LF FD.io CSIT testing environment, in either virtual or physical test beds.

Functional tests description

   description of topologies tested
   description of each test case
       taken from documentation from .robot files(?)
       reference specs of tested functionality i.e. IETF RFC
   list of executed test cases and summary of results PASS/FAIL/ISSUE

Functional tests results

Dump of a vpp-csit-verify-virl-1606 job from console in ASCII.

Performance tests description

   description of topologies tested
   description of each test case, test case type
       taken from documentation from .robot files(?)
       reference specs of tested functionality i.e. IETF RFC
   list of executed test cases and summary results
       long tests - graphs with discovered NDR and PDR rates
       short tests - list with PASS/FAIL/ISSUE

Performance tests results

Set of graphs from vpp-csit-verify-hw-perf-1606-long job and trending from semiweekly. For more granular view, we can also link to wiki page with console output in ASCII and also links to robot log.html and report.html, if we find a way to host them somewhere.

Functional tests environment

   Config Data: VPP (DUT)
       VPP Running Configuration
       VPP Startup Configuration
       VPP Version
       VPP Compile Parameters
       VPP Build Script used - pointer if default

   Config Data: VM (DUT)
       VM OS Information

   Config Data: Traffic Generator - Scapy
       TG Version

   Test Data: VPP (DUT)
       VPP telemetry cli Outputs

   Test Data: Traffic Generator - Scapy

Performance tests environment

   Compute Node Environment for VPP (DUT)
       Host hardware: cpu, memory, nics
       Host BIOS Information
       Host OS Linux distro ver
       Host OS Linux kernel ver
       Host OS Linux additional patch references
       Host OS Qemu ver
   Config Data: VPP (DUT)
       NUMA Node Location of VPP Interfaces
           NIC types
       DPDK version
       VPP Version
       VPP Compile Parameters
       VPP Build Script used - pointer if default
       VPP Startup Configuration
       VPP Running Configuration

   Compute Node Environment for Traffic Generator (TG)
       Host BIOS Information
       Host OS Linux distro ver
       Host OS Linux kernel ver
       Host OS Linux additional patch references

   Config Data: Traffic Generator - T-Rex
       TG Version
           DPDK version - if applicable
       TG Compile Parameters
       TG Build Script used - pointer if default
       TG Startup Configuration
       TG Running Configuration
       TG common API calls?

   Test Data: VPP (DUT)
       VPP Telemetry Outputs collected at NDR and PDR

   Test Data: Compute Host (DUT)
       CPU Perf Counters collected at NDR - not collected today

   Test Data: Traffic Generator - T-Rex
       TG (T-Rex) Outputs During & After Test at 64B NDR
       TG (T-Rex) Outputs During & After Test at IMIX NDR
       TG (T-Rex) Outputs During & After Test at 1518 NDR
       TG (T-Rex) Outputs During & After Test at 9000 NDR

CSIT release notes

   jira release notes
   complete dump of test cases from stable/1606 branch
       functional
       performance