Difference between revisions of "CSIT"

Revision as of 17:52, 13 April 2016

Description

CSIT (Continuous System Integration and Testing) project is focusing on:

1. Development of software code for fully automated VPP code testing, functionality, performance, regression and new functions.
2. Execution of CSIT test suites on VPP code running on LF FD.io virtual and physical compute environments.
3. Integration with FD.io continuous integration systems (Gerrit, Jenkins and such).
4. Identified existing FD.io project dependencies and interactions:
   • vpp - Vector Packet Processing.
   • honeycomb - Honeycomb Agent for management plane testing.
   • ci-management - Management repo for Jenkins Job Builder, script and management related to the Jenkins CI configuration.

Get Involved

• Weekly CSIT Meeting.
• Join the CSIT Mailing List.
• Join fdio-csit IRC channel.

Scope

Fd.io CSIT project scope:

1. Automated regression testing of VPP code changes
   • Functionality of VPP data plane, network control plane, management plane against functional specifications.
   • Performance of VPP data plane including non-drop-rate packet throughput and delay, against established reference benchmarks.
   • Performance of network control plane against established reference benchmarks.
   • Performance of management plane against established reference benchmarks.
2. Test case definitions driven by supported and planned VPP functionality, interfaces and performance:
   • Uni-dimensional tests: Data plane, (Network) Control plane, Management plane.
   • Multi-dimensional tests: Use case driven.
3. Integration with FD.io Continuous Integration system including FD.io Gerrit and Jenkins
   • Automated test execution triggered by VPP-VERIFY jobs other VPP and CSIT project jobs.
4. Integration with LF VPP test execution environment
   • Functional tests execution on LF hosted VM environment.
   • Performance and functional tests execution on LF hosted physical compute environment.
   • Subset of tests executed on LF hosted physical compute running VIRL (Virtual Internet Routing Lab).

CSIT Code Structure

CSIT project consists of the following:

• RobotFramework tests, resources, and libraries.
• bash scripts – tools, and anything system-related (copying files, installing SW on nodes, ...).
• Python libraries
  • the brains of the execution.
  • for different functionality there is a different module, i.e.
    • vpp
      • ipv4 utils.
      • ipv6 utils.
      • xconnect.
      • bdomain.
      • VAT (vpp_api_test) helpers.
      • Config generator.
    • ssh.
    • topology.
    • packet verifier – packet generator and validator.
    • v4/v6 ip network and host address generator.
• vpp_api_test templates.

Each RF testsuite/case has TAGs associated with it that describe what environment that it can be run on: HW/VM, or what topology it requires. RobotFramework is executed with parameter that links to topology description file, we call it topology for simplicity. This file is parsed to variable “nodes” and later used in test cases and libraries.

In general test cases are written in readable English, so that even non-coders can understand it. These top level test cases should stay the same; in other words the testcase text should not represent “how” the test is done, but “what” the test case does.

Libraries to handle VPP functionality are written in Python and are separated on per-feature basis: v4, v6, interface (admin up, state status and so on), xconnect and bdomain. More modules are going to be implemented when needed.

Performance tests are executed using packet traffic generators external to servers running VPP code. Python APIs are used to control the traffic generators. Linux Foundation hosts physical infrastructure dedicated to FD.io, consisting of three of 3-compute-node performance testbeds (compute node = x86_64 multi-core server). Two of the compute nodes run VPP code, one runs a software traffic generator. Currently CSIT performance tests are executed using trex.

CSIT Test Code Guidelines - more examples will be added

Here are some guidelines for writing reliable, maintainable, reusable and readable Robot Framework (RF) test code. There is used Robot Framework version 2.9.2 (user guide) in CSIT.

RobotFramework test case files and resource files

• General
  • RobotFramework test case files and resource files use special extension .robot
  • Usage of pipe and space separated file format is strongly recommended. Tabs are invisible characters which is error prone.
  • Files should be encoded in ASCII. Non-ASCII characters are allowed but they must be encoded in UTF8 (the default Robot source file encoding).
  • Line length is limited to 80 characters.
  • There must be included licence (/csit/docs/licence.rst) at the begging of each file.
  • Copy-pasting of the code is unwanted practice, any code that could be re-used has to be put into RF keyword (KW) or python library instead of copy-pasted.

• Test cases
  • Test cases are written in Behavior-driven style – i.e. in readable English so that even non-technical project stakeholders can understand it:

  *** Test Cases ***
  | VPP can encapsulate L2 in VXLAN over IPv4 over Dot1Q
  | | Given Path for VXLAN testing is set
  | | ...   | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']}
  | | And   Interfaces in path are up
  | | And   Vlan interfaces for VXLAN are created | ${VLAN}
  | |       ...                                   | ${dut1} | ${dut1s_to_dut2}
  | |       ...                                   | ${dut2} | ${dut2s_to_dut1}
  | | And   IP addresses are set on interfaces
  | |       ...         | ${dut1} | ${dut1s_vlan_name} | ${dut1s_vlan_index}
  | |       ...         | ${dut2} | ${dut2s_vlan_name} | ${dut2s_vlan_index}
  | | ${dut1s_vxlan}= | When Create VXLAN interface     | ${dut1} | ${VNI}
  | |                 | ...  | ${dut1s_ip_address} | ${dut2s_ip_address}
  | |                   And  Interfaces are added to BD | ${dut1} | ${BID}
  | |                   ...  | ${dut1s_to_tg} | ${dut1s_vxlan}
  | | ${dut2s_vxlan}= | And  Create VXLAN interface     | ${dut2} | ${VNI}
  | |                 | ...  | ${dut2s_ip_address} | ${dut1s_ip_address}
  | |                   And  Interfaces are added to BD | ${dut2} | ${BID}
  | |                   ...  | ${dut2s_to_tg} | ${dut2s_vxlan}
  | | Then Send and receive ICMPv4 bidirectionally
  | | ... | ${tg} | ${tgs_to_dut1} | ${tgs_to_dut2}

• • Every test case should contain short documentation. (example will be added) This documentation will be used by testdoc tool - Robot Framework's built-in tool for generating high level documentation based on test cases.
  • Do not use hard-coded constants. It is recommended to use the variable table (***Variables***) to define test case specific values. Use the assignment sign = after the variable name to make assigning variables slightly more explicit:

  *** Variables ***
  | ${VNI}= | 23

• • Common test case specific settings of the test environment should be done in Test Setup part of the Setting table ease on (***Settings***).
  • Post-test cleaning and processing actions should be done in Test Teardown part of the Setting table (e.g. download statistics from VPP nodes). This part is executed even if the test case has failed. On the other hand it is possible to disable the tear-down from command line, thus leaving the system in “broken” state for investigation.
  • Every TC must be correctly tagged. List of defined tags is in /csit/docs/tag_documentation.rst file.
  • User high-level keywords specific for the particular test case can be implemented in the keyword table of the test case to enable readability and code-reuse.

• Resource files
  • Used to implement higher-level keywords that are used in test cases or other higher-level keywords.
  • Every keyword must contains Documentation where the purpose and arguments of the KW are described.
  • The best practice is that the KW usage example is the part of the Documentation. It is recommended to use pipe and space separated format for the example.
  • Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).

Python library files

• General
  • Used to implement low-level keywords that are used in resource files (to create higher-level keywords) or in test cases.
  • Higher-level keywords can be implemented in python library file too, especially in the case that their implementation in resource file would be too difficult or impossible, e.g. nested FOR loops or branching.
  • Every keyword, Python module, class, method, enums has to contain documentation string with the short description and used input parameters and possible return value(s).
  • The best practice is that the KW usage example is the part of the Documentation. It should contains two parts – RobotFramework example and Python example. It is recommended to use pipe and space separated format in case of RobotFramework example.
  • KW usage examples can be grouped and used in the class documentation string to provide better overview of the usage and relationships between KWs.
  • Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).
  • There must be included licence (/csit/docs/licence.rst) at the begging of each file.

• Coding
  • It is recommended to use some standard development tool (e.g. PyCharm Community Edition) and follow PEP-8 recommendations.
  • All python code (not only RF libraries) must adhere to PEP-8 standard. This is enforced by CSIT Jenkins verify job.
  • Indentation – do not use tab for indents! Indent is defined as four spaces.
  • Line length – limited to 80 characters.
  • Imports - use the full pathname location of the module, e.g. from resources.libraries.python.topology import Topology. Imports should be grouped in the following order: 1. standard library imports, 2. related third party imports, 3. local application/library specific imports. You should put a blank line between each group of imports.
  • Blank lines - Two blank lines between top-level definitions, one blank line between method definitions.
  • Do not use global variables inside library files.
  • Comparisons – should be in format 0 == ret_code not ret_code == 0 to avoid possible interchange of = (assignment) and == (equal to) that could be difficult to identify such error.
  • Constants – avoid to use hard-coded constants (e.g. numbers, paths without any description). Use configuration file(s), like /csit/resources/libraries/python/constants.py, with appropriate comments.
  • Logging – log at the lowest possible level of implementation (debugging purposes). Use same style for similar events. Keep logging as verbose as necessary.
  • Exceptions – use the most appropriate exception not general one („Exception“ ) if possible. Create your own exception if necessary and implement there logging, level debug.

CSIT working testcases

~/csit $ grep "^| [a-zA-Z]" -nr tests | grep -vi " | "  | grep -v performance
tests/suites/bridge_domain/test.robot:27:| VPP reports interfaces
tests/suites/bridge_domain/test.robot:30:| Vpp forwards packets via L2 bridge domain 2 ports
tests/suites/bridge_domain/test.robot:42:| Vpp forwards packets via L2 bridge domain in circular topology
tests/suites/bridge_domain/test.robot:58:| Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
tests/suites/ipv4/ipv4_untagged.robot:31:| VPP replies to ICMPv4 echo request
tests/suites/ipv4/ipv4_untagged.robot:40:| TG can route to DUT egress interface
tests/suites/ipv4/ipv4_untagged.robot:49:| TG can route to DUT2 through DUT1
tests/suites/ipv4/ipv4_untagged.robot:58:| TG can route to DUT2 egress interface through DUT1
tests/suites/ipv4/ipv4_untagged.robot:67:| TG can route to TG through DUT1 and DUT2
tests/suites/ipv4/ipv4_untagged.robot:88:| VPP can process ICMP echo request from min to 1500B packet size with 1B increment
tests/suites/ipv4/ipv4_untagged.robot:92:| VPP can process ICMP echo request from 1500B to max packet size with 10B increment
tests/suites/ipv4/ipv4_untagged.robot:110:| VPP responds to ARP request
tests/suites/ipv6/ipv6_untagged.robot:33:| VPP replies to ICMPv6 echo request
tests/suites/ipv6/ipv6_untagged.robot:37:| VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
tests/suites/ipv6/ipv6_untagged.robot:41:| VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
tests/suites/ipv6/ipv6_untagged.robot:59:| TG can route to first DUT egress interface
tests/suites/ipv6/ipv6_untagged.robot:64:| TG can route to second DUT through first DUT
tests/suites/ipv6/ipv6_untagged.robot:69:| TG can route to second DUT egress interface through first DUT
tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
tests/suites/l2_xconnect/l2_xconnect_untagged.robot:26:| Vpp forwards packets via L2 xconnect in circular topology
tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
tests/suites/vxlan/vxlan_xconnect_untagged.robot:29:| VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect

CSIT code in development

• 802.1ad/QinQ.
• vhost-user.
• basic HoneyComb handling in tests.

CSIT test cases development - draft plan

• CSIT functional tests - draft dev plan.

CSIT Tutorials

• Local execution of CSIT using Vagrant

CSIT Materials

• FD.io DevBoot - San Jose 4th to 7th of April 2016 - CSIT + Performance testing

CSIT Project Page Content to_be_added

• Documentation
  • Descriptions
    • CSIT code that's running: functional and performance tests.
    • CSIT testbeds
      • Operational: physical, virtual-VIRL.
      • Non-operational: virtual-VM.
    • Running CSIT integration with FD.io gerrit and jenkins.
  • CSIT usability manual
    • Instructions how to use CSIT to verify VPP code commits and avoid breaking it.
    • Onboarding of new VPP test cases onto CSIT system.
• CSIT work in progress
  • Code in development, priorities.
  • Development plan, priorities.
  • FD.io jira for tracking execution against the plan, scheme proposal.