CSIT

From fd.io
Revision as of 11:56, 30 March 2016 by Mackonstan (Talk | contribs)

Jump to: navigation, search
CSIT Facts

Project Lead: {{{projectLead}}}
Committers:

  • Matej Klotton
  • Stefan Kobza
  • Carsten Koester
  • Maciek Konstantynowicz
  • Miroslav Miklus
  • Dave Wallace

Repository: git clone https://gerrit.fd.io/r/csit
Mailing List: csit-dev@lists.fd.io
Jenkins: jenkins silo
Gerrit Patches: code patches/reviews
Bugs: CSIT bugs

Description

CSIT (Continuous System Integration and Testing) project is focusing on:

  1. Development of software code for fully automated VPP code testing, functionality, performance, regression and new functions.
  2. Execution of CSIT test suites on VPP code running on LF FD.io virtual and physical compute environments.
  3. Integration with FD.io continuous integration systems (Gerrit, Jenkins and such).
  4. Identified existing FD.io project dependencies and interactions:
    • vpp - Vector Packet Processing.
    • honeycomb - Honeycomb Agent for management plane testing.
    • ci-management - Management repo for Jenkins Job Builder, script and management related to the Jenkins CI configuration.

Get Involved

Scope

Fd.io CSIT project scope:

  1. Automated regression testing of VPP code changes
    • Functionality of VPP data plane, network control plane, management plane against functional specifications.
    • Performance of VPP data plane including non-drop-rate packet throughput and delay, against established reference benchmarks.
    • Performance of network control plane against established reference benchmarks.
    • Performance of management plane against established reference benchmarks.
  2. Test case definitions driven by supported and planned VPP functionality, interfaces and performance:
    • Uni-dimensional tests: Data plane, (Network) Control plane, Management plane.
    • Multi-dimensional tests: Use case driven.
  3. Integration with FD.io Continuous Integration system including FD.io Gerrit and Jenkins
    • Automated test execution triggered by VPP-VERIFY jobs other VPP and CSIT project jobs.
  4. Integration with LF VPP test execution environment
    • Functional tests execution on LF hosted VM environment.
    • Performance and functional tests execution on LF hosted physical compute environment.
    • Subset of tests executed on LF hosted physical compute running VIRL (Virtual Internet Routing Lab).

CSIT Code Structure

CSIT project consists of the following:

  • RobotFramework tests, resources, and libraries.
  • bash scripts – tools, and anything system-related (copying files, installing SW on nodes, ...).
  • Python libraries
    • the brains of the execution.
    • for different functionality there is a different module, i.e.
      • vpp
        • ipv4 utils.
        • ipv6 utils.
        • xconnect.
        • bdomain.
        • VAT (vpp_api_test) helpers.
        • Config generator.
      • ssh.
      • topology.
      • packet verifier – packet generator and validator.
      • v4/v6 ip network and host address generator.
  • vpp_api_test templates.

Each RF testsuite/case has TAGs associated with it that describe what environment that it can be run on: HW/VM, or what topology it requires. RobotFramework is executed with parameter that links to topology description file, we call it topology for simplicity. This file is parsed to variable “nodes” and later used in test cases and libraries.

In general test cases are written in readable English, so that even non-coders can understand it. These top level test cases should stay the same; in other words the testcase text should not represent “how” the test is done, but “what” the test case does.

Libraries to handle VPP functionality are written in Python and are separated on per-feature basis: v4, v6, interface (admin up, state status and so on), xconnect and bdomain. More modules are going to be implemented when needed.

Performance tests are executed using packet traffic generators external to servers running VPP code. Python APIs are used to control the traffic generators. Linux Foundation hosts physical infrastructure dedicated to FD.io, consisting of three of 3-compute-node performance testbeds (compute node = x86_64 multi-core server). Two of the compute nodes run VPP code, one runs a software traffic generator. Currently CSIT performance tests are executed using trex.

CSIT working testcases

~/csit $ grep "^| [a-zA-Z]" -nr tests | grep -vi " | "  | grep -v performance
tests/suites/bridge_domain/test.robot:27:| VPP reports interfaces
tests/suites/bridge_domain/test.robot:30:| Vpp forwards packets via L2 bridge domain 2 ports
tests/suites/bridge_domain/test.robot:42:| Vpp forwards packets via L2 bridge domain in circular topology
tests/suites/bridge_domain/test.robot:58:| Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
tests/suites/ipv4/ipv4_untagged.robot:31:| VPP replies to ICMPv4 echo request
tests/suites/ipv4/ipv4_untagged.robot:40:| TG can route to DUT egress interface
tests/suites/ipv4/ipv4_untagged.robot:49:| TG can route to DUT2 through DUT1
tests/suites/ipv4/ipv4_untagged.robot:58:| TG can route to DUT2 egress interface through DUT1
tests/suites/ipv4/ipv4_untagged.robot:67:| TG can route to TG through DUT1 and DUT2
tests/suites/ipv4/ipv4_untagged.robot:88:| VPP can process ICMP echo request from min to 1500B packet size with 1B increment
tests/suites/ipv4/ipv4_untagged.robot:92:| VPP can process ICMP echo request from 1500B to max packet size with 10B increment
tests/suites/ipv4/ipv4_untagged.robot:110:| VPP responds to ARP request
tests/suites/ipv6/ipv6_untagged.robot:33:| VPP replies to ICMPv6 echo request
tests/suites/ipv6/ipv6_untagged.robot:37:| VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
tests/suites/ipv6/ipv6_untagged.robot:41:| VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
tests/suites/ipv6/ipv6_untagged.robot:59:| TG can route to first DUT egress interface
tests/suites/ipv6/ipv6_untagged.robot:64:| TG can route to second DUT through first DUT
tests/suites/ipv6/ipv6_untagged.robot:69:| TG can route to second DUT egress interface through first DUT
tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
tests/suites/l2_xconnect/l2_xconnect_untagged.robot:26:| Vpp forwards packets via L2 xconnect in circular topology
tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
tests/suites/vxlan/vxlan_xconnect_untagged.robot:29:| VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect

CSIT code in development

  • 802.1ad/QinQ.
  • vhost-user.
  • basic HoneyComb handling in tests.

CSIT Project Page Content to_be_added

  • Documentation
    • Descriptions
      • CSIT code that's running: functional and performance tests.
      • CSIT testbeds
        • Operational: physical, virtual-VIRL.
        • Non-operational: virtual-VM.
      • Running CSIT integration with FD.io gerrit and jenkins.
    • CSIT usability manual
      • Instructions how to use CSIT to verify VPP code commits and avoid breaking it.
      • Onboarding of new VPP test cases onto CSIT system.
  • CSIT work in progress
    • Code in development, priorities.
    • Development plan, priorities.
    • FD.io jira for tracking execution against the plan, scheme proposal.
Retrieved from "https://wiki.fd.io/index.php?title=CSIT&oldid=680"