|
|
| (110 intermediate revisions by 12 users not shown) |
| Line 1: |
Line 1: |
| | + | |
| | {{Project Facts | | {{Project Facts |
| | |name=CSIT | | |name=CSIT |
| | |shortname=csit | | |shortname=csit |
| | |jiraName=CSIT | | |jiraName=CSIT |
| | + | |projectLead=Maciek Konstantynowicz |
| | |committers= | | |committers= |
| − | * Matej Klotton | + | * Tibor Frank |
| − | * Stefan Kobza
| + | |
| − | * Carsten Koester
| + | |
| | * Maciek Konstantynowicz | | * Maciek Konstantynowicz |
| − | * Miroslav Miklus | + | * Peter Mikus |
| | + | * Vratko Polak |
| | * Dave Wallace | | * Dave Wallace |
| | }} | | }} |
| | | | |
| − | ==Description==
| + | == Get Involved == |
| − | CSIT (Continuous System Integration and Testing) project is focusing on:
| + | |
| − | # Development of software code for fully automated VPP code testing, functionality, performance, regression and new functions.
| + | |
| − | # Execution of CSIT test suites on VPP code running on LF FD.io virtual and physical compute environments.
| + | |
| − | # Integration with FD.io continuous integration systems (Gerrit, Jenkins and such).
| + | |
| − | # Identified existing FD.io project dependencies and interactions:
| + | |
| − | #* vpp - Vector Packet Processing.
| + | |
| − | #* honeycomb - Honeycomb Agent for management plane testing.
| + | |
| − | #* ci-management - Management repo for Jenkins Job Builder, script and management related to the Jenkins CI configuration.
| + | |
| − | | + | |
| − | ==Get Involved== | + | |
| | * [[CSIT/Meeting|Weekly CSIT Meeting]]. | | * [[CSIT/Meeting|Weekly CSIT Meeting]]. |
| | + | * [https://wiki.fd.io/view/CSIT/TWS On-demand Technical Workstream Meetings]. |
| | * [https://lists.fd.io/mailman/listinfo/csit-dev Join the CSIT Mailing List]. | | * [https://lists.fd.io/mailman/listinfo/csit-dev Join the CSIT Mailing List]. |
| | * [[IRC | Join fdio-csit IRC channel]]. | | * [[IRC | Join fdio-csit IRC channel]]. |
| | | | |
| − | ==Scope== | + | == Start Here == |
| − | Fd.io CSIT project scope:
| + | * [[CSIT/Description | What is CSIT (Continuous System Integration and Testing)?]] |
| | + | * [https://csit.fd.io/cdocs/overview/csit/design/ CSIT Design] |
| | + | * [https://csit.fd.io/cdocs/overview/c_dash/design/ CDASH Design] |
| | + | * [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=8 CSIT Development Tasks (Jira)] |
| | + | * [https://jenkins.fd.io/view/csit/ CSIT Jenkins Jobs ] |
| | + | * [https://csit.fd.io/cdocs/overview/csit/branching_strategy/ CSIT Branching Strategy] |
| | + | * [https://csit.fd.io/cdocs/infrastructure/fdio_dc_testbed_specifications/ FD.io CSIT testbeds specification] |
| | + | * [https://csit.fd.io/ CSIT Performance Dashboard] |
| | + | ** [https://csit.fd.io/trending/ Performance Trending] |
| | + | ** [https://csit.fd.io/report/ Per Release Performance] |
| | + | ** [https://csit.fd.io/stats/ Test Job Statistics] |
| | + | ** [https://csit.fd.io/news/ Failures and Anomalies] |
| | + | * [https://wiki.fd.io/view/CSIT/TestFailuresTracking CSIT Test Failures Summary and Status] |
| | | | |
| − | # Automated regression testing of VPP code changes
| + | == Test Reports == |
| − | #* Functionality of VPP data plane, network control plane, management plane against functional specifications.
| + | * [https://csit.fd.io/cdocs/release_notes/current/ Current release] |
| − | #* Performance of VPP data plane including non-drop-rate packet throughput and delay, against established reference benchmarks.
| + | * [https://csit.fd.io/cdocs/release_notes/previous/ Previous releases] |
| − | #* Performance of network control plane against established reference benchmarks.
| + | |
| − | #* Performance of management plane against established reference benchmarks.
| + | |
| − | # Test case definitions driven by supported and planned VPP functionality, interfaces and performance:
| + | |
| − | #* Uni-dimensional tests: Data plane, (Network) Control plane, Management plane.
| + | |
| − | #* Multi-dimensional tests: Use case driven.
| + | |
| − | # Integration with FD.io Continuous Integration system including FD.io Gerrit and Jenkins
| + | |
| − | #* Automated test execution triggered by VPP-VERIFY jobs other VPP and CSIT project jobs.
| + | |
| − | # Integration with LF VPP test execution environment
| + | |
| − | #* Functional tests execution on LF hosted VM environment.
| + | |
| − | #* Performance and functional tests execution on LF hosted physical compute environment.
| + | |
| − | #* Subset of tests executed on LF hosted physical compute running VIRL (Virtual Internet Routing Lab).
| + | |
| | | | |
| − | ==CSIT Code Structure== | + | == CSIT Code Documentation == |
| | + | * [https://s3-docs.fd.io/csit/master/docs CSIT master Documentation] |
| | + | * [https://s3-docs.fd.io/csit/rls2206/docs/ CSIT-2206 Documentation] |
| | + | * [https://s3-docs.fd.io/csit/rls2202/docs/ CSIT-2202 Documentation] |
| | + | * [https://s3-docs.fd.io/csit/rls2110/docs/ CSIT-2110 Documentation] |
| | + | * [https://s3-docs.fd.io/csit/rls2106/docs/ CSIT-2106 Documentation] |
| | + | * [https://docs.fd.io/csit/rls2101/doc/ CSIT-2101 Documentation] |
| | + | * [https://docs.fd.io/csit/rls2009/doc/ CSIT-2009 Documentation] |
| | + | * [https://docs.fd.io/csit/rls2005/doc/ CSIT-2005 Documentation] |
| | + | * [https://docs.fd.io/csit/rls2001/doc/ CSIT-2001 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1908/doc/ CSIT-1908 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1904/doc/ CSIT-1904 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1901/doc/ CSIT-1901 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1810/doc/ CSIT-1810 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1804/doc/ CSIT-1804 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1801/doc/ CSIT-1801 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1710/doc/ CSIT-1710 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1707/doc/ CSIT-1707 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1704/doc/ CSIT-1704 Documentation] |
| | + | * [https://docs.fd.io/csit/rls1701/doc/ CSIT-1701 Documentation] |
| | + | * [[CSIT/Documentation| CSIT 16.09/16.06 Documentation]] |
| | | | |
| − | CSIT project consists of the following:
| + | == Training == |
| − | * RobotFramework tests, resources, and libraries.
| + | |
| − | * bash scripts – tools, and anything system-related (copying files, installing SW on nodes, ...).
| + | |
| − | * Python libraries
| + | |
| − | ** the brains of the execution.
| + | |
| − | ** for different functionality there is a different module, i.e.
| + | |
| − | *** vpp
| + | |
| − | **** ipv4 utils.
| + | |
| − | **** ipv6 utils.
| + | |
| − | **** xconnect.
| + | |
| − | **** bdomain.
| + | |
| − | **** VAT (vpp_api_test) helpers.
| + | |
| − | **** Config generator.
| + | |
| − | *** ssh.
| + | |
| − | *** topology.
| + | |
| − | *** packet verifier – packet generator and validator.
| + | |
| − | *** v4/v6 ip network and host address generator.
| + | |
| − | * vpp_api_test templates.
| + | |
| − | | + | |
| − | Each RF testsuite/case has TAGs associated with it that describe what environment that it can be run on: HW/VM, or what topology it requires. RobotFramework is executed with parameter that links to topology description file, we call it topology for simplicity. This file is parsed to variable “nodes” and later used in test cases and libraries.
| + | |
| − | | + | |
| − | In general test cases are written in readable English, so that even non-coders can understand it. These top level test cases should stay the same; in other words the testcase text should not represent “how” the test is done, but “what” the test case does.
| + | |
| − | | + | |
| − | Libraries to handle VPP functionality are written in Python and are separated on per-feature basis: v4, v6, interface (admin up, state status and so on), xconnect and bdomain. More modules are going to be implemented when needed.
| + | |
| − | | + | |
| − | Performance tests are executed using packet traffic generators external to servers running VPP code. Python APIs are used to control the traffic generators. Linux Foundation hosts physical infrastructure dedicated to FD.io, consisting of three of 3-compute-node performance testbeds (compute node = x86_64 multi-core server). Two of the compute nodes run VPP code, one runs a software traffic generator. Currently CSIT performance tests are executed using [https://github.com/cisco-system-traffic-generator/trex-core trex].
| + | |
| − | | + | |
| − | ==CSIT Test Code Guidelines - more examples will be added== | + | |
| − | | + | |
| − | Here are some guidelines for writing reliable, maintainable, reusable and readable Robot Framework (RF) test code. There is used Robot Framework version 2.9.2 ([http://robotframework.org/robotframework/2.9.2/RobotFrameworkUserGuide.html user guide]) in CSIT.
| + | |
| − | | + | |
| − | RobotFramework test case files and resource files
| + | |
| − | | + | |
| − | * General
| + | |
| − | ** RobotFramework test case files and resource files use special extension .robot
| + | |
| − | ** Usage of pipe and space separated file format is strongly recommended. Tabs are invisible characters which is error prone.
| + | |
| − | ** Files should be encoded in ASCII. Non-ASCII characters are allowed but they must be encoded in UTF8 (the default Robot source file encoding).
| + | |
| − | ** Line length is limited to 80 characters.
| + | |
| − | ** There must be included licence (/csit/docs/licence.rst) at the begging of each file.
| + | |
| − | ** Copy-pasting of the code is unwanted practice, any code that could be re-used has to be put into RF keyword (KW) or python library instead of copy-pasted.
| + | |
| − | | + | |
| − | * Test cases
| + | |
| − | ** Test cases are written in Behavior-driven style – i.e. in readable English so that even non-technical project stakeholders can understand it:
| + | |
| − | *** Test Cases ***
| + | |
| − | | VPP can encapsulate L2 in VXLAN over IPv4 over Dot1Q
| + | |
| − | | | Given Path for VXLAN testing is set
| + | |
| − | | | ... | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']}
| + | |
| − | | | And Interfaces in path are up
| + | |
| − | | | And Vlan interfaces for VXLAN are created | ${VLAN}
| + | |
| − | | | ... | ${dut1} | ${dut1s_to_dut2}
| + | |
| − | | | ... | ${dut2} | ${dut2s_to_dut1}
| + | |
| − | | | And IP addresses are set on interfaces
| + | |
| − | | | ... | ${dut1} | ${dut1s_vlan_name} | ${dut1s_vlan_index}
| + | |
| − | | | ... | ${dut2} | ${dut2s_vlan_name} | ${dut2s_vlan_index}
| + | |
| − | | | ${dut1s_vxlan}= | When Create VXLAN interface | ${dut1} | ${VNI}
| + | |
| − | | | | ... | ${dut1s_ip_address} | ${dut2s_ip_address}
| + | |
| − | | | And Interfaces are added to BD | ${dut1} | ${BID}
| + | |
| − | | | ... | ${dut1s_to_tg} | ${dut1s_vxlan}
| + | |
| − | | | ${dut2s_vxlan}= | And Create VXLAN interface | ${dut2} | ${VNI}
| + | |
| − | | | | ... | ${dut2s_ip_address} | ${dut1s_ip_address}
| + | |
| − | | | And Interfaces are added to BD | ${dut2} | ${BID}
| + | |
| − | | | ... | ${dut2s_to_tg} | ${dut2s_vxlan}
| + | |
| − | | | Then Send and receive ICMPv4 bidirectionally
| + | |
| − | | | ... | ${tg} | ${tgs_to_dut1} | ${tgs_to_dut2}
| + | |
| − | ** Every test case should contain short documentation. (example will be added) This documentation will be used by testdoc tool - Robot Framework's built-in tool for generating high level documentation based on test cases.
| + | |
| − | ** Do not use hard-coded constants. It is recommended to use the variable table (***Variables***) to define test case specific values. Use the assignment sign = after the variable name to make assigning variables slightly more explicit:
| + | |
| − | *** Variables ***
| + | |
| − | | ${VNI}= | 23
| + | |
| − | ** Common test case specific settings of the test environment should be done in Test Setup part of the Setting table ease on (***Settings***).
| + | |
| − | ** Post-test cleaning and processing actions should be done in Test Teardown part of the Setting table (e.g. download statistics from VPP nodes). This part is executed even if the test case has failed. On the other hand it is possible to disable the tear-down from command line, thus leaving the system in “broken” state for investigation.
| + | |
| − | ** Every TC must be correctly tagged. List of defined tags is in /csit/docs/tag_documentation.rst file.
| + | |
| − | ** User high-level keywords specific for the particular test case can be implemented in the keyword table of the test case to enable readability and code-reuse.
| + | |
| − | | + | |
| − | * Resource files
| + | |
| − | ** Used to implement higher-level keywords that are used in test cases or other higher-level keywords.
| + | |
| − | ** Every keyword must contains Documentation where the purpose and arguments of the KW are described.
| + | |
| − | ** The best practice is that the KW usage example is the part of the Documentation. It is recommended to use pipe and space separated format for the example.
| + | |
| − | ** Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).
| + | |
| − | | + | |
| − | | + | |
| − | Python library files
| + | |
| − | | + | |
| − | * General
| + | |
| − | ** Used to implement low-level keywords that are used in resource files (to create higher-level keywords) or in test cases.
| + | |
| − | ** Higher-level keywords can be implemented in python library file too, especially in the case that their implementation in resource file would be too difficult or impossible, e.g. nested FOR loops or branching.
| + | |
| − | ** Every keyword, Python module, class, method, enums has to contain documentation string with the short description and used input parameters and possible return value(s).
| + | |
| − | ** The best practice is that the KW usage example is the part of the Documentation. It should contains two parts – RobotFramework example and Python example. It is recommended to use pipe and space separated format in case of RobotFramework example.
| + | |
| − | ** KW usage examples can be grouped and used in the class documentation string to provide better overview of the usage and relationships between KWs.
| + | |
| − | ** Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).
| + | |
| − | ** There must be included licence (/csit/docs/licence.rst) at the begging of each file.
| + | |
| − | | + | |
| − | * Coding
| + | |
| − | ** It is recommended to use some standard development tool (e.g. PyCharm Community Edition) and follow [https://www.python.org/dev/peps/pep-0008/ PEP-8] recommendations.
| + | |
| − | ** All python code (not only RF libraries) must adhere to PEP-8 standard. This is enforced by CSIT Jenkins verify job.
| + | |
| − | ** Indentation – do not use tab for indents! Indent is defined as four spaces.
| + | |
| − | ** Line length – limited to 80 characters.
| + | |
| − | ** Imports - use the full pathname location of the module, e.g. from resources.libraries.python.topology import Topology. Imports should be grouped in the following order: 1. standard library imports, 2. related third party imports, 3. local application/library specific imports. You should put a blank line between each group of imports.
| + | |
| − | ** Blank lines - Two blank lines between top-level definitions, one blank line between method definitions.
| + | |
| − | ** Do not use global variables inside library files.
| + | |
| − | ** Comparisons – should be in format 0 == ret_code not ret_code == 0 to avoid possible interchange of = (assignment) and == (equal to) that could be difficult to identify such error.
| + | |
| − | ** Constants – avoid to use hard-coded constants (e.g. numbers, paths without any description). Use configuration file(s), like /csit/resources/libraries/python/constants.py, with appropriate comments.
| + | |
| − | ** Logging – log at the lowest possible level of implementation (debugging purposes). Use same style for similar events. Keep logging as verbose as necessary.
| + | |
| − | ** Exceptions – use the most appropriate exception not general one („Exception“ ) if possible. Create your own exception if necessary and implement there logging, level debug.
| + | |
| − | | + | |
| − | ==CSIT working testcases==
| + | |
| − | | + | |
| − | ~/csit $ grep "^| [a-zA-Z]" -nr tests | grep -vi " | " | grep -v performance
| + | |
| − | tests/suites/bridge_domain/test.robot:27:| VPP reports interfaces
| + | |
| − | tests/suites/bridge_domain/test.robot:30:| Vpp forwards packets via L2 bridge domain 2 ports
| + | |
| − | tests/suites/bridge_domain/test.robot:42:| Vpp forwards packets via L2 bridge domain in circular topology
| + | |
| − | tests/suites/bridge_domain/test.robot:58:| Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:31:| VPP replies to ICMPv4 echo request
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:40:| TG can route to DUT egress interface
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:49:| TG can route to DUT2 through DUT1
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:58:| TG can route to DUT2 egress interface through DUT1
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:67:| TG can route to TG through DUT1 and DUT2
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:88:| VPP can process ICMP echo request from min to 1500B packet size with 1B increment
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:92:| VPP can process ICMP echo request from 1500B to max packet size with 10B increment
| + | |
| − | tests/suites/ipv4/ipv4_untagged.robot:110:| VPP responds to ARP request
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:33:| VPP replies to ICMPv6 echo request
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:37:| VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:41:| VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:59:| TG can route to first DUT egress interface
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:64:| TG can route to second DUT through first DUT
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:69:| TG can route to second DUT egress interface through first DUT
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
| + | |
| − | tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
| + | |
| − | tests/suites/l2_xconnect/l2_xconnect_untagged.robot:26:| Vpp forwards packets via L2 xconnect in circular topology
| + | |
| − | tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
| + | |
| − | tests/suites/vxlan/vxlan_xconnect_untagged.robot:29:| VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
| + | |
| − | | + | |
| − | ==CSIT code in development==
| + | |
| − | | + | |
| − | * 802.1ad/QinQ.
| + | |
| − | * vhost-user.
| + | |
| − | * basic HoneyComb handling in tests.
| + | |
| − | | + | |
| − | ==CSIT test cases development - draft plan==
| + | |
| − | | + | |
| − | * [[CSIT/FuncTestPlan|CSIT functional tests - draft dev plan]].
| + | |
| − | | + | |
| − | ==CSIT Tutorials==
| + | |
| − | * [[CSIT/Tutorials/Vagrant | Local execution of CSIT using Vagrant]]
| + | |
| − | | + | |
| − | ==CSIT Materials==
| + | |
| | * FD.io DevBoot - San Jose 4th to 7th of April 2016 - [https://drive.google.com/file/d/0B8xKFVHWtNPAdFBKcjhLQ2NLSjA/view?usp=sharing CSIT + Performance testing] | | * FD.io DevBoot - San Jose 4th to 7th of April 2016 - [https://drive.google.com/file/d/0B8xKFVHWtNPAdFBKcjhLQ2NLSjA/view?usp=sharing CSIT + Performance testing] |
| | + | * FD.io /dev/boot | June 2016 | Paris, France |
| | + | ** [https://wiki.fd.io/view/Events/Training-2016-05-Content Content and slides] |
| | + | *** [https://drive.google.com/file/d/0B0ZT6l65-x47MXo3OU1zdl9BOG8/view Day One - 1.3 CSIT Overview] |
| | + | *** [https://drive.google.com/file/d/0B0ZT6l65-x47Tmx6RlBmZlVWbkU/view Day Three - 3.5 Introduction to CSIT] |
| | + | ** [https://wiki.fd.io/view/Events/Training-2016-05-Content Videos] |
| | + | *** [https://www.youtube.com/watch?v=3tIXfx5QTlw&list=PLWHpG2-3ZXXuFm0r_TmpxzUV-Rr8L7Uc-&index=5 Day One - 1.3 CSIT Overview] |
| | + | *** [https://www.youtube.com/watch?v=BTjSHCgRpIg&list=PLWHpG2-3ZXXuFm0r_TmpxzUV-Rr8L7Uc-&index=23 Day Three - 3.5 Introduction to CSIT] |
| | + | * [[CSIT/TOIs | CSIT TOI Sessions]] |
| | + | * [[CSIT/CSIT Developer Documentation Notes | CSIT Developer Documentation Notes]] |
| | | | |
| − | ==CSIT Project Page Content to_be_added== | + | == List of all subpages (used or unused) == |
| − | | + | |
| − | * Documentation
| + | {{Special:PrefixIndex |prefix=CSIT|hideredirects=1}} |
| − | ** Descriptions
| + | |
| − | *** CSIT code that's running: functional and performance tests.
| + | |
| − | *** CSIT testbeds
| + | |
| − | **** Operational: physical, virtual-VIRL.
| + | |
| − | **** Non-operational: virtual-VM.
| + | |
| − | *** Running CSIT integration with FD.io gerrit and jenkins.
| + | |
| − | ** CSIT usability manual
| + | |
| − | *** Instructions how to use CSIT to verify VPP code commits and avoid breaking it.
| + | |
| − | *** Onboarding of new VPP test cases onto CSIT system.
| + | |
| − | * CSIT work in progress
| + | |
| − | ** Code in development, priorities.
| + | |
| − | ** Development plan, priorities.
| + | |
| − | ** FD.io jira for tracking execution against the plan, scheme proposal.
| + | |
