Difference between revisions of "CSIT"

From fd.io
Jump to: navigation, search
(Reapplying subpage list edit and CDash releases edit.)
 
(115 intermediate revisions by 13 users not shown)
Line 1: Line 1:
 +
 
{{Project Facts
 
{{Project Facts
 
|name=CSIT
 
|name=CSIT
 
|shortname=csit
 
|shortname=csit
 
|jiraName=CSIT
 
|jiraName=CSIT
 +
|projectLead=Maciek Konstantynowicz
 
|committers=
 
|committers=
−
* Matej Klotton
+
* Tibor Frank
−
* Stefan Kobza
+
−
* Carsten Koester
+
 
* Maciek Konstantynowicz
 
* Maciek Konstantynowicz
−
* Miroslav Miklus
+
* Peter Mikus
 +
* Vratko Polak
 
* Dave Wallace
 
* Dave Wallace
 
}}
 
}}
  
−
==Description==
+
== Get Involved ==
−
CSIT (Continuous System Integration and Testing) project is focusing on:
+
−
# Development of software code for fully automated VPP code testing, functionality, performance, regression and new functions.
+
−
# Execution of CSIT test suites on VPP code running on LF FD.io virtual and physical compute environments.
+
−
# Integration with FD.io continuous integration systems (Gerrit, Jenkins and such).
+
−
# Identified existing FD.io project dependencies and interactions:
+
−
#* vpp - Vector Packet Processing.
+
−
#* honeycomb - Honeycomb Agent for management plane testing.
+
−
#* ci-management - Management repo for Jenkins Job Builder, script and management related to the Jenkins CI configuration.
+
−
 
+
−
==Get Involved==
+
 
* [[CSIT/Meeting|Weekly CSIT Meeting]].
 
* [[CSIT/Meeting|Weekly CSIT Meeting]].
−
* [https://lists.fd.io/mailman3/lists/csit-dev.lists.fd.io/ Join the CSIT Mailing List].
+
* [https://wiki.fd.io/view/CSIT/TWS On-demand Technical Workstream Meetings].
 +
* [https://lists.fd.io/mailman/listinfo/csit-dev Join the CSIT Mailing List].
 
* [[IRC | Join fdio-csit IRC channel]].
 
* [[IRC | Join fdio-csit IRC channel]].
  
−
==Scope==
+
== Start Here ==
−
Fd.io CSIT project scope:
+
* [[CSIT/Description | What is CSIT (Continuous System Integration and Testing)?]]
−
 
+
* [https://csit.fd.io/cdocs/overview/csit/design/ CSIT Design]
−
# Automated regression testing of VPP code changes
+
* [https://csit.fd.io/cdocs/overview/c_dash/design/ CDASH Design]
−
#* Functionality of VPP data plane, network control plane, management plane against functional specifications.
+
* [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=8 CSIT Development Tasks (Jira)]
−
#* Performance of VPP data plane including non-drop-rate packet throughput and delay, against established reference benchmarks.
+
* [https://jenkins.fd.io/view/csit/ CSIT Jenkins Jobs ]
−
#* Performance of network control plane against established reference benchmarks.
+
* [https://csit.fd.io/cdocs/overview/csit/branching_strategy/ CSIT Branching Strategy]
−
#* Performance of management plane against established reference benchmarks.
+
* [https://csit.fd.io/cdocs/infrastructure/fdio_dc_testbed_specifications/ FD.io CSIT testbeds specification]
−
# Test case definitions driven by supported and planned VPP functionality, interfaces and performance:
+
* [https://csit.fd.io/ CSIT Performance Dashboard]
−
#* Uni-dimensional tests: Data plane, (Network) Control plane, Management plane.
+
** [https://csit.fd.io/trending/ Performance Trending]
−
#* Multi-dimensional tests: Use case driven.
+
** [https://csit.fd.io/report/ Per Release Performance]
−
# Integration with FD.io Continuous Integration system including FD.io Gerrit and Jenkins
+
** [https://csit.fd.io/stats/ Test Job Statistics]
−
#* Automated test execution triggered by VPP-VERIFY jobs other VPP and CSIT project jobs.
+
** [https://csit.fd.io/news/ Failures and Anomalies]
−
# Integration with LF VPP test execution environment
+
* [https://wiki.fd.io/view/CSIT/TestFailuresTracking CSIT Test Failures Summary and Status]
−
#* Functional tests execution on LF hosted VM environment.
+
−
#* Performance and functional tests execution on LF hosted physical compute environment.
+
−
#* Subset of tests executed on LF hosted physical compute running VIRL (Virtual Internet Routing Lab).
+
−
 
+
−
==CSIT Code Structure==
+
−
 
+
−
CSIT project consists of the following:
+
−
* RobotFramework tests, resources, and libraries.
+
−
* bash scripts – tools, and anything system-related (copying files, installing SW on nodes, ...).
+
−
* Python libraries
+
−
** the brains of the execution.
+
−
** for different functionality there is a different module, i.e.
+
−
*** vpp
+
−
**** ipv4 utils.
+
−
**** ipv6 utils.
+
−
**** xconnect.
+
−
**** bdomain.
+
−
**** VAT (vpp_api_test) helpers.
+
−
**** Config generator.
+
−
*** ssh.
+
−
*** topology.
+
−
*** packet verifier – packet generator and validator.
+
−
*** v4/v6 ip network and host address generator.
+
−
* vpp_api_test templates.
+
−
 
+
−
Each RF testsuite/case has TAGs associated with it that describe what environment that it can be run on: HW/VM, or what topology it requires. RobotFramework is executed with parameter that links to topology description file, we call it topology for simplicity. This file is parsed to variable “nodes” and later used in test cases and libraries.
+
−
 
+
−
In general test cases are written in readable English, so that even non-coders can understand it. These top level test cases should stay the same; in other words the testcase text should not represent “how” the test is done, but “what” the test case does.
+
−
 
+
−
Libraries to handle VPP functionality are written in Python and are separated on per-feature basis: v4, v6, interface (admin up, state status and so on), xconnect and bdomain. More modules are going to be implemented when needed.
+
−
 
+
−
Performance tests are executed using packet traffic generators external to servers running VPP code. Python APIs are used to control the traffic generators. Linux Foundation hosts physical infrastructure dedicated to FD.io, consisting of three of 3-compute-node performance testbeds (compute node = x86_64 multi-core server). Two of the compute nodes run VPP code, one runs a software traffic generator. Currently CSIT performance tests are executed using [https://github.com/cisco-system-traffic-generator/trex-core trex].
+
−
 
+
−
==CSIT working testcases==
+
−
 
+
−
~/csit $ grep "^| [a-zA-Z]" -nr tests | grep -vi " | "  | grep -v performance
+
−
tests/suites/bridge_domain/test.robot:27:| VPP reports interfaces
+
−
tests/suites/bridge_domain/test.robot:30:| Vpp forwards packets via L2 bridge domain 2 ports
+
−
tests/suites/bridge_domain/test.robot:42:| Vpp forwards packets via L2 bridge domain in circular topology
+
−
tests/suites/bridge_domain/test.robot:58:| Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
+
−
tests/suites/ipv4/ipv4_untagged.robot:31:| VPP replies to ICMPv4 echo request
+
−
tests/suites/ipv4/ipv4_untagged.robot:40:| TG can route to DUT egress interface
+
−
tests/suites/ipv4/ipv4_untagged.robot:49:| TG can route to DUT2 through DUT1
+
−
tests/suites/ipv4/ipv4_untagged.robot:58:| TG can route to DUT2 egress interface through DUT1
+
−
tests/suites/ipv4/ipv4_untagged.robot:67:| TG can route to TG through DUT1 and DUT2
+
−
tests/suites/ipv4/ipv4_untagged.robot:88:| VPP can process ICMP echo request from min to 1500B packet size with 1B increment
+
−
tests/suites/ipv4/ipv4_untagged.robot:92:| VPP can process ICMP echo request from 1500B to max packet size with 10B increment
+
−
tests/suites/ipv4/ipv4_untagged.robot:110:| VPP responds to ARP request
+
−
tests/suites/ipv6/ipv6_untagged.robot:33:| VPP replies to ICMPv6 echo request
+
−
tests/suites/ipv6/ipv6_untagged.robot:37:| VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
+
−
tests/suites/ipv6/ipv6_untagged.robot:41:| VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
+
−
tests/suites/ipv6/ipv6_untagged.robot:59:| TG can route to first DUT egress interface
+
−
tests/suites/ipv6/ipv6_untagged.robot:64:| TG can route to second DUT through first DUT
+
−
tests/suites/ipv6/ipv6_untagged.robot:69:| TG can route to second DUT egress interface through first DUT
+
−
tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
+
−
tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
+
−
tests/suites/l2_xconnect/l2_xconnect_untagged.robot:26:| Vpp forwards packets via L2 xconnect in circular topology
+
−
tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
+
−
tests/suites/vxlan/vxlan_xconnect_untagged.robot:29:| VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
+
  
−
==CSIT code in development==
+
== Test Reports ==
 +
* [https://csit.fd.io/cdocs/release_notes/current/ Current release]
 +
* [https://csit.fd.io/cdocs/release_notes/previous/ Previous releases]
  
−
* 802.1ad/QinQ.
+
== CSIT Code Documentation ==
−
* vhost-user.
+
* [https://s3-docs.fd.io/csit/master/docs CSIT master Documentation]
−
* basic HoneyComb handling in tests.
+
* [https://s3-docs.fd.io/csit/rls2206/docs/ CSIT-2206 Documentation]
 +
* [https://s3-docs.fd.io/csit/rls2202/docs/ CSIT-2202 Documentation]
 +
* [https://s3-docs.fd.io/csit/rls2110/docs/ CSIT-2110 Documentation]
 +
* [https://s3-docs.fd.io/csit/rls2106/docs/ CSIT-2106 Documentation]
 +
* [https://docs.fd.io/csit/rls2101/doc/ CSIT-2101 Documentation]
 +
* [https://docs.fd.io/csit/rls2009/doc/ CSIT-2009 Documentation]
 +
* [https://docs.fd.io/csit/rls2005/doc/ CSIT-2005 Documentation]
 +
* [https://docs.fd.io/csit/rls2001/doc/ CSIT-2001 Documentation]
 +
* [https://docs.fd.io/csit/rls1908/doc/ CSIT-1908 Documentation]
 +
* [https://docs.fd.io/csit/rls1904/doc/ CSIT-1904 Documentation]
 +
* [https://docs.fd.io/csit/rls1901/doc/ CSIT-1901 Documentation]
 +
* [https://docs.fd.io/csit/rls1810/doc/ CSIT-1810 Documentation]
 +
* [https://docs.fd.io/csit/rls1804/doc/ CSIT-1804 Documentation]
 +
* [https://docs.fd.io/csit/rls1801/doc/ CSIT-1801 Documentation]
 +
* [https://docs.fd.io/csit/rls1710/doc/ CSIT-1710 Documentation]
 +
* [https://docs.fd.io/csit/rls1707/doc/ CSIT-1707 Documentation]
 +
* [https://docs.fd.io/csit/rls1704/doc/ CSIT-1704 Documentation]
 +
* [https://docs.fd.io/csit/rls1701/doc/ CSIT-1701 Documentation]
 +
* [[CSIT/Documentation| CSIT 16.09/16.06 Documentation]]
  
−
==CSIT Project Page Content to_be_added==
+
== Training ==
 +
* FD.io DevBoot - San Jose 4th to 7th of April 2016 -  [https://drive.google.com/file/d/0B8xKFVHWtNPAdFBKcjhLQ2NLSjA/view?usp=sharing CSIT + Performance testing]
 +
* FD.io /dev/boot | June 2016 | Paris, France
 +
** [https://wiki.fd.io/view/Events/Training-2016-05-Content Content and slides]
 +
*** [https://drive.google.com/file/d/0B0ZT6l65-x47MXo3OU1zdl9BOG8/view Day One - 1.3 CSIT Overview]
 +
*** [https://drive.google.com/file/d/0B0ZT6l65-x47Tmx6RlBmZlVWbkU/view Day Three - 3.5 Introduction to CSIT]
 +
** [https://wiki.fd.io/view/Events/Training-2016-05-Content Videos]
 +
*** [https://www.youtube.com/watch?v=3tIXfx5QTlw&list=PLWHpG2-3ZXXuFm0r_TmpxzUV-Rr8L7Uc-&index=5 Day One - 1.3 CSIT Overview]
 +
*** [https://www.youtube.com/watch?v=BTjSHCgRpIg&list=PLWHpG2-3ZXXuFm0r_TmpxzUV-Rr8L7Uc-&index=23 Day Three - 3.5 Introduction to CSIT]
 +
* [[CSIT/TOIs | CSIT TOI Sessions]]
 +
* [[CSIT/CSIT Developer Documentation Notes | CSIT Developer Documentation Notes]]
  
−
* Documentation
+
== List of all subpages (used or unused) ==
−
** Descriptions
+
−
*** CSIT code that's running: functional and performance tests.
+
{{Special:PrefixIndex |prefix=CSIT|hideredirects=1}}
−
*** CSIT testbeds
+
−
**** Operational: physical, virtual-VIRL.
+
−
**** Non-operational: virtual-VM.
+
−
*** Running CSIT integration with FD.io gerrit and jenkins.
+
−
** CSIT usability manual
+
−
*** Instructions how to use CSIT to verify VPP code commits and avoid breaking it.
+
−
*** Onboarding of new VPP test cases onto CSIT system.
+
−
* CSIT work in progress
+
−
** Code in development, priorities.
+
−
** Development plan, priorities.
+
−
** FD.io jira for tracking execution against the plan, scheme proposal.
+

Latest revision as of 14:31, 4 March 2025

CSIT Facts

Project Lead: Maciek Konstantynowicz
Committers:

  • Tibor Frank
  • Maciek Konstantynowicz
  • Peter Mikus
  • Vratko Polak
  • Dave Wallace

Repository: git clone https://gerrit.fd.io/r/csit
Mailing List: csit-dev@lists.fd.io
Jenkins: jenkins silo
Gerrit Patches: code patches/reviews
Bugs: CSIT bugs

Get Involved

Start Here

Test Reports

CSIT Code Documentation

Training

List of all subpages (used or unused)

CSITCSIT/AArch64CSIT/Branching Strategy
CSIT/CSIT Developer Documentation NotesCSIT/CSIT LF VIRL testbedCSIT/CSIT LF testbed
CSIT/CSIT Update TestbedCSIT/Compute hardware settingsCSIT/DPDK16.07 performance
CSIT/DescriptionCSIT/Design OptimizationsCSIT/Documentation
CSIT/FuncTestPlanCSIT/Honeycomb/HC1704 planCSIT/Honeycomb/HC1707 plan
CSIT/Honeycomb/HC performanceCSIT/Honeycomb/StatusCSIT/Honeycomb/Summary
CSIT/JobsCSIT/LFN-Dev-and-Test-Forum-June2023CSIT/Meeting
CSIT/Meeting-minutes-2019-12-18CSIT/Meeting MinutesCSIT/Meeting Minutes 2016
CSIT/Meeting Minutes 2017CSIT/Meeting Minutes 2018CSIT/Meeting Minutes 2019
CSIT/Meeting Minutes 2020CSIT/Meeting Minutes 2021CSIT/Meeting Minutes 2022
CSIT/Meeting Minutes 2023CSIT/Meeting Minutes 2024CSIT/PerfTestPlan
CSIT/PerfTestPlan/SRv6CSIT/PerformanceTrendingAnalysisCSIT/Submitting Patches
CSIT/TOIsCSIT/TWSCSIT/TestFailuresTracking
CSIT/Testbeds: Xeon Hsw, VIRL.CSIT/Testbeds: Xeon Skx, Arm, Atom.CSIT/Tracking
CSIT/Tutorials/Vagrant/Virtualbox/MacOsXCSIT/Tutorials/Vagrant/Virtualbox/UbuntuCSIT/Tutorials/Vagrant/Virtualbox/Windows7
CSIT/VPP-16.06 Test ReportCSIT/VPP-16.06 Test Report DraftCSIT/VPP-16.06 Test Report Sandbox
CSIT/VPP-16.09 Test ReportCSIT/VPP-16.09 Test Report DraftCSIT/VPP-ubuntu-compare-1404to1604
CSIT/VPP Device test designCSIT/VnetInfraPlanCSIT/csit-func-jobs-reorg-proposal
CSIT/csit-func-tests-reorg-proposalCSIT/csit-perf-env-tuning-ubuntu1604-obsoleteCSIT/csit-perf-jobs-reorg-proposal
CSIT/csit-perf-tc-naming-changeCSIT/csit-test-namingCSIT/csit1701 plan
CSIT/csit1701 report outlineCSIT/csit1704 planCSIT/csit1707 plan
CSIT/csit1710 planCSIT/csit1801 planCSIT/csit1804 plan
CSIT/csit1807 planCSIT/csit1810 planCSIT/csit1901 plan
CSIT/csit1904 planCSIT/csit1908 planCSIT/csit2001 plan
CSIT/csit2005 planCSIT/csit2009 planCSIT/csit2101 plan
CSIT/csit2106 planCSIT/csit2110 planCSIT/csit2202 plan
CSIT/csit2206 planCSIT/csit2210 planCSIT/csit2302 plan
CSIT/csit2306 planCSIT/csit2310 planCSIT/csit2402 plan
CSIT/csit2406 planCSIT/csit2410 planCSIT/csit2502 plan
CSIT/csit lab 2022 planCSIT/csit tg serversCSIT/fdio csit lab ext lld draft
CSIT/func1609-draftCSIT/long1076vs1080-ubuntu1404vs1604-analysisCSIT/long1076vs1080-ubuntu1404vs1604-thput-decr
CSIT/long1076vs1080-ubuntu1404vs1604-thput-incrCSIT/old yaml wip modelCSIT/perf1609-draft
CSIT/vhostuser test scenariosCSIT/vnetinfra1609-draft
Retrieved from "https://wiki.fd.io/index.php?title=CSIT&oldid=12849"