Difference between revisions of "CSIT/Design Optimizations"
m (→RF Keywords Naming Guidelines) |
m |
||
(One intermediate revision by one other user not shown) | |||
Line 1: | Line 1: | ||
− | |||
− | |||
__TOC__ | __TOC__ | ||
− | |||
− | |||
==Motivation== | ==Motivation== | ||
Line 102: | Line 98: | ||
## testbed environment setup ansible playbooks; | ## testbed environment setup ansible playbooks; | ||
## operational debugging scripts; | ## operational debugging scripts; | ||
+ | |||
+ | ==Presentation and Analytics Layer== | ||
+ | |||
+ | The presentation and analytics layer (PAL) is the fourth layer of CSIT | ||
+ | hierarchy. | ||
+ | |||
+ | [[File:Pal layers.png|Presentation and Analytic Layer]] | ||
+ | |||
+ | The model of presentation and analytics layer consists of four | ||
+ | sub-layers, bottom up: | ||
+ | |||
+ | * sL1 - Data - input data to be processed: | ||
+ | **Static content - .rst text files, .svg static figures, and other files stored in the CSIT git repository. | ||
+ | **Data to process - .xml files generated by Jenkins jobs executing tests, stored as robot results files (output.xml). | ||
+ | **Specification - .yaml file with the models of report elements (tables, plots, layout, ...) generated by this tool. There is also the configuration of the tool and the specification of input data (jobs and builds). | ||
+ | *sL2 - Data processing | ||
+ | **The data are read from the specified input files (.xml) and stored as multi-indexed pandas.Series. | ||
+ | **This layer provides also interface to input data and filtering of the input data. | ||
+ | *sL3 - Data presentation - This layer generates the elements specified in the specification file: | ||
+ | **Tables: .csv files linked to static .rst files. | ||
+ | **Plots: .html files generated using plot.ly linked to static .rst files. | ||
+ | **Files: Generated .rst files | ||
+ | *sL4 - Report generation - Sphinx generates required formats and versions: | ||
+ | **formats: html, pdf | ||
+ | **versions: html version includes all information, pdf version is generated without detailed information about tests | ||
==Test Lifecycle Abstraction== | ==Test Lifecycle Abstraction== |
Latest revision as of 08:34, 12 October 2017
Contents
- 1 Motivation
- 2 Proposed CSIT Design Optimizations
- 3 Proposed CSIT Robot Framework Keywords Optimisations
- 4 CSIT System Design Hierarchy
- 5 Presentation and Analytics Layer
- 6 Test Lifecycle Abstraction
- 7 RF Keywords Functional Classification
- 8 RF Keywords Naming Guidelines
- 9 Test Suite Naming Guidelines
- 10 Patch Review Checklist
- 11 CSIT Directory Structure
- 12 Applying CSIT Design Guidelines
Motivation
FD.io CSIT system design needs to meet continuously expanding requirements of FD.io projects including VPP, related sub-systems (e.g. plugin applications, DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as growing number of compute platforms running those applications. With CSIT project scope and charter including both FD.io continuous testing AND performance trending/comparisons, those evolving requirements further amplify the need for CSIT framework modularity, flexibility and usability.
Following analysis of the existing CSIT design and related inventory of CSIT Robot Framework KeyWords and libraries, CSIT project team decided to make a number of CSIT design optimizations to address those evolving requirements.
This note describes the proposed CSIT design optimizations, and positions them in the context of the updated CSIT framework design. It also includes the updated Robot Framework (RF) Keyword (KW) Naming Style Guidelines to ensure readability and usability of CSIT test framework, and to address existing CSIT KW naming inconsistencies.
Proposed CSIT Design Optimizations
Proposed CSIT design optimizations focus on updated CSIT design guidelines for Robot Framework Test and Keywords usage and can be summarized as follows:
- Strictly follow the CSIT Design Hierarchy described in this document;
- Tests MUST use only L2 RF Keywords (KWs);
- All L2 KWs are composed of L1 KWs (either RF or Python) and/or other L2 KWs;
- Python L1 KWs are preferred to RF L1 KWs;
- RF L1 KWs should be reduced to minimum;
- Test and KW naming should be following CSIT naming guidelines;
For more detailed description of how these guidelines apply to developing new CSIT Tests and Keywords see section "Applying CSIT Design Guidelines".
Proposed CSIT Robot Framework Keywords Optimisations
- RF KW renaming
- see naming guidelines in this document;
- EF KW re-classification
- breakdown libraries into multiple files
- mainly performance.robot, as everything is in one file;
- group KWs per infra and functional area similar to structure of csit rls reports;
- breakdown libraries into multiple files
- RF KW move levels
- few L2 KWs should be L1 KWs, as never used in tests, and dealing with lower level abstractions;
- EF KW refactoring;
- merge KWs that are always used together;
- KWs that are used 1-2 times should be merged to parent KW, unless further re-use expected;
- for TestSetup and TestTeardown - use one L2 KW only;
- SuiteSetup and SuiteTeardown already follow this;
- unused KW removal
- remove unused and unwanted RF KWs;
- apart from the ones prepared for future use of course;
- update CSIT framework documentation
- use content of this document;
- see CSIT wiki link
- Getting started with CSIT development;
- https://wiki.fd.io/view/CSIT/Documentation;
CSIT System Design Hierarchy
CSIT follows a hierarchical system design with SUTs and DUTs at the bottom level, and presentation level at the top level, with a number of functional layers in-between. The current CSIT design including CSIT framework is depicted in the diagram below.
A brief bottom-up description is provided here:
- SUTs, DUTs, TGs:
- SUTs - Systems Under Test;
- DUTs - Devices Under Test;
- TGs - Traffic Generators;
- Level-1 libraries - Robot and Python:
- lowest level CSIT libraries abstracting underlying test environment, SUT, DUT and TG specifics;
- used commonly across multiple L2 KWs;
- performance and functional tests:
- L1 KWs (KeyWords) are implemented as RF libraries and Python libraries;
- proposal is to stop implementing L1 functions with RF KWs and do them in Python only instead
- honeycomb - good example, L1 all in .py, L2 all in RF KWs;
- over time convert perf and func L1 RF KWs to Python;
- performance TG L1 KWs:
- all L1 KWs are implemented as Python libraries;
- support for TRex only today;
- need to add IXIA;
- performance data plane traffic profiles:
- TG-specific stream profiles provide full control of:
- packet definition – layers, MACs, IPs, ports, combinations thereof e.g. IPs and UDP ports;
- stream definitions - different streams can run together, delayed, one after each other;
- stream profiles are independent of CSIT framework and can be used in any T-rex setup, can be sent anywhere to repeat tests with exactly the same setup;
- easily extensible – one can create a new stream profile that meets tests requirements;
- same stream profile can be used for different tests with the same traffic needs;
- TG-specific stream profiles provide full control of:
- functional data plane traffic scripts:
- Scapy specific traffic scripts;
- Level-2 libraries - Robot resource files:
- higher level CSIT libraries abstracting required functions for executing tests;
- L2 KWs are classified into the following functional categories:
- configuration, test, verification, state report;
- suite setup, suite teardown;
- test setup, test teardown;
- Tests - Robot:
- Test suites with test cases;
- Functional tests using VIRL environment:
- VPP;
- HoneyComb;
- Performance tests using physical testbed environment:
- VPP;
- Testpmd;
- Tools:
- documentation generator;
- report generator;
- testbed environment setup ansible playbooks;
- operational debugging scripts;
Presentation and Analytics Layer
The presentation and analytics layer (PAL) is the fourth layer of CSIT hierarchy.
The model of presentation and analytics layer consists of four sub-layers, bottom up:
- sL1 - Data - input data to be processed:
- Static content - .rst text files, .svg static figures, and other files stored in the CSIT git repository.
- Data to process - .xml files generated by Jenkins jobs executing tests, stored as robot results files (output.xml).
- Specification - .yaml file with the models of report elements (tables, plots, layout, ...) generated by this tool. There is also the configuration of the tool and the specification of input data (jobs and builds).
- sL2 - Data processing
- The data are read from the specified input files (.xml) and stored as multi-indexed pandas.Series.
- This layer provides also interface to input data and filtering of the input data.
- sL3 - Data presentation - This layer generates the elements specified in the specification file:
- Tables: .csv files linked to static .rst files.
- Plots: .html files generated using plot.ly linked to static .rst files.
- Files: Generated .rst files
- sL4 - Report generation - Sphinx generates required formats and versions:
- formats: html, pdf
- versions: html version includes all information, pdf version is generated without detailed information about tests
Test Lifecycle Abstraction
A well coded test must follow a disciplined abstraction of the test lifecycles that includes setup, configuration, test and verification. In addition to improve test execution efficiency, the commmon aspects of test setup and configuration shared across multiple test cases should be done only once. Translating these high-level guidelines into the Robot Framework one arrives to definition of a well coded RF tests for FD.io CSIT.
Anatomy of Good Tests for CSIT:
- Suite Setup - Suite startup Configuration common to all Test Cases in suite: uses Configuration KWs, Verification KWs, StateReport KWs;
- Test Setup - Test startup Configuration common to multiple Test Cases: uses Configuration KWs, StateReport KWs;
- Test Case - uses L2 KWs with RF Gherkin style:
- prefixed with {Given} - Verification of Test setup, reading state: uses Configuration KWs, Verification KWs, StateReport KWs;
- prefixed with {When} - Test execution: Configuration KWs, Test KWs;
- prefixed with {Then} - Verification of Test execution, reading state: uses Verification KWs, StateReport KWs;
- Test Teardown - post Test teardown with Configuration cleanup and Verification common to multiple Test Cases - uses: Configuration KWs, Verification KWs, StateReport KWs;
- Suite Teardown - Suite post-test Configuration cleanup: uses Configuration KWs, Verification KWs, StateReport KWs;
RF Keywords Functional Classification
CSIT RF KWs are classified into the functional categories matching the test lifecycle events described earlier. All CSIT RF L2 and L1 KWs have been grouped into the following functional categories:
- Configuration;
- Test;
- Verification;
- StateReport;
- SuiteSetup;
- TestSetup;
- SuiteTeardown;
- TestTeardown;
All existing CSIT L1 and L2 RF KWs have been classified into one of above categories in the spreadsheet that is accompanying this document [mk][add reference to link/xls filename]. Going forward all coded RF Keywords will be tagged using functional categories listed above. To improve readability and usability of CSIT RF KWs, a simple naming convention is proposed for each of the KW categories identified above. Hence while coding RF KWs it is important to keep in mind the category of the coded KW.
RF Keywords Naming Guidelines
Readability counts: "..code is read much more often than it is written." Hence following a good and consistent grammar practice is important when writing RF KeyWords and Tests.
All CSIT test cases are coded using Gherkin style and include only L2 KWs references. L2 KWs are coded using simple style and include L2 KWs, L1 KWs, and L1 python references. To improve readability, the proposal is to use the same grammar for both RF KW styles, and to formalize the grammar of English sentences used for naming the RF KWs.
RF KWs names are short sentences expressing functional description of the command. They must follow English sentence grammar in one of the following forms:
- Imperative - verb-object(s): "Do something", verb in base form.
- Declarative - subject–verb–object(s): "Subject does something", verb in a third-person singular present tense form.
- Affirmative - modal_verb-verb-object(s): "Subject should be something", "Object should exist", verb in base form.
- Negative - modal_verb-Not-verb-object(s): "Subject should not be something", "Object should not exist", verb in base form.
Passive form MUST NOT be used. However a usage of past participle as an adjective is okay. See usage examples.
Following sections list applicability of the above grammar forms to different RF KW categories. Usage examples are provided, both good and bad.
Configuration KWs
Configuration KWs MUST use the following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: add, assign, apply, build, create, clear, connect, configure, disable, enable, generate, initialize, kill, read, remove, save, set, stop, update, write;
- Used objects: topology, path, result;
- Good examples:
- {Stop honeycomb service on DUTs};
- {Build QEMU on all DUTs};
- {Stop and Clear QEMU};
- {Clear all ACL settings};
- Bad examples => Renaming:
- {Setup Topology for IPv4 IPsec testing} => {Configure topology for IPv4 IPsec testing};
- {Setup all DUTs before test} => {Configure all DUTs before test};
- {Path for 3-node BD-SHG testing is set} => {Configure path for 3-node BD-SHG test};
- {Bridge domain on DUT node is created} => {Create L2 bridge-domain on DUT node};
- {L2 bridge domain initialized in a 3-node circular topology} => {Initialize L2 bridge-domain in 3-node topology};
- {Routes are set up for IPv4 testing} => {Configure routes for IPv4 test};
- {VPP is restarted} => {Restart VPP};
- {Setup ipv6 to all dut in topology} => {Configure ipv6 on all DUTs};
- . Declarative - subject–verb–object(s): "Subject does something", verb in a third-person singular present tense form:
- Used subjects: Honeycomb;
- Used verbs: adds, assigns, applies, builds, creates, clears, connects, configures, disables, enables, generates, initializes, kills, reads, removes, saves, sets, stops, updates, writes;
- Used objects: topology, path, result;
- Good examples:
- {Honeycomb configures DHCP relay};
- {Honeycomb adds locator set};
- {Honeycomb sets interface state};
- {Honeycomb creates vhost-user interface};
- Bad examples => Renaming:
- {Vpp show stats} => {Show vpp statistics};
- Usage notes:
- use imperative form if interacting directly with DUT;
- use declarative form if interacting with DUT via control agent, use agent name as the Subject;
Test KWs
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: execute, find, get, measure, ping, put, send, test;
- Used objects: ICMPv4, {NDR using binary search}, {latency}, {IP address};
- Good examples:
- {Display result of NDR search};
- {Clear and show runtime counters with running traffic};
- Bad examples => Renaming:
- {Find NDR using binary search and pps} => {Find NDR using binary search};
- {Measure latency pps} => {Measure packet latency};
- {Ipv4 icmp echo sweep} => {Execute IPv4 icmp echo sweep};
- {Node replies to ICMP echo request} => {Node replies to ICMP echo request};
- {Ipv6 tg to dut1 egress} => {Send IPv6 icmp echo request to DUT1 egress interface and verify answer};
- {Send and receive ICMPv4 bidirectionally} => {Send ICMPv4 bidirectionally and verify received packets};
- {Send and receive ICMP Packet} => {Send IPv6 icmp echo request and verify answer};
Verification KWs
Verification KWs MUST use following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: check, fail, get, receive, send, show;
- Used objects: ... ;
- Good examples:
- {Send ICMPv4 and check received GRE header};
- {Ping and verify IP address};
- {Send IPv6 UDP and check IPv4 headers for MAP-T};
- Bad examples => Renaming:
- {Check DHCP DISCOVER header} => {Verify DHCP DISCOVER header};
- {Send and receive ICMP Packet} => {Send ICMP packet and verify received packet};
- {Send ARP Request should failed} => {ARP request should fail};
- Affirmative - Subject-modal_verb-verb-object(s): "Subject should be in some state", "Subject should exist", verb in base form:
- Used verbs: exist, ... ;
- Used objects: empty, ... ;
- Good examples:
- {IPv4 address from Honeycomb should be empty};
- {VAT should show interfaces assigned to bridge domain};
- {Traffic should pass with no loss};
- Bad examples => Renaming:
- {Interface ACL settings from Honeycomb should be} => {Interface ACL settings from Honeycomb should exist};
- {ACL session from Honeycomb should be} => {ACL session from Honeycomb should exist};
- {Bridge domain configuration from Honeycomb should be} => {Bridge domain configuration from Honeycomb should exist};
- {Lisp locator_set is set correctly} => {Lisp locator_set should be set correctly};
- Negative - Subject-modal_verb-Not-verb-object(s): "Subject should not be in some state", "Subject should not exist", verb in base form:
- Used verbs: exist, ... ;
- Used objects: empty, ... ;
- Good examples:
- {ACL table from VAT should not exist};
- {Lisp mappings from Honeycomb should not exist};
- Bad examples => Renaming:
- {Send TCP or UDP packet should failed} => {TCP or UDP packet transmission should fail};
- {Send packet from Port to Port should failed} => {Packet transmission from port to port should fail};
- {Send ARP Request should failed} => {ARP request should fail};
StateReport KWs
StateReport KWs MUST use following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: log, dump, report, show;
- Used objects: SNAT verbos, stats, trace dump, vpp version, vpp vhost;
- Good examples:
- {Show SNAT verbose};
- Bad examples => Renaming:
- {Log routing configuration from VAT} => {Log routing configuration using VAT};
- {VPP reports interfaces through VAT on} => {Report interfaces using VAT};
- {VPP reports interfaces on} => {Report interfaces};
SuiteSetup KWs
SuiteSetup KWs MUST use following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: set up, update;
- Used objects: framework, all DUTs, all interface data on all nodes;
- Good examples:
- {Set up 3-node topology};
- Bad examples => Renaming:
- {3-node circular Topology Variables Setup} => {Set variables in 3-node topology};
- {2-node circular Topology Variables Setup with DUT interface model} => {Set variables in 2-node topology with DUT interface model};
- {2-node-switched Performance SuiteSetup with DUT's NIC model} => {Set up 2-node-switched performance suite with DUT NIC model};
- Usage notes:
- Preference to use a single SuiteSetup KW in Suite Setup section in test file;
SuiteTeardown KWs
SuiteTeardown KWs MUST use following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: tear down;
- Used objects: framework, all DUTs, all interface data on all nodes;
- Good examples:
- {...};
- Bad examples => Renaming:
- {3-node Performance Suite Teardown} => {Tear down 3-node performance topology};
- {2-node Performance Suite Teardown} => {Tear down 2-node performance topology};
- Usage notes:
- Preference to use a single SuiteTeardown KW in Suite Teardown section in test file;
TestSetup KWs
TestSetup KWs MUST use following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form:
- Used verbs: set up, update;
- Used objects: all DUTs, all interface data on all nodes;
- Good examples:
- {...};
- Bad examples => Renaming:
- {Func Test Setup} => {Set up functional test};
- {Setup all DUTs before test} => {Configure all DUTs before test};
- {Setup all TGs before traffic script} => {Configure all TGs for traffic script};
- Usage notes:
- Preference to use a single TestSetup KW in Test Setup section in test file;
TestTeardown KWs
TestTeardown KWs MUST use following grammar forms:
- Imperative - verb-object(s): "Do something", verb in base form;
- . Used verbs: tear down;
- Used objects: framework, all DUTs, all interface data on all nodes;
- Good examples:
- {Show runtime counters on all DUTs};
- Bad examples => Renaming:
- {Func Test Teardown} => {Tear down functional test};
- {Performance test teardown} => {Tear down topology for performance discovery tests};
- {Performance ndrchk test teardown} => {Tear down topology for performance ndrchk tests};
- {Performance test with vhost and VM with dpdk-testpmd teardown} => {Tear down topology for performance tests with vhost and VM with dpdk-testpmd};
- {Qemu Teardown} => {Tear down QEMU};
- {Guest VM with dpdk-testpmd Teardown} => {Tear down guest VM with dpdk-testpmd};
- Usage notes:
- Preference to use a single TestTeardown KW in Test Teardown section in test file;
Usage of Variables in Tests and KW Name
Additional good practice guidelines for using variables in Keyword names:
- Variables should be used in KW names if it improves readability
- Use this approach if KWs requires 1 or 2 arguments
- do not use this approach if more than 2..3 arguments needed by KW;
- do not use arguments in the name and then use the argument section;
- Good examples:
- {Add '${m}' worker threads and rxqueues '${n}' in 3-node single-link topo}
- Use this approach if KWs requires 1 or 2 arguments
For L2 KWs used in tests avoid using arguments as return values from keyword. This will improve readibility
Bad example - avoid:
| | ${dut1_gre_interface} | ${dut1_gre_index}= | | ... | When GRE tunnel interface is created and up | | ... | ${dut_node} | ${dut1_ip_address} | ${dut2_ip_address}
Good example - use instead:
| | When GRE tunnel interface is created and up | | ... | ${dut_node} | ${dut1_ip_address} | ${dut2_ip_address} // and the variables ${dut1_gre_interface} and ${dut1_gre_index} set as the test variables using the KW "Set Test Variable": | When GRE tunnel interface is created and up | | [Documentation] | | <keyword implemetation> | | Set Test Variable | ${dut1_gre_interface} | ${dut1_gre_interface} | | Set Test Variable | ${dut1_gre_index} | ${dut1_gre_index} // [mk] shouldn't this be | When GRE tunnel interface is created and up | | [Documentation] | | <keyword implemetation> | | Set Test Variable | ${dut1_gre_interface} | | Set Test Variable | ${dut1_gre_index}
If there are more than one L2 KW in Given/When/Then part, join them in one L2 KW This will improve readibility
Bad example - avoid:
| | ${dut1_gre_interface} | ${dut1_gre_index}= | | ... | When GRE tunnel interface is created and up | | ... | ${dut1_node} | ${dut1_ip_address} | ${dut2_ip_address} | | ${dut2_gre_interface} | ${dut2_gre_index}= | | ... | And GRE tunnel interface is created and up | | ... | ${dut2_node} | ${dut2_ip_address} | ${dut1_ip_address} | | And IP addresses are set on interfaces | | ... | ${dut1_node} | ${dut1_gre_index} | ${dut1_gre_ip} | ${prefix} | | ... | ${dut2_node} | ${dut2_gre_index} | ${dut2_gre_ip} | ${prefix} | | And Vpp Route Add | ${dut1_node} | ${net2_address} | ${prefix} | | ... | ${dut2_gre_ip} | ${dut1_gre_index}
Good example: use instead:
Define a new configuration keyword:
| Configure GRE tunnel interfaces | | [Documentation] | | [Arguments] | <list of arguments> | | ${dut1_gre_interface} | ${dut1_gre_index}= | | ... | GRE tunnel interface is created and up | | ... | ${dut1_node} | ${dut1_ip_address} | ${dut2_ip_address} | | ${dut2_gre_interface} | ${dut2_gre_index}= | | ... | GRE tunnel interface is created and up | | ... | ${dut2_node} | ${dut2_ip_address} | ${dut1_ip_address} | | IP addresses are set on interfaces | | ... | ${dut1_node} | ${dut1_gre_index} | ${dut1_gre_ip} | ${prefix} | | ... | ${dut2_node} | ${dut2_gre_index} | ${dut2_gre_ip} | ${prefix} | | Vpp Route Add | ${dut1_node} | ${net2_address} | ${prefix} | | ... | ${dut2_gre_ip} | ${dut1_gre_index}
Once new composite keyword is defined it can be used in the tests, making the test much simpler to read:
| | When Configure GRE tunnel interfaces | <list of arguments>
Test Suite Naming Guidelines
The approach is to use tree naming convention and to encode following testing information into the test names:
- packet network port configuration
- physical port type;
- NIC model;
- port-NIC locality;
- packet encapsulations
- VPP packet processing
- packet forwarding mode
- packet processing function(s)
- packet forwarding path
- if present, network functions (processes, containers, VMs) and their topology within the computer;
- main measured variable, type of test;
Proposed convention is to encode physical ports and NIC on the left (underlay), followed by outer most frame header, then other stacked headers up to the header processed by vSwitch-VPP, then VPP forwarding function, then encap on vhost interface, number of vhost interfaces, number of VMs. If chained VMs present, they get added on the right. Test topology is expected to be symmetric, in other words packets enter and leave SUT through ports specified on the left of the test name. Here some examples to illustrate the convention followed by the complete legend, and tables mapping the new test filenames to old ones.
For more detail see CSIT wiki: https://wiki.fd.io/view/CSIT/csit-test-naming.
Patch Review Checklist
CSIT Directory Structure
General recommendations of CSIT directory structure:
- There should be only one directory with tests;
- The first level should be the tested device;
- The second level should be /perf and /func;
- The third level should be feature or area;
- No further levels;
The new structure
Tests
$CSIT/ tests/ vpp/ func/ l2bd/ l2xc/ ip4/ ip6/ ip4_tunnels/ ip6_tunnels/ vm_vhost/ crypto/ interfaces/ telemetry/ honeycomb/ perf/ l2/ ip4/ ip6/ ip4_tunnels/ ip6_tunnels/ vm_vhost/ crypto/ dpdk/ func/ perf/ nsh_sfc/ func/ perf/ tldk/ func/ perf/
Keywords
$CSIT/ resources/ libraries/ bash/ python/ packages/ (dirs): feature | area modules/ robot/ shared/ l2/ ip/ overlay/ vm/ crypto/ dpdk/ nsh_sfc/ tldk/ honeycomb/ performance/ telemetry/ features/ fds/
Other resources
$CSIT/ resources/ templates/ vat/ honeycomb/ test_data/ honeycomb/ lisp/ softwire/ tools/ disk_image_builder/ doc_gen/ report_gen/ scripts/ testbed_setup/ topology/ trex/ vagrant/ virl/ topology_schemas/ traffic_scripts/ traffic_profiles/ trex/ ixia/
Tests for VPP
The chapters below reflect the status of functional and performance tests for VPP on 29th of June 2017.
Performance
L2 Ethernet Switching
l2/ 10ge2p1vic1227-eth-l2bdbasemaclrn-ndrpdrdisc.robot 10ge2p1x520-dot1ad-l2xcbase-ndrchk.robot 10ge2p1x520-dot1ad-l2xcbase-ndrpdrdisc.robot 10ge2p1x520-dot1q-l2xcbase-ndrchk.robot 10ge2p1x520-dot1q-l2xcbase-ndrpdrdisc.robot 10ge2p1x520-eth-l2bdbasemaclrn-ndrchk.robot 10ge2p1x520-eth-l2bdbasemaclrn-ndrpdrdisc.robot 10ge2p1x520-eth-l2bdbasemaclrn-pdrchk.robot 10ge2p1x520-eth-l2xcbase-ndrchk.robot 10ge2p1x520-eth-l2xcbase-ndrpdrdisc.robot 10ge2p1x520-eth-l2xcbase-pdrchk.robot 10ge2p1x710-eth-l2bdbasemaclrn-ndrpdrdisc.robot 40ge2p1vic1385-eth-l2bdbasemaclrn-ndrpdrdisc.robot 40ge2p1xl710-eth-l2bdbasemaclrn-ndrpdrdisc.robot 40ge2p1xl710-eth-l2xcbase-ndrpdrdisc.robot
IPv4 Routed-Forwarding
ip4/ 10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrchk.robot 10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrchk.robot 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4base-ipolicemarkbase-ndrchk.robot 10ge2p1x520-ethip4-ip4base-ipolicemarkbase-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4base-ndrchk.robot 10ge2p1x520-ethip4-ip4base-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4base-pdrchk.robot 10ge2p1x520-ethip4-ip4base-snat-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4scale200k-ndrchk.robot 10ge2p1x520-ethip4-ip4scale200k-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4scale20k-ndrchk.robot 10ge2p1x520-ethip4-ip4scale20k-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4scale2m-ndrchk.robot 10ge2p1x520-ethip4-ip4scale2m-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4scale2m-pdrchk.robot 10ge2p1x520-ethip4udp-ip4base-udpsrcscale15-snat-ndrpdrdisc.robot 10ge2p1x520-ethip4udp-ip4scale1000-udpsrcscale15-snat-ndrpdrdisc.robot 10ge2p1x520-ethip4udp-ip4scale100-udpsrcscale15-snat-ndrpdrdisc.robot 10ge2p1x520-ethip4udp-ip4scale10-udpsrcscale15-snat-ndrpdrdisc.robot 10ge2p1x520-ethip4udp-ip4scale2000-udpsrcscale15-snat-ndrpdrdisc.robot 10ge2p1x520-ethip4udp-ip4scale4000-udpsrcscale15-snat-ndrpdrdisc.robot 40ge2p1xl710-ethip4-ip4base-ndrpdrdisc.robot
IPv6 Routed-Forwarding
ip6/ 10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrchk.robot 10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrpdrdisc.robot 10ge2p1x520-ethip6-ip6base-iacldstbase-ndrchk.robot 10ge2p1x520-ethip6-ip6base-iacldstbase-ndrpdrdisc.robot 10ge2p1x520-ethip6-ip6base-ndrchk.robot 10ge2p1x520-ethip6-ip6base-ndrpdrdisc.robot 10ge2p1x520-ethip6-ip6base-pdrchk.robot 10ge2p1x520-ethip6-ip6scale200k-ndrchk.robot 10ge2p1x520-ethip6-ip6scale200k-ndrpdrdisc.robot 10ge2p1x520-ethip6-ip6scale20k-ndrchk.robot 10ge2p1x520-ethip6-ip6scale20k-ndrpdrdisc.robot 10ge2p1x520-ethip6-ip6scale2m-ndrchk.robot 10ge2p1x520-ethip6-ip6scale2m-ndrpdrdisc.robot 10ge2p1x520-ethip6-ip6scale2m-pdrchk.robot 40ge2p1xl710-ethip6-ip6base-ndrpdrdisc.robot
IPv4 Overlay Tunnels
ip4_tunnels/ 10ge2p1x520-ethip4lispip4-ip4base-ndrchk.robot 10ge2p1x520-ethip4lispip4-ip4base-ndrpdrdisc.robot 10ge2p1x520-ethip4lispip4-ip4base-pdrchk.robot 10ge2p1x520-ethip4lispip6-ip4base-ndrchk.robot 10ge2p1x520-ethip4lispip6-ip4base-ndrpdrdisc.robot 10ge2p1x520-ethip4lispip6-ip4base-pdrchk.robot 10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrpdrdisc.robot 10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-pdrchk.robot 10ge2p1x520-ethip4vxlan-l2xcbase-ndrchk.robot 10ge2p1x520-ethip4vxlan-l2xcbase-ndrpdrdisc.robot 10ge2p1x520-ethip4vxlan-l2xcbase-pdrchk.robot
IPv6 Overlay Tunnels
ip6_tunnels/ 10ge2p1x520-ethip6lispip4-ip6base-ndrchk.robot 10ge2p1x520-ethip6lispip4-ip6base-ndrpdrdisc.robot 10ge2p1x520-ethip6lispip4-ip6base-pdrchk.robot 10ge2p1x520-ethip6lispip6-ip6base-ndrchk.robot 10ge2p1x520-ethip6lispip6-ip6base-ndrpdrdisc.robot 10ge2p1x520-ethip6lispip6-ip6base-pdrchk.robot
VM vhost Connections
vm_vhost/ 10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot 10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-pdrchk.robot 10ge2p1x520-dot1q-l2xcbase-eth-2vhost-1vm-ndrpdrdisc.robot 10ge2p1x520-dot1q-l2xcbase-eth-2vhost-1vm-pdrchk.robot 10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-ndrpdrdisc.robot 10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-pdrchk.robot 10ge2p1x520-ethip4-ip4base-eth-4vhost-2vm-ndrpdrdisc.robot 10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot 10ge2p1x520-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot 10ge2p1x520-eth-l2bdbasemaclrn-eth-4vhost-2vm-ndrpdrdisc.robot 10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-ndrpdrdisc.robot 10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-pdrchk.robot 10ge2p1x520-eth-l2xcbase-eth-4vhost-2vm-ndrpdrdisc.robot 10ge2p1x710-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot 40ge2p1xl710-ethip4-ip4base-eth-4vhost-2vm-ndrpdrdisc.robot 40ge2p1xl710-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot 40ge2p1xl710-eth-l2bdbasemaclrn-eth-4vhost-2vm-ndrpdrdisc.robot 40ge2p1xl710-eth-l2xcbase-eth-4vhost-2vm-ndrpdrdisc.robot
IPSec Crypto HW: IP4 Routed-Forwarding
crypto/ 40ge2p1xl710-ethip4ipsecbasetnl-ip4base-int-aes-gcm-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecbasetnl-ip4base-int-cbc-sha1-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecbasetnl-ip4base-tnl-aes-gcm-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecbasetnl-ip4base-tnl-cbc-sha1-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-int-aes-gcm-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-int-cbc-sha1-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-tnl-aes-gcm-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-tnl-cbc-sha1-ndrpdrdisc.robot 40ge2p1xl710-ethip4ipsectptlispgpe-ip4base-cbc-sha1-ndrpdrdisc.robot
Functional
L2 Ethernet Switching
l2bd/ eth2p-dot1ad--dot1q-l2bdbasemaclrn-vlantrans21-func.robot eth2p-dot1ad-l2bdbasemaclrn-vlantrans22-func.robot eth2p-dot1q--dot1ad-l2bdbasemaclrn-vlantrans12-func.robot eth2p-dot1q-l2bdbasemaclrn-vlantrans11-func.robot eth2p-eth-l2bdbasemaclrn-func.robot eth2p-eth-l2bdbasemacstc-func.robot eth4p-eth-l2bdbasemaclrn-l2shg-func.robot
l2xc/ eth2p-dot1ad--dot1q-l2xcbase-vlantrans21-func.robot eth2p-dot1ad-l2xcbase-func.robot eth2p-dot1ad-l2xcbase-vlantrans22-func.robot eth2p-dot1q--dot1ad-l2xcbase-vlantrans12-func.robot eth2p-dot1q-l2xcbase-vlantrans11-func.robot eth2p-eth-l2xcbase-func.robot eth2p-eth-l2xcbase-iaclbase-func.robot
IPv4 Routed-Forwarding
ip4/ eth2p-dot1q-ip4base-func.robot eth2p-ethip4-ip4base-copblklistbase-func.robot eth2p-ethip4-ip4base-copwhlistbase-func.robot eth2p-ethip4-ip4base-func.robot eth2p-ethip4-ip4base-iaclbase-func.robot eth2p-ethip4-ip4base-ip4arp-func.robot eth2p-ethip4-ip4base-ip4dhcpclient-func.robot eth2p-ethip4-ip4base-ip4dhcpproxy-func.robot eth2p-ethip4-ip4base-ip4ecmp-func.robot eth2p-ethip4-ip4base-ip4proxyarp-func.robot eth2p-ethip4-ip4base-ipolicemarkbase-func.robot eth2p-ethip4-ip4base-rpf-func.robot eth2p-ethip4-ip4basevrf-func.robot
IPv6 Routed-Forwarding
ip6/ eth2p-ethip6-ip6base-copblklistbase-func.robot eth2p-ethip6-ip6base-copwhlistbase-func.robot eth2p-ethip6-ip6base-func.robot eth2p-ethip6-ip6base-iaclbase-func.robot eth2p-ethip6-ip6base-ip6dhcpproxy-func.robot eth2p-ethip6-ip6base-ip6ecmp-func.robot eth2p-ethip6-ip6base-ip6ra-func.robot eth2p-ethip6-ip6base-ipolicemarkbase-func.robot eth2p-ethip6-ip6basevrf-func.robot
IPv4 Overlay Tunnels
ip4_tunnels/ gre/ eth2p-ethip4gre-ip4base-func.robot lisp/ api-crud-lisp-func.robot eth2p-ethip4lispgpe-ip4base-func.robot eth2p-ethip4lispgpe-ip4basevrf-func.robot eth2p-ethip4lispgpe-ip6base-func.robot eth2p-ethip4lispgpe-ip6basevrf-func.robot eth2p-ethip4lisp-ip4base-func.robot eth2p-ethip4lisp-l2bdbasemaclrn-func.robot softwire/ eth2p-ethip4--ethip6ip4-ip4base--ip6base-swiremapt-func.robot eth2p-ethip4--ethip6ip4-ip4base--ip6base-swirelw46-func.robot eth2p-ethip4--ethip6ip4-ip4base--ip6base-swiremape-func.robot vxlan/ eth2p-dot1qip4vxlan-l2bdbasemaclrn-func.robot eth4p-ethip4vxlan-l2bdbasemaclrn-l2shg-func.robot eth2p-ethip4vxlan-l2bdbasemaclrn-func.robot eth2p-ethip4vxlan-l2xcbase-func.robot
IPv6 Overlay Tunnels
ip6_tunnels/ lisp/ eth2p-ethip6lispgpe-ip6base-func.robot eth2p-ethip6lispgpe-ip6basevrf-func.robot eth2p-ethip6lispgpe-ip4base-func.robot eth2p-ethip6lisp-l2bdbasemaclrn-func.robot vxlan/ eth2p-ethip6vxlan-l2bdbasemaclrn-func.robot eth4p-ethip6vxlan-l2bdbasemaclrn-l2shg-func.robot
VM vhost Connections
vm_vhost/ l2bd/ eth2p-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-func.robot eth2p-dot1q-l2bdbasemaclrn-eth-4vhost-2vm-fds-provider-nets-func.robot eth2p-ethip4-l2bdbase-vhost-client-reconnect-2vm-func.robot eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot eth2p-ethip4vxlan-l2bdbasemaclrn--eth-4vhost-2vm-fds-tenant-nets-func.robot eth2p-ethip6vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot eth2p-eth-l2bdbasemaclrn-eth-2vhost-1vm-func.robot eth2p-eth-l2bdbasemacstc-eth-2vhost-1vm-func.robot l2xc/ eth2p-eth-l2xcbase-eth-2vhost-1vm-func.robot ip4/ eth2p-ethip4-ip4base-eth-2vhost-1vm.robot eth2p-ethip4ipsectptlispgpe-ip4base-eth-2vhost-1vm-func.robot eth2p-ethip4ipsectptlispgpe-ip6base-eth-2vhost-1vm-func.robot eth2p-ethip4lispgpe-ip4base-eth-2vhost-1vm-func.robot eth2p-ethip4lispgpe-ip4basevrf-eth-2vhost-1vm-func.robot eth2p-ethip4lispgpe-ip6base-eth-2vhost-1vm-func.robot ip6/ eth2p-ethip6ipsectptlispgpe-ip4base-eth-2vhost-1vm-func.robot eth2p-ethip6ipsectptlispgpe-ip6base-eth-2vhost-1vm-func.robot eth2p-ethip6lispgpe-ip6base-eth-2vhost-1vm-func.robot eth2p-ethip6lispgpe-ip6basevrf-eth-2vhost-1vm-func.robot
Crypto HW: IP4 Routed-Forwarding
crypto/ eth2p-ethip4ipsectnl-ip4base-func.robot eth2p-ethip4ipsectpt-ip4base-func.robot eth2p-ethip4ipsectptlispgpe-ip4base-func.robot eth2p-ethip4ipsectptlispgpe-ip6base-func.robot eth2p-ethip4ipsectptlispgpe-ip6basevrf-func.robot eth2p-ethip6ipsectnl-ip6base-func.robot eth2p-ethip6ipsectpt-ip6base-func.robot eth2p-ethip6ipsectptlispgpe-ip4base-func.robot eth2p-ethip6ipsectptlispgpe-ip6base-func.robot
Honecomb
honeycomb/ mgmt-cfg-l2fib-apihc-apivat-func.robot mgmt-cfg-slaac-apihc-func.robot mgmt-cfg-l2bd-apihc-apivat-func.robot mgmt-cfg-lisp-apihc-apivat-func.robot mgmt-cfg-intip4-intip6-apihc-apivat-func.robot mgmt-cfg-nsh-apihc-apivat-func.robot mgmt-cfg-proxyarp-apihc-func.robot mgmt-cfg-int-subint-apihc-apivat-func.robot mgmt-cfg-snat44-apihc-apivat-func.robot mgmt-cfg-vxlangpe-apihc-apivat-func.robot mgmt-cfg-pluginacl-apihc-apivat-func.robot mgmt-cfg-dhcp-apihc-apivat-func.robot mgmt-cfg-inttap-apihc-apivat-func.robot mgmt-cfg-routing-apihc-apivat-func.robot mgmt-cfg-spanrx-apihc-apivat-func.robot mgmt-cfg-vxlan-apihc-apivat-func.robot mgmt-cfg-policer-apihc-func.robot mgmt-cfg-intvhost-apihc-apivat-func.robot mgmt-notif-apihcnc-func.robot mgmt-cfg-proxynd6-apihc-func.robot mgmt-cfg-pbb-apihc-apivat-func.robot mgmt-statepersist-apihc-func.robot mgmt-cfg-int-apihcnc-func.robot mgmt-cfg-acl-apihc-apivat-func.robot
Telemetry
telemetry/ eth2p-ethip4-ip4base-ip4ipfixbase-func.robot eth2p-ethip4-ip4base-ip4ipfixscale-func.robot eth2p-ethip4-ip4base-spanrx-func.robot eth2p-ethip6-ip6base-ip6ipfixbase-func.robot eth2p-ethip6-ip6base-ip6ipfixscale-func.robot eth2p-ethip6-ip6base-spanrx-func.robot
Interface
interfaces/ api-crud-tap-func.robot eth2p-ethip4-ip4base-eth-1tap-func.robot eth2p-eth-l2bdbasemaclrn-eth-2tap-func.robot eth2p-eth-l2bdbasemaclrn-l2shg-eth-2tap-func.robot
Keywords
L2 Robot keywords
shared/ counters.robot default.robot interfaces.robot traffic.robot testing_path.robot lxc.robot l2/ bridge_domain.robot l2_traffic.robot l2_xconnect.robot tagging.robot ip/ ipv4.robot ipv6.robot snat.robot map.robot overlay/ gre.robot lisp_static_adjacency.robot lispgpe.robot l2lisp.robot lisp_api.robot vxlan.robot vm/ double_qemu_setup.robot qemu.robot crypto/ ipsec.robot dpdk/ default.robot nsh_sfc/ default.robot tldk/ TLDKUtils.robot honeycomb/ policer.robot nat.robot port_mirroring.robot vhost_user.robot netconf.robot nsh.robot vxlan.robot dhcp.robot slaac.robot notifications.robot routing.robot access_control_lists.robot tap.robot interfaces.robot honeycomb.robot persistence.robot sub_interface.robot provider_backbone_bridge.robot vxlan_gpe.robot proxyarp.robot bridge_domain.robot lisp.robot l2_fib.robot performance/ performance_configuration.robot performance_utils.robot performance_setup.robot telemetry/ span.robot ipfix.robot features/ policer.robot dhcp_client.robot dhcp_proxy.robot fds/ default.robot
Applying CSIT Design Guidelines
General Notes
- Strictly follow the CSIT Design Hierarchy described in this document;
- Tests MUST use only L2 Keywords (KWs);
- All L2 KWs are composed of L1 KWs (either RF or Py) and/or other L2 KWs;
- Python L1 KWs are preferred to RF L1 KWs;
- RF L1 KWs should be reduced to minimum;
- Test and KW naming should be following CSIT naming guidelines;
Suite
Follow this Suite structure:
- Suite Setup;
- Tests;
- Suite Teardown;
Detailed breakdown:
- Suite Setup
- MAY BE one KW; [mk] s/MAY BE/SHOULD BE/ ?
- Configures the SUT for all tests in the suite, this configuration is the start point for all tests/suites in the suite;
- Use Configuration and/or Verification and/or State Report KWs;
- Tests
- Optional. There are suites that include only other suites;
- See the details below;
- Suite Teardown
- The Teardown MUST perform the environment clean up. The state of SUT SHOULD be the same as at the beginning of the suite;
- Use Configuration and/or Verification and/or State Report KWs;
Tests
There is a clear preference in CSIT project to use data driven tests structure due to simpler code and ease of maintenance. If not applicable, use keyword driven tests approach. Good examples of good candidates for data driven tests are all CSIT performance tests.
Examples of good data driven tests:
- 10ge2p1vic1227-eth-l2bdbasemaclrn-ndrpdrdisc.robot
- eth2p-ethip4--ethip6ip4-ip4base--ip6base-swiremape-func.robot
Examples of tests where data driven test should have been used instead of keyword driven:
- 10ge2p1x520-eth-l2xcbase-ndrpdrdisc.robot
Follow this Test structure:
- Tests Setup;
- Configuration;
- Verification of the configuration;
- Tests;
- Verification of test;
- Test Teardown;
Detailed breakdown:
- Test Setup
- SHOULD be one KW common for all tests in the suite;
- Use Configuration and/or StateReport KWs;
- Configuration
- Optional;
- Test specific configuration that is not possible to move to the Test Setup;
- Use one KW if possible;
- Use Configuration and/or StateReport KWs;
- Verification of the configuration
- Optional;
- Verify if the configuration was sucessful;
- Use one KW if possible;
- Use Verification and/or StateReport KWs;
- Test
- Use one KW if possible;
- Use Test KWs;
- Verification of test
- Use one KW if possible;
- Use Verification and/or StateReport KWs;
- The result of the test is PASS / FAIL; or value(s) in the discovery tests (e.g. performance);
- Test Teardown
- The Teardown MUST perform the environment clean up. The state of SUT SHOULD be the same as at the beginning of the test;
- SHOULD be one KW common for all tests in the suite;
- Use Configuration and/or Verification and/or StateReport KWs;