CSIT/Design Optimizations

From fd.io
< CSIT
Revision as of 11:53, 29 June 2017 by Tifrank (Talk | contribs)

Jump to: navigation, search

WARNING - DRAFT UNDER REVIEW - WORK IN PROGRESS

WARNING - DRAFT UNDER REVIEW - WORK IN PROGRESS

Motivation

FD.io CSIT system design needs to meet continuously expanding requirements of FD.io projects including VPP, related sub-systems (e.g. plugin applications, DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as growing number of compute platforms running those applications. With CSIT project scope and charter including both FD.io continuous testing AND performance trending/comparisons, those evolving requirements further amplify the need for CSIT framework modularity, flexibility and usability.

Following analysis of the existing CSIT design and related inventory of CSIT Robot Framework KeyWords and libraries, CSIT project team decided to make a number of CSIT design optimizations to address those evolving requirements.

This note describes the proposed CSIT design optimizations, and positions them in the context of the updated CSIT framework design. It also includes the updated Robot Framework (RF) Keyword (KW) Naming Style Guidelines to ensure readability and usability of CSIT test framework, and to address existing CSIT KW naming inconsistencies.

Proposed CSIT Design Optimizations

Proposed CSIT design optimizations focus on updated CSIT design guidelines for Robot Framework Test and Keywords usage and can be summarized as follows:

  1. Strictly follow the CSIT Design Hierarchy described in this document;
  2. Tests MUST use only L2 RF Keywords (KWs);
  3. All L2 KWs are composed of L1 KWs (either RF or Python) and/or other L2 KWs;
  4. Python L1 KWs are preferred to RF L1 KWs;
  5. RF L1 KWs should be reduced to minimum;
  6. Test and KW naming should be following CSIT naming guidelines;

For more detailed description of how these guidelines apply to developing new CSIT Tests and Keywords see section "Applying CSIT Design Guidelines".

Proposed CSIT Robot Framework Keywords Optimisations

  1. RF KW renaming
    • see naming guidelines in this document;
  2. EF KW re-classification
    • breakdown libraries into multiple files
      • mainly performance.robot, as everything is in one file;
    • group KWs per infra and functional area similar to structure of csit rls reports;
  3. RF KW move levels
    • few L2 KWs should be L1 KWs, as never used in tests, and dealing with lower level abstractions;
  4. EF KW refactoring;
    • merge KWs that are always used together;
    • KWs that are used 1-2 times should be merged to parent KW, unless further re-use expected;
    • for TestSetup and TestTeardown - use one L2 KW only;
      • SuiteSetup and SuiteTeardown already follow this;
  5. unused KW removal
    • remove unused and unwanted RF KWs;
    • apart from the ones prepared for future use of course;
  6. update CSIT framework documentation

CSIT System Design Hierarchy

CSIT follows a hierarchical system design with SUTs and DUTs at the bottom level, and presentation level at the top level, with a number of functional layers in-between. The current CSIT design including CSIT framework is depicted in the diagram below.

CSIT System Design Hierarchy

A brief bottom-up description is provided here:

  1. SUTs, DUTs, TGs:
    1. SUTs - Systems Under Test;
    2. DUTs - Devices Under Test;
    3. TGs - Traffic Generators;
  2. Level-1 libraries - Robot and Python:
    1. lowest level CSIT libraries abstracting underlying test environment, SUT, DUT and TG specifics;
    2. used commonly across multiple L2 KWs;
    3. performance and functional tests:
      • L1 KWs (KeyWords) are implemented as RF libraries and Python libraries;
      • proposal is to stop implementing L1 functions with RF KWs and do them in Python only instead
        • honeycomb - good example, L1 all in .py, L2 all in RF KWs;
        • over time convert perf and func L1 RF KWs to Python;
    4. performance TG L1 KWs:
      • all L1 KWs are implemented as Python libraries;
      • support for TRex only today;
      • need to add IXIA;
    5. performance data plane traffic profiles:
      • TG-specific stream profiles provide full control of:
        • packet definition – layers, MACs, IPs, ports, combinations thereof e.g. IPs and UDP ports;
        • stream definitions - different streams can run together, delayed, one after each other;
        • stream profiles are independent of CSIT framework and can be used in any T-rex setup, can be sent anywhere to repeat tests with exactly the same setup;
        • easily extensible – one can create a new stream profile that meets tests requirements;
        • same stream profile can be used for different tests with the same traffic needs;
    6. functional data plane traffic scripts:
      • Scapy specific traffic scripts;
  3. Level-2 libraries - Robot resource files:
    1. higher level CSIT libraries abstracting required functions for executing tests;
    2. L2 KWs are classified into the following functional categories:
      • configuration, test, verification, state report;
      • suite setup, suite teardown;
      • test setup, test teardown;
  4. Tests - Robot:
    1. Test suites with test cases;
    2. Functional tests using VIRL environment:
      • VPP;
      • HoneyComb;
    3. Performance tests using physical testbed environment:
      • VPP;
      • Testpmd;
  5. Tools:
    1. documentation generator;
    2. report generator;
    3. testbed environment setup ansible playbooks;
    4. operational debugging scripts;

Test Lifecycle Abstraction

A well coded test must follow a disciplined abstraction of the test lifecycles that includes setup, configuration, test and verification. In addition to improve test execution efficiency, the commmon aspects of test setup and configuration shared across multiple test cases should be done only once. Translating these high-level guidelines into the Robot Framework one arrives to definition of a well coded RF tests for FD.io CSIT.

Anatomy of Good Tests for CSIT:

  1. Suite Setup - Suite startup Configuration common to all Test Cases in suite: uses Configuration KWs, Verification KWs, StateReport KWs;
  2. Test Setup - Test startup Configuration common to multiple Test Cases: uses Configuration KWs, StateReport KWs;
  3. Test Case - uses L2 KWs with RF Gherkin style:
    • prefixed with {Given} - Verification of Test setup, reading state: uses Configuration KWs, Verification KWs, StateReport KWs;
    • prefixed with {When} - Test execution: Configuration KWs, Test KWs;
    • prefixed with {Then} - Verification of Test execution, reading state: uses Verification KWs, StateReport KWs;
  4. Test Teardown - post Test teardown with Configuration cleanup and Verification common to multiple Test Cases - uses: Configuration KWs, Verification KWs, StateReport KWs;
  5. Suite Teardown - Suite post-test Configuration cleanup: uses Configuration KWs, Verification KWs, StateReport KWs;

RF Keywords Functional Classification

CSIT RF KWs are classified into the functional categories matching the test lifecycle events described earlier. All CSIT RF L2 and L1 KWs have been grouped into the following functional categories:

  1. Configuration;
  2. Test;
  3. Verification;
  4. StateReport;
  5. SuiteSetup;
  6. TestSetup;
  7. SuiteTeardown;
  8. TestTeardown;

All existing CSIT L1 and L2 RF KWs have been classified into one of above categories in the spreadsheet that is accompanying this document [mk][add reference to link/xls filename]. Going forward all coded RF Keywords will be tagged using functional categories listed above. To improve readability and usability of CSIT RF KWs, a simple naming convention is proposed for each of the KW categories identified above. Hence while coding RF KWs it is important to keep in mind the category of the coded KW.

RF Keywords Naming Guidelines

Readability counts: "..code is read much more often than it is written." Hence following a good and consistent grammar practice is important when writing RF KeyWords and Tests.

All CSIT test cases are coded using Gherkin style and include only L2 KWs references. L2 KWs are coded using simple style and include L2 KWs, L1 KWs, and L1 python references. To improve readability, the proposal is to use the same grammar for both RF KW styles, and to formalize the grammar of English sentences used for naming the RF KWs.

RF KWs names are short sentences expressing functional description of the command. They must follow English sentence grammar in one of the following forms:

  1. Imperative - verb-object(s): "Do something", verb in base form.
  2. Declarative - subject–verb–object(s): "Subject does something", verb in a third-person singular present tense form.
  3. Affirmative - modal_verb-verb-object(s): "Subject should be something", "Object should exist", verb in base form.
  4. Negative - modal_verb-Not-verb-object(s): "Subject should not be something", "Object should not exist", verb in base form.

Passsive form MUST NOT be used. However a usage of past participle as an adjective is okay. See usage examples.

Following sections list applicability of the above grammar forms to different RF KW categories. Usage examples are provided, both good and bad.

Configuration KWs

Test KWs

Verification KWs

StateReport KWs

SuiteSetup KWs

SuiteTeardown KWs

TestSetup KWs

TestTeardown KWs

Usage of Variables in Tests and KW Name

Test Suite Naming Guidelines

Patch Review Checklist

CSIT Directory Structure

General recommendations of CSIT directory structure:

  • There should be only one directory with tests;
  • The first level should be the tested device;
  • The second level should be /perf and /func;
  • The third level should be feature or area;
  • No further levels;

The new structure

Tests

   $CSIT/
       tests/
           vpp/
               func/
                   l2bd/
                   l2xc/
                   ip4/
                   ip6/
                   ip4_tunnels/
                   ip6_tunnels/
                   vm_vhost/
                   crypto/
                   interfaces/
                   telemetry/
                   honeycomb/
               perf/
                   l2/
                   ip4/
                   ip6/
                   ip4_tunnels/
                   ip6_tunnels/
                   vm_vhost/
                   crypto/
           dpdk/
               func/
               perf/
           nsh_sfc/
               func/
               perf/
           tldk/
               func/
               perf/

Keywords

   $CSIT/
       resources/
           libraries/
               bash/
               python/
                   packages/ (dirs): feature | area
                       modules/
               robot/
                   shared/
                   l2/
                   ip/
                   overlay/
                   vm/
                   crypto/
                   dpdk/
                   nsh_sfc/
                   tldk/
                   honeycomb/
                   performance/
                   telemetry/
                   features/
                   fds/

Other resources

   $CSIT/
       resources/
           templates/
               vat/
               honeycomb/
           test_data/
               honeycomb/
               lisp/
               softwire/
           tools/
               disk_image_builder/
               doc_gen/
               report_gen/
               scripts/
               testbed_setup/
               topology/
               trex/
               vagrant/
               virl/
           topology_schemas/
           traffic_scripts/
           traffic_profiles/
               trex/
               ixia/

Tests for VPP

The chapters below reflect the status of functional and performance tests for VPP on 29th of June 2017.

Performance

L2 Ethernet Switching

   l2/
       10ge2p1vic1227-eth-l2bdbasemaclrn-ndrpdrdisc.robot
       10ge2p1x520-dot1ad-l2xcbase-ndrchk.robot
       10ge2p1x520-dot1ad-l2xcbase-ndrpdrdisc.robot
       10ge2p1x520-dot1q-l2xcbase-ndrchk.robot
       10ge2p1x520-dot1q-l2xcbase-ndrpdrdisc.robot
       10ge2p1x520-eth-l2bdbasemaclrn-ndrchk.robot
       10ge2p1x520-eth-l2bdbasemaclrn-ndrpdrdisc.robot
       10ge2p1x520-eth-l2bdbasemaclrn-pdrchk.robot
       10ge2p1x520-eth-l2xcbase-ndrchk.robot
       10ge2p1x520-eth-l2xcbase-ndrpdrdisc.robot
       10ge2p1x520-eth-l2xcbase-pdrchk.robot
       10ge2p1x710-eth-l2bdbasemaclrn-ndrpdrdisc.robot
       40ge2p1vic1385-eth-l2bdbasemaclrn-ndrpdrdisc.robot
       40ge2p1xl710-eth-l2bdbasemaclrn-ndrpdrdisc.robot
       40ge2p1xl710-eth-l2xcbase-ndrpdrdisc.robot

IPv4 Routed-Forwarding

   ip4/
       10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrchk.robot
       10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4base-iacldstbase-ndrchk.robot
       10ge2p1x520-ethip4-ip4base-iacldstbase-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4base-ipolicemarkbase-ndrchk.robot
       10ge2p1x520-ethip4-ip4base-ipolicemarkbase-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4base-ndrchk.robot
       10ge2p1x520-ethip4-ip4base-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4base-pdrchk.robot
       10ge2p1x520-ethip4-ip4base-snat-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4scale200k-ndrchk.robot
       10ge2p1x520-ethip4-ip4scale200k-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4scale20k-ndrchk.robot
       10ge2p1x520-ethip4-ip4scale20k-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4scale2m-ndrchk.robot
       10ge2p1x520-ethip4-ip4scale2m-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4scale2m-pdrchk.robot
       10ge2p1x520-ethip4udp-ip4base-udpsrcscale15-snat-ndrpdrdisc.robot
       10ge2p1x520-ethip4udp-ip4scale1000-udpsrcscale15-snat-ndrpdrdisc.robot
       10ge2p1x520-ethip4udp-ip4scale100-udpsrcscale15-snat-ndrpdrdisc.robot
       10ge2p1x520-ethip4udp-ip4scale10-udpsrcscale15-snat-ndrpdrdisc.robot
       10ge2p1x520-ethip4udp-ip4scale2000-udpsrcscale15-snat-ndrpdrdisc.robot
       10ge2p1x520-ethip4udp-ip4scale4000-udpsrcscale15-snat-ndrpdrdisc.robot
       40ge2p1xl710-ethip4-ip4base-ndrpdrdisc.robot

IPv6 Routed-Forwarding

   ip6/
       10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrchk.robot
       10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrpdrdisc.robot
       10ge2p1x520-ethip6-ip6base-iacldstbase-ndrchk.robot
       10ge2p1x520-ethip6-ip6base-iacldstbase-ndrpdrdisc.robot
       10ge2p1x520-ethip6-ip6base-ndrchk.robot
       10ge2p1x520-ethip6-ip6base-ndrpdrdisc.robot
       10ge2p1x520-ethip6-ip6base-pdrchk.robot
       10ge2p1x520-ethip6-ip6scale200k-ndrchk.robot
       10ge2p1x520-ethip6-ip6scale200k-ndrpdrdisc.robot
       10ge2p1x520-ethip6-ip6scale20k-ndrchk.robot
       10ge2p1x520-ethip6-ip6scale20k-ndrpdrdisc.robot
       10ge2p1x520-ethip6-ip6scale2m-ndrchk.robot
       10ge2p1x520-ethip6-ip6scale2m-ndrpdrdisc.robot
       10ge2p1x520-ethip6-ip6scale2m-pdrchk.robot
       40ge2p1xl710-ethip6-ip6base-ndrpdrdisc.robot

IPv4 Overlay Tunnels

   ip4_tunnels/
       10ge2p1x520-ethip4lispip4-ip4base-ndrchk.robot
       10ge2p1x520-ethip4lispip4-ip4base-ndrpdrdisc.robot
       10ge2p1x520-ethip4lispip4-ip4base-pdrchk.robot
       10ge2p1x520-ethip4lispip6-ip4base-ndrchk.robot
       10ge2p1x520-ethip4lispip6-ip4base-ndrpdrdisc.robot
       10ge2p1x520-ethip4lispip6-ip4base-pdrchk.robot
       10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrpdrdisc.robot
       10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-pdrchk.robot
       10ge2p1x520-ethip4vxlan-l2xcbase-ndrchk.robot
       10ge2p1x520-ethip4vxlan-l2xcbase-ndrpdrdisc.robot
       10ge2p1x520-ethip4vxlan-l2xcbase-pdrchk.robot

IPv6 Overlay Tunnels

   ip6_tunnels/
       10ge2p1x520-ethip6lispip4-ip6base-ndrchk.robot
       10ge2p1x520-ethip6lispip4-ip6base-ndrpdrdisc.robot
       10ge2p1x520-ethip6lispip4-ip6base-pdrchk.robot
       10ge2p1x520-ethip6lispip6-ip6base-ndrchk.robot
       10ge2p1x520-ethip6lispip6-ip6base-ndrpdrdisc.robot
       10ge2p1x520-ethip6lispip6-ip6base-pdrchk.robot

VM vhost Connections

   vm_vhost/
       10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot
       10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-pdrchk.robot
       10ge2p1x520-dot1q-l2xcbase-eth-2vhost-1vm-ndrpdrdisc.robot
       10ge2p1x520-dot1q-l2xcbase-eth-2vhost-1vm-pdrchk.robot
       10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-ndrpdrdisc.robot
       10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-pdrchk.robot
       10ge2p1x520-ethip4-ip4base-eth-4vhost-2vm-ndrpdrdisc.robot
       10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot
       10ge2p1x520-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot
       10ge2p1x520-eth-l2bdbasemaclrn-eth-4vhost-2vm-ndrpdrdisc.robot
       10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-ndrpdrdisc.robot
       10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-pdrchk.robot
       10ge2p1x520-eth-l2xcbase-eth-4vhost-2vm-ndrpdrdisc.robot
       10ge2p1x710-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot
       40ge2p1xl710-ethip4-ip4base-eth-4vhost-2vm-ndrpdrdisc.robot
       40ge2p1xl710-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrpdrdisc.robot
       40ge2p1xl710-eth-l2bdbasemaclrn-eth-4vhost-2vm-ndrpdrdisc.robot
       40ge2p1xl710-eth-l2xcbase-eth-4vhost-2vm-ndrpdrdisc.robot

IPSec Crypto HW: IP4 Routed-Forwarding

   crypto/
       40ge2p1xl710-ethip4ipsecbasetnl-ip4base-int-aes-gcm-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecbasetnl-ip4base-int-cbc-sha1-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecbasetnl-ip4base-tnl-aes-gcm-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecbasetnl-ip4base-tnl-cbc-sha1-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-int-aes-gcm-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-int-cbc-sha1-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-tnl-aes-gcm-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsecscale1000tnl-ip4base-tnl-cbc-sha1-ndrpdrdisc.robot
       40ge2p1xl710-ethip4ipsectptlispgpe-ip4base-cbc-sha1-ndrpdrdisc.robot

Functional

L2 Ethernet Switching

   l2bd/
       eth2p-dot1ad--dot1q-l2bdbasemaclrn-vlantrans21-func.robot
       eth2p-dot1ad-l2bdbasemaclrn-vlantrans22-func.robot
       eth2p-dot1q--dot1ad-l2bdbasemaclrn-vlantrans12-func.robot
       eth2p-dot1q-l2bdbasemaclrn-vlantrans11-func.robot
       eth2p-eth-l2bdbasemaclrn-func.robot
       eth2p-eth-l2bdbasemacstc-func.robot
       eth4p-eth-l2bdbasemaclrn-l2shg-func.robot

   l2xc/
       eth2p-dot1ad--dot1q-l2xcbase-vlantrans21-func.robot
       eth2p-dot1ad-l2xcbase-func.robot
       eth2p-dot1ad-l2xcbase-vlantrans22-func.robot
       eth2p-dot1q--dot1ad-l2xcbase-vlantrans12-func.robot
       eth2p-dot1q-l2xcbase-vlantrans11-func.robot
       eth2p-eth-l2xcbase-func.robot
       eth2p-eth-l2xcbase-iaclbase-func.robot

IPv4 Routed-Forwarding

   ip4/
       eth2p-dot1q-ip4base-func.robot
       eth2p-ethip4-ip4base-copblklistbase-func.robot
       eth2p-ethip4-ip4base-copwhlistbase-func.robot
       eth2p-ethip4-ip4base-func.robot
       eth2p-ethip4-ip4base-iaclbase-func.robot
       eth2p-ethip4-ip4base-ip4arp-func.robot
       eth2p-ethip4-ip4base-ip4dhcpclient-func.robot
       eth2p-ethip4-ip4base-ip4dhcpproxy-func.robot
       eth2p-ethip4-ip4base-ip4ecmp-func.robot
       eth2p-ethip4-ip4base-ip4proxyarp-func.robot
       eth2p-ethip4-ip4base-ipolicemarkbase-func.robot
       eth2p-ethip4-ip4base-rpf-func.robot
       eth2p-ethip4-ip4basevrf-func.robot

IPv6 Routed-Forwarding

   ip6/
       eth2p-ethip6-ip6base-copblklistbase-func.robot
       eth2p-ethip6-ip6base-copwhlistbase-func.robot
       eth2p-ethip6-ip6base-func.robot
       eth2p-ethip6-ip6base-iaclbase-func.robot
       eth2p-ethip6-ip6base-ip6dhcpproxy-func.robot
       eth2p-ethip6-ip6base-ip6ecmp-func.robot
       eth2p-ethip6-ip6base-ip6ra-func.robot
       eth2p-ethip6-ip6base-ipolicemarkbase-func.robot
       eth2p-ethip6-ip6basevrf-func.robot

IPv4 Overlay Tunnels

   ip4_tunnels/
       gre/
           eth2p-ethip4gre-ip4base-func.robot
       lisp/
           api-crud-lisp-func.robot
           eth2p-ethip4lispgpe-ip4base-func.robot
           eth2p-ethip4lispgpe-ip4basevrf-func.robot
           eth2p-ethip4lispgpe-ip6base-func.robot
           eth2p-ethip4lispgpe-ip6basevrf-func.robot
           eth2p-ethip4lisp-ip4base-func.robot
           eth2p-ethip4lisp-l2bdbasemaclrn-func.robot
       softwire/
           eth2p-ethip4--ethip6ip4-ip4base--ip6base-swiremapt-func.robot
           eth2p-ethip4--ethip6ip4-ip4base--ip6base-swirelw46-func.robot
           eth2p-ethip4--ethip6ip4-ip4base--ip6base-swiremape-func.robot
       vxlan/
           eth2p-dot1qip4vxlan-l2bdbasemaclrn-func.robot
           eth4p-ethip4vxlan-l2bdbasemaclrn-l2shg-func.robot
           eth2p-ethip4vxlan-l2bdbasemaclrn-func.robot
           eth2p-ethip4vxlan-l2xcbase-func.robot

IPv6 Overlay Tunnels

   ip6_tunnels/
       lisp/
           eth2p-ethip6lispgpe-ip6base-func.robot
           eth2p-ethip6lispgpe-ip6basevrf-func.robot
           eth2p-ethip6lispgpe-ip4base-func.robot
           eth2p-ethip6lisp-l2bdbasemaclrn-func.robot
       vxlan/
           eth2p-ethip6vxlan-l2bdbasemaclrn-func.robot
           eth4p-ethip6vxlan-l2bdbasemaclrn-l2shg-func.robot

VM vhost Connections

   vm_vhost/
       l2bd/
           eth2p-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-func.robot
           eth2p-dot1q-l2bdbasemaclrn-eth-4vhost-2vm-fds-provider-nets-func.robot
           eth2p-ethip4-l2bdbase-vhost-client-reconnect-2vm-func.robot
           eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot
           eth2p-ethip4vxlan-l2bdbasemaclrn--eth-4vhost-2vm-fds-tenant-nets-func.robot
           eth2p-ethip6vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot
           eth2p-eth-l2bdbasemaclrn-eth-2vhost-1vm-func.robot
           eth2p-eth-l2bdbasemacstc-eth-2vhost-1vm-func.robot
       l2xc/
           eth2p-eth-l2xcbase-eth-2vhost-1vm-func.robot
       ip4/
           eth2p-ethip4-ip4base-eth-2vhost-1vm.robot
           eth2p-ethip4ipsectptlispgpe-ip4base-eth-2vhost-1vm-func.robot
           eth2p-ethip4ipsectptlispgpe-ip6base-eth-2vhost-1vm-func.robot
           eth2p-ethip4lispgpe-ip4base-eth-2vhost-1vm-func.robot
           eth2p-ethip4lispgpe-ip4basevrf-eth-2vhost-1vm-func.robot
           eth2p-ethip4lispgpe-ip6base-eth-2vhost-1vm-func.robot
       ip6/
           eth2p-ethip6ipsectptlispgpe-ip4base-eth-2vhost-1vm-func.robot
           eth2p-ethip6ipsectptlispgpe-ip6base-eth-2vhost-1vm-func.robot
           eth2p-ethip6lispgpe-ip6base-eth-2vhost-1vm-func.robot
           eth2p-ethip6lispgpe-ip6basevrf-eth-2vhost-1vm-func.robot

Crypto HW: IP4 Routed-Forwarding

   crypto/
       eth2p-ethip4ipsectnl-ip4base-func.robot
       eth2p-ethip4ipsectpt-ip4base-func.robot
       eth2p-ethip4ipsectptlispgpe-ip4base-func.robot
       eth2p-ethip4ipsectptlispgpe-ip6base-func.robot
       eth2p-ethip4ipsectptlispgpe-ip6basevrf-func.robot
       eth2p-ethip6ipsectnl-ip6base-func.robot
       eth2p-ethip6ipsectpt-ip6base-func.robot
       eth2p-ethip6ipsectptlispgpe-ip4base-func.robot
       eth2p-ethip6ipsectptlispgpe-ip6base-func.robot

Honecomb

   honeycomb/
       mgmt-cfg-l2fib-apihc-apivat-func.robot
       mgmt-cfg-slaac-apihc-func.robot
       mgmt-cfg-l2bd-apihc-apivat-func.robot
       mgmt-cfg-lisp-apihc-apivat-func.robot
       mgmt-cfg-intip4-intip6-apihc-apivat-func.robot
       mgmt-cfg-nsh-apihc-apivat-func.robot
       mgmt-cfg-proxyarp-apihc-func.robot
       mgmt-cfg-int-subint-apihc-apivat-func.robot
       mgmt-cfg-snat44-apihc-apivat-func.robot
       mgmt-cfg-vxlangpe-apihc-apivat-func.robot
       mgmt-cfg-pluginacl-apihc-apivat-func.robot
       mgmt-cfg-dhcp-apihc-apivat-func.robot
       mgmt-cfg-inttap-apihc-apivat-func.robot
       mgmt-cfg-routing-apihc-apivat-func.robot
       mgmt-cfg-spanrx-apihc-apivat-func.robot
       mgmt-cfg-vxlan-apihc-apivat-func.robot
       mgmt-cfg-policer-apihc-func.robot
       mgmt-cfg-intvhost-apihc-apivat-func.robot
       mgmt-notif-apihcnc-func.robot
       mgmt-cfg-proxynd6-apihc-func.robot
       mgmt-cfg-pbb-apihc-apivat-func.robot
       mgmt-statepersist-apihc-func.robot
       mgmt-cfg-int-apihcnc-func.robot
       mgmt-cfg-acl-apihc-apivat-func.robot

Telemetry

   telemetry/
       eth2p-ethip4-ip4base-ip4ipfixbase-func.robot
       eth2p-ethip4-ip4base-ip4ipfixscale-func.robot
       eth2p-ethip4-ip4base-spanrx-func.robot
       eth2p-ethip6-ip6base-ip6ipfixbase-func.robot
       eth2p-ethip6-ip6base-ip6ipfixscale-func.robot
       eth2p-ethip6-ip6base-spanrx-func.robot

Interface

   interfaces/
       api-crud-tap-func.robot
       eth2p-ethip4-ip4base-eth-1tap-func.robot
       eth2p-eth-l2bdbasemaclrn-eth-2tap-func.robot
       eth2p-eth-l2bdbasemaclrn-l2shg-eth-2tap-func.robot

Keywords

L2 Robot keywords

   shared/
       counters.robot
       default.robot
       interfaces.robot
       traffic.robot
       testing_path.robot
       lxc.robot
   l2/
       bridge_domain.robot
       l2_traffic.robot
       l2_xconnect.robot
       tagging.robot
   ip/
       ipv4.robot
       ipv6.robot
       snat.robot
       map.robot
   overlay/
       gre.robot
       lisp_static_adjacency.robot
       lispgpe.robot
       l2lisp.robot
       lisp_api.robot
       vxlan.robot
   vm/
       double_qemu_setup.robot
       qemu.robot
   crypto/
       ipsec.robot
   dpdk/
       default.robot
   nsh_sfc/
       default.robot
   tldk/
       TLDKUtils.robot
   honeycomb/
       policer.robot
       nat.robot
       port_mirroring.robot
       vhost_user.robot
       netconf.robot
       nsh.robot
       vxlan.robot
       dhcp.robot
       slaac.robot
       notifications.robot
       routing.robot
       access_control_lists.robot
       tap.robot
       interfaces.robot
       honeycomb.robot
       persistence.robot
       sub_interface.robot
       provider_backbone_bridge.robot
       vxlan_gpe.robot
       proxyarp.robot
       bridge_domain.robot
       lisp.robot
       l2_fib.robot
   performance/
       performance_configuration.robot
       performance_utils.robot
       performance_setup.robot
   telemetry/
       span.robot
       ipfix.robot
   features/
       policer.robot
       dhcp_client.robot
       dhcp_proxy.robot
   fds/
       default.robot

Applying CSIT Design Guidelines

General Notes

  • Strictly follow the CSIT Design Hierarchy described in this document;
  • Tests MUST use only L2 Keywords (KWs);
  • All L2 KWs are composed of L1 KWs (either RF or Py) and/or other L2 KWs;
  • Python L1 KWs are preferred to RF L1 KWs;
  • RF L1 KWs should be reduced to minimum;
  • Test and KW naming should be following CSIT naming guidelines;

Suite

Follow this Suite structure:

  1. Suite Setup;
  2. Tests;
  3. Suite Teardown;

Detailed breakdown:

  1. Suite Setup
    1. MAY BE one KW; [mk] s/MAY BE/SHOULD BE/ ?
    2. Configures the SUT for all tests in the suite, this configuration is the start point for all tests/suites in the suite;
    3. Use Configuration and/or Verification and/or State Report KWs;
  2. Tests
    1. Optional. There are suites that include only other suites;
    2. See the details below;
  3. Suite Teardown
    1. The Teardown MUST perform the environment clean up. The state of SUT SHOULD be the same as at the beginning of the suite;
    2. Use Configuration and/or Verification and/or State Report KWs;

Tests

There is a clear preference in CSIT project to use data driven tests structure due to simpler code and ease of maintenance. If not applicable, use keyword driven tests approach. Good examples of good candidates for data driven tests are all CSIT performance tests.

Examples of good data driven tests:

  • 10ge2p1vic1227-eth-l2bdbasemaclrn-ndrpdrdisc.robot
  • eth2p-ethip4--ethip6ip4-ip4base--ip6base-swiremape-func.robot

Examples of tests where data driven test should have been used instead of keyword driven:

  • 10ge2p1x520-eth-l2xcbase-ndrpdrdisc.robot

Follow this Test structure:

  1. Tests Setup;
  2. Configuration;
  3. Verification of the configuration;
  4. Tests;
  5. Verification of test;
  6. Test Teardown;

Detailed breakdown:

  1. Test Setup
    1. SHOULD be one KW common for all tests in the suite;
    2. Use Configuration and/or StateReport KWs;
  2. Configuration
    1. Optional;
    2. Test specific configuration that is not possible to move to the Test Setup;
    3. Use one KW if possible;
    4. Use Configuration and/or StateReport KWs;
  3. Verification of the configuration
    1. Optional;
    2. Verify if the configuration was sucessful;
    3. Use one KW if possible;
    4. Use Verification and/or StateReport KWs;
  4. Test
    1. Use one KW if possible;
    2. Use Test KWs;
  5. Verification of test
    1. Use one KW if possible;
    2. Use Verification and/or StateReport KWs;
    3. The result of the test is PASS / FAIL; or value(s) in the discovery tests (e.g. performance);
  6. Test Teardown
    1. The Teardown MUST perform the environment clean up. The state of SUT SHOULD be the same as at the beginning of the test;
    2. SHOULD be one KW common for all tests in the suite;
    3. Use Configuration and/or Verification and/or StateReport KWs;
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/Design_Optimizations&oldid=5436"