CSIT/Documentation

From fd.io
< CSIT
Revision as of 15:59, 5 May 2016 by Skobza (Talk | contribs)

Jump to: navigation, search


WORK IN PROGRESS

CSIT components

FD.io CSIT system is developed using two main coding building blocks: RobotFramework (referred to as RF from now on) and Python.


Robot Framework

RF is test automation tool, which allows to select and run test suites and test cases, on selected target topology (nodes) and provide us with output logs in readable and parse-able form. RF uses column-based formatting for its files. For clarity and to avoid problems, CSIT team has selected the "pipe and space separated" format (rather than tab-separated format). RF is case insensitive, but we strive to be consistent in how we use upper/lower case in RF files. The value of RF really is its readability; anyone who spends a little time reading RF source files can understand what is going on, and what is more important - even non-programmers can understand the essence of what given test case is doing. There are only two types of RF files stored in CSIT: resource (aka library) files, and test suites (any RF file containing test case). In resource files we store all Keywords (RF's name for functions/methods) that are generic, and could be re-used. For example NIC manipulation is going to be re-used in nearly all test-suites, hence it is placed in file for common reuse as a RF library. All tests in RF format are stored in tests/suites/ sub directories. RF interprets every directory here as test suite, as well as all files containing test cases are taken as test suites.

Python

Python is a component that needs no introduction by itself. In CSIT, we use Python (latest release of 2.7.x series) to perform tasks that are unsuitable for RF format (e.g. lower level code, or anything that starts to feels more like coding - conditionals/loops etc.). Since RF is written in Python, it integrates with Python extensions quite nicely and easily. In RF file, one can import .py scripts directly, and reference provided classes and functions from the imported Py module. Here's a link to RF documentation on how to create and import library to RF file.

Other use of Python code in CSIT framework is in traffic scripts. In typical test, one have to verify whether the dataplane of VPP does what the configuration says it should, and in CSIT we do that by sending crafted packets and validating them on receive. There is a Python tool, that allows one to easily create, dissect, manipulate and pretty-print packets named Scapy. Here's an example how a Python traffic script looks like. These scripts are re-used as much as possible, so that duplicated code is minimized. Traffic scripts are being run on the TG (traffic generator) node in topology using SSH command, which passes variables to the script as command line arguments. There's a utility method in CSIT Python module that is used to generate command line with arguments with values based on passed parameters. This is re-used as some of the parameters are common in these scripts, such as MAC/IP addresses to use for traffic, or which interface to use as TX and RX.

All Python code submitted to CSIT should conform to PEP-8 style. In fact, there's a Jenkins job hooked to Gerrit watching CSIT project changes, that runs Pylint against all resources/**/ .py files. No submitted patch shall increase number of Pylint violations (such as [1]), but rather should work on lowering the number of warnings. In some cases it is impossible to do so, so exceptions might apply, but in general one should work their hardest toward having clean Python code.

Starting CSIT / Robot Framework tests

pybot

RF provides Python executable pybot, which is currently the main entry point to CSIT tests. This documentation here will provide only usage examples usable for CSIT execution, not deep-dive into all various options for pybot. Such information is left for self-study (use pybot --help to see all available options).

Log levels

All CSIT Python libraries use RF logging sub-system for logging. To specify logging verbosity level, add -L LEVEL as your command line argument. E.g. pybot -L TRACE ... to log everything. Logginb information is then stored in output.xml and log.html output files. RF supported log levels are from RF user guide:

  • FAIL - Used when a keyword fails. Can be used only by Robot Framework itself.
  • WARN - Used to display warnings. They shown also in the console and in the Test Execution Errors section in log files, but they do not affect the test case status.
  • INFO - The default level for normal messages. By default, messages below this level are not shown in the log file.
  • DEBUG - Used for debugging purposes. Useful, for example, for logging what libraries are doing internally. When a keyword fails, a traceback showing where in the code the failure occurred is logged using this level automatically.
  • TRACE - More detailed debugging level. The keyword arguments and return values are automatically logged using this level.

Test Suites / Test Cases

In CSIT, the design is to use one directory for all tests, conveniently named tests. All tests are grouped by suites - every directory is understood as a suite in RF, and every file (CSIT uses exclusively .robot file suffix) too. This information can be leveraged by starting pybot -s suite_name to execute only test cases from that particular suite (wildcards are supported, just be aware of shell globbing). Here's more detailed explanation.

Selection of a concrete test case is possible too, just use -t name_of_testcase option. You can combine -s and -t parameters.

Tags

To help selection of test cases for per-patch run, CSIT has developed TAGging scheme, which is documented and should be reachable in gerrit.fd.io repo. At minimum, each CSIT test case must provide TAG from two groups:

  • environment tag
    • defines on which environment this test case can be run
    • e.g. VM_ENV, HW_ENV
  • topology tag
    • defines what topology this test requires
    • e.g. 3_NODE_SINGLE_LINK_TOPO

Therefore, to run all test cases that can be run on VM environment (VIRL, VmWare, VirtualBox, KVM, etc), and requires at least one link in between nodes, one can type pybot --include VM_ENVand3_NODE_SINGLE_LINK_TOPO ([2]).

Topology

Further component to tests the execution, probably the most important one, is the topology information for those tests to run on. Every test case uses topology to execute on. Topology is collection of nodes with settings (like IP addresses of nodes, login credentials, and most importantly NIC and link information). Currently there is only one type of topology being used - a three node topology with one, or two links between each pair of nodes. See diagram below. 

        +---------+                      +---------+
        |         <---------------------->         |
        |   DUT   |                      |   DUT   |
        |         <---------------------->         |
        +--^---^--+                      +--^----^-+
           |   |                            |    |
           |   |                            |    |
           |   |         +---------+        |    |
           |   +--------->         <--------+    |
           |             |   TG    |             |
           +------------->         <-------------+
                         +---------+

Each of the nodes above is missing one additional NIC drawing, which is connected to the MGMT network, so that these nodes are reachable during test. Each connection that is drawn on the diagram is potentially going to be used by test cases, and it is essential that there is no other traffic going on these links than what traffic generator scripts send out or what DUT VPPs send out. Often times other protocol packets (e.g. LLDP, DHCP) may be present on these links, and this will cause test case failures.

Topology information is stored in YAML format in topologies/ sub-directories. To help in early syntax error detection, we provide YAML schemas in resources/topology_schemas sub-directory. Contents of your topology file have to represent your physical or virtual lab setup. Don't forget to pay special attention to login information, IP addresses and interfaces details.

Topology information is loaded from YAML file, processed and provided by CSIT Python library as a global variable named nodes during CSIT start. You might have spotted occurrences of it in .robot files as ${nodes['TG']} - this represents reference to dictionary value of parsed node named TG from topology file. This is then used with other/all CSIT Python libraries (e.g. SSH, DUT setup, and so on).

To specify your concrete topology file, pass -v TOPOLOGY_PATH:topologies/enabled/topology.yaml parameter as your pybot command line argument, just replace the topologies/enabled/topology.yaml with path to your own topology definition. Complete example of how to start bridge domain tests with custom topology file in ~/my_topo.yaml looks like this: pybot -L TRACE -v TOPOLOGY_PATH:~/my_topo.yaml --include vm_envAND3_node_single_link_topo -s "bridge domain"


Test execution walk-through

After pybot execution, RF starts looking for test suites to execute. All test suites are stored in tests/ sub-directory. It'll look recursively for files in this directory, and naturally it'll come to tests/suites/__init__.robot. This file is loaded before any other suite - it acts as initialization file for all suites in given directory. CSIT uses this file to initialize test run-time before we run any test case. The setup currently consists of:

  • Setup Framework prepares each node in topology for use by the framework for testing
    • executed only and exactly once, before any test case execution,
    • uploads the whole CSIT directory (CSIT top level dir with contents) to every ${node}:/tmp/openvpp-testing directory for use during test,
    • make sure all dependencies are installed on Nodes if needed, and prepare Python Virtualenv.
    • this step is done parallel for each topology Node to save up time (significantly).
  • Setup All DUTs prepares all DUTs for test execution
    • executed potentially(?) before any test case
    • restarts VPP, makes sure VPP is up, performs common setup before test execution
  • Update All Interface Data On All Nodes
    • downloads VPP interfaces data from each DUT, and stores sw_if_index of interfaces into the topology information for later use (VPP API commands take sw_if_index as parameter instead of NIC names)

After above mentioned procedures, RF/pybot loads test suites and test cases based on selection criteria given in command line. Going forward from this, it is just plain old RF test cases execution as per RF documentation - typically there are some keywords mentioned in suite setup and some in test case setup which are executed before the test case, and from there it's just test case execution and evaluation.

Jenkins

The whole purpose of CSIT is to be continuous. To achieve just that, all possible functional test cases have to be executed every time a patch is introduced to VPP. This is achieved by executing CSIT tests against given all patches in vpp project in gerrit. If any test unexpected test fails, the Gerrit review will receive -Verified vote from Jenkins user, hence rendering the review not submittable (by normal means). This approach helps discovering problems early before VPP code is submitted to VPP master branch. Such jobs (which are triggered by a change in Gerrit review) are in general called verify jobs, and can be spotted by "verify" in their name (e.g. csit-verify, vpp-verify etc.). Analogically there are merge jobs, which are triggered after a review in Gerrit is submitted to the master/parent branch. These jobs are typically used for artefact publishing to nexus and such (e.g. VPP .deb packages).

CSIT has to have its own Jenkins jobs, that help validate incoming tests/code in all ways: code functionality (all test are executed) and code cleanliness.


All CSIT jobs with their description are listed here.

CSIT Jobs

These jobs role is to verify CSIT code is still working. For every review in CSIT gerrit for functional tests, all tests are executed to verify the patch didn't bring in any collateral. Furthermore if there is new test case/suite added, these are executed too. This way we know if the implemented test cases work as they should. We use "golden" VPP version from Nexus, that has been previously validated by CSIT that works fine - this is to eliminate multiple variables in test (i.e. have only CSIT change, leaving VPP version constant between runs).

CSIT requires PEP-8 code style, and to help code reviews pylint job has been created: csit-validate-pylint. This job's task is to detect formatting violations and report them.

VPP Jobs

VPP jobs created by CSIT have the sole purpose of validating VPP review submissions (patchsets if you will). This is achieved by executing verified version of CSIT code on top of build of VPP (built from git parent version + applied patch from given gerrit review). The verified CSIT code version is identified by a git tag (a branch currently, but that's not important), that points to a known-100%-passing-CSIT-version. Therefore if a VPP test fails, it is due to VPP code change instead of potential problem in CSIT.

VPP verification job vpp-csit-verify-virl, builds VPP, checks out csit-verified and uses the built .deb packages to test the VPP. That's how VPP diff gets tested.


Starting RobotFramework

All examples below expects cloned CSIT repository, and created virtual environment, achieved like: 

username@vpp64:~/vpp/fd.io/csit$ virtualenv env
New python executable in env/bin/python
Installing setuptools, pip, wheel...done.
username@vpp64:~/vpp/fd.io/csit$ source env/bin/activate
(env)username@vpp64:~/vpp/fd.io/csit$ pip install -r requirements.txt
Collecting robotframework==2.9.2 (from -r requirements.txt (line 1))
/home/username/vpp/fd.io/csit/env/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Collecting paramiko==1.16.0 (from -r requirements.txt (line 2))
  Using cached paramiko-1.16.0-py2.py3-none-any.whl
Collecting scp==0.10.2 (from -r requirements.txt (line 3))
  Using cached scp-0.10.2-py2.py3-none-any.whl
Collecting ipaddress==1.0.16 (from -r requirements.txt (line 4))
  Using cached ipaddress-1.0.16-py27-none-any.whl
Collecting interruptingcow==0.6 (from -r requirements.txt (line 5))
Collecting PyYAML==3.11 (from -r requirements.txt (line 6))
Collecting pykwalify==1.5.0 (from -r requirements.txt (line 7))
Collecting scapy==2.3.1 (from -r requirements.txt (line 8))
Collecting enum34==1.1.2 (from -r requirements.txt (line 9))
Collecting requests==2.9.1 (from -r requirements.txt (line 10))
  Downloading requests-2.9.1-py2.py3-none-any.whl (501kB)
    100% |████████████████████████████████| 503kB 582kB/s
Collecting ecdsa>=0.11 (from paramiko==1.16.0->-r requirements.txt (line 2))
  Using cached ecdsa-0.13-py2.py3-none-any.whl
Collecting pycrypto!=2.4,>=2.1 (from paramiko==1.16.0->-r requirements.txt (line 2))
Collecting docopt==0.6.2 (from pykwalify==1.5.0->-r requirements.txt (line 7))
Collecting python-dateutil==2.4.2 (from pykwalify==1.5.0->-r requirements.txt (line 7))
  Using cached python_dateutil-2.4.2-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil==2.4.2->pykwalify==1.5.0->-r requirements.txt (line 7))
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: robotframework, ecdsa, pycrypto, paramiko, scp, ipaddress, interruptingcow, PyYAML, docopt, six, python-dateutil, pykwalify, scapy, enum34, requests
Successfully installed PyYAML-3.11 docopt-0.6.2 ecdsa-0.13 enum34-1.1.2 interruptingcow-0.6 ipaddress-1.0.16 paramiko-1.16.0 pycrypto-2.6.1 pykwalify-1.5.0 python-dateutil-2.4.2 requests-2.9.1 robotframework-2.9.2 scapy-2.3.1 scp-0.10.2 six-1.10.0
/home/username/vpp/fd.io/csit/env/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
You are using pip version 7.1.2, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(env)username@vpp64:~/vpp/fd.io/csit$

Start Specific Test Suite

(env)username@vpp64:~/vpp/fd.io/csit$ pybot -L TRACE -v TOPOLOGY_PATH:topologies/available/my_topo.yaml -s ipv4 tests

pybot
RobotFramework executable
-L TRACE
set the debugging level to TRACE
-v TOPOLOGY_PATH
topologies/available/my_topo.yaml
load the topology file topologies/available/my_topo.yaml
-s ipv4
start all test suites that match name ipv4>
tests
path to start tests from. In our case tests is directory containing subdirs with test suites. RF reads these sub-directories in recursive manner in search for test cases.

Start Specific Test Case

Typical problems

vpp-csit-verify job failed, now what?

how to read console logs

reading robotframework log

timeout problems

Writing a test case

This section talks about steps one has to take to add a new fully functional test. A new test will most probably require opening multiple files in different locations in CSIT code base, e.g. Python library, VAT template and RobotFramework suite/test. Usually developer has idea what they want to test - they may come with bunch of scripts they used during unit testing of their feature, or they might just have a list of newly added VPP API functions. This guide will walk one through from nothing to full working test case. For this purpose, bridge domain test case is chosen as an example.

Test case design

First, we have to specify what the given test is proving. In our case, we want to test if VPP's bridge domain (with enabled learning) forwards packets as expected. From VPP's perspective, we want to: - create a bridge domain - add two interfaces to it - set interfaces state to up

Then we will send packet to one interface and expect it will appear unchanged on the other interface. Since the bridge domain is configured as learning, and the dst mac is unknown to VPP the packet has to appear on the other interface.

To create a VPP bridge domain, VPP API has to be used: 

bridge_domain_add_del
sw_interface_set_l2_bridge
sw_interface_set_flags

Lastly, one has to realize they work in a given topology. Most used topology currently is the circular topology with one link between nodes - this one is the easiest one to set up. In this topology each node has two interfaces. TG is connected to both DUTs, and DUTs are interconnected with a cable too. Therefore to test the packet flowing through the VPP's bridge domain, we have to duplicate the bridge domain configuration to both DUTs.

Test suite creation

In this example, we are going to create a new test suite, because we know no bridge domain tests are present (suppose). We have to create a new directory in tests/suites directory. Let's call it bridge_domain. RobotFramework will take the name of the directory as test suite name. We will create then a new file in this subdirectory, which will contain test cases for bridge_domain suite. For now let's call it test.robot (this could be something more explanatory like "untagged" to mark tests that are not using tagged packets). 

$ cd $CSIT_ROOT
$ mkdir tests/suites/bridge_domain
$ touch tests/suites/bridge_domain/test.robot
$ ${EDITOR} tests/suites/bridge_domain/test.robot

Every RobotFramework test suite file will in the end have at least two sections: Settings and Test Cases.

Settings section

This section sets properties for the whole test suite, like imports of libraries, imports of other robot framework files, tags settings, and setup/teardown phase hooks settings. Add these lines to your newly created test.robot file: 

*** Settings ***
| Resource | resources/libraries/robot/default.robot

We've created a new (still empty) test suite, with two sections. In the Settings section we added a directive to make RobotFramework import other RobotFramework file. default.robot is CSIT created file with default imports that are recurring in multiple test suites. In the default.robot file, there is Variables directive, which causes the topology information to be loaded for later use. Therefore in our test cases we will have ${nodes} populated properly - reference to topology nodes information from topology yaml configuration as passed to the pybot command.

We know that we are going to send crafted packets using our traffic scripts, therefore we have to prepare TG node for this. The reason is that functional tests could get intermixed with perf tests on the same topology, and perf tests could take NICs from linux kernel and use special driver for specialized packet generation. To get around this, one should make sure the NICs are visible in linux by using RobotFramework keyword from defaul.robot: 

| Setup all TGs before traffic script                                            
| | [Documentation] | Prepare all TGs before traffic scripts execution           
| | All TGs Set Interface Default Driver | ${nodes}

We have to call this only oncce, because in our suite we want to use traffic scripts only. To make RobotFramework call this KW only once, we can add Suite Setup directive to Settings section: 

| Suite Setup | Setup all TGs before traffic script

Furthermore we want to make sure VPP is running on DUTs, and if not start it up (VPP might have crashed during previous test). To do this before every test case, we can add following line to Settings section: 

| Test Setup | Setup all DUTs before test

Test Cases section

Now we define the test case. Test case starts with its name as first column. One can write whatever they want here, since it is just taken by RobotFramework as plain text and used as such. In CSIT we strive to have self-explanatory test case names. The goal is to have the test name short, with detail and self-explanatory. Or at least attempt to do so. The approach that we've took is to take the names of the testcases as if one would be explaining the feature to others. We claim, that VPP can forward packets through bridge domain, and we are doing it in circular topology. More documentation can be added later in [Documentation] section in test case definition, but just from the name one should be able what the test case does. So here goes an attempt to name the test case: 

| Vpp forwards packets via L2 bridge domain in circular topology

When RobotFramework will be executing these instructions in this test case we writin, the Test Setup phase has been already executed for us. Therefore VPPs on DUTs are in up state, and we can fire off VPP commands we want to test. First thing to realize is that we have to have reference to a DUT we want to execute these commands at. For this purpose utility library NodePath has been created. It allows one to create virtual "packet" (data) path through the topology nodes and their NICs. To use the library (any library really) in this test suite, we have to import it in Settings section: 

| Library | resources.libraries.python.NodePath

We know we plan to go through the topology in circular manner, so our packets are going to go out of TG to DUT1 to DUT2 and back to TG. We already have ${nodes} object available to us in RobotFramework files through the default.robot import. Now we have to select NICs on these ports. To refrain from using concrete port names/numbers in RobotFramework, we can do following: 

| | Append Nodes | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']}          
| | ...          | ${nodes['TG']}                                                
| | Compute Path

Note that we didn't have to write NodePath.Append Nodes. That's because RobotFramework resolves non-conflicting names automatically. One can confirm that in log.html output, where RobotFramework will prepend name of library before the function name.

This will create node traversal path using the NodePath library. 

| | ${tg_if1} | ${tg}= | Next Interface                                          
| | ${dut1_if1} | ${dut1}= | Next Interface                                      
| | ${dut1_if2} | ${dut1}= | Next Interface                                      
| | ${dut2_if1} | ${dut2}= | Next Interface                                      
| | ${dut2_if2} | ${dut2}= | Next Interface                                      
| | ${tg_if2} | ${tg}= | Next Interface

Now we have variables filled with interface data. For example tg_if1 contains interface information about that interface on that given node (TG). We will use these variables later in the test, and pass them to other functions and RobotFramework keywords as arguments.

Now we have to create the bridge domain, add interfaces to it, set them up and do this for both DUTs. So we know we are going to repeat some work. Why not create a RobotFramework function, also known as Keyword, for it? This operation is going to be reused number of times, so let's create a common RobotFramework resource file for it. The common RobotFramework libraries sit in resources/libraries/robot directory. Let's add bridge_domain.robot file there: 

$ ${EDITOR} $CSIT_ROOT/resources/libraries/robot/bridge_domain.robot

Robot Framework's resource file's difference from RobotFramework suite file is that it does not contain any Test Cases section. It rather has Keywords section.

Let's add the KW in the newly opened resource file: 

 *** Keywords ***                                                                 
 | Vpp l2bd forwarding setup                                                      
 | | [Documentation] | Setup bridge domain between 2 interfaces on VPP node  
 | | [Arguments] | ${node} | ${if1} | ${if2} | ${learn}=${TRUE}
 | | Set Interface State | ${node} | ${if1} | up                                  
 | | Set Interface State | ${node} | ${if2} | up                                  
 | | Vpp Add L2 Bridge Domain | ${node} | ${1} | ${if1} | ${if2} | ${learn}       
 | | All Vpp Interfaces Ready Wait | ${nodes}

We've named the KW as Vpp l2bd forwarding setup, and it takes 4 arguments: node to create the bridge domain on, two interfaces to add to that bridge domain, and learning option setting (which we set to default to TRUE). In the new Keyword code we actually do what we've written above: set interfaces up, add them to a bridge domain, and then wait for those interfaces to come up. Let's go over them one by one.

  • Set Interface State

Set Interface State is is located in InterfaceUtil.py: 

<snip>
   @staticmethod                                                                
   def set_interface_state(node, interface, state):                             
       """Set interface state on a node.                                        
                                                                                
       Function can be used for DUTs as well as for TGs.                        
                                                                                
       :param node: node where the interface is                                 
       :param interface: interface name or sw_if_index                          
       :param state: one of 'up' or 'down'                                      
       :type node: dict                                                         
       :type interface: str or int                                              
       :type state: str                                                         
       :return: nothing                                                         
       """                                                                      
<snip>

RobotFramework allows calling of functions implemented in Python directly, you just have to know how the name of the function is translated to RobotFramework formatting. Every Python underscore is translated to space in RobotFramework, and case don't matter. Hence set_interface_state is called Set Interface State in RobotFramework. Parameters are delimited by pipe.

  • Vpp Add L2 Bridge Domain | ${node} | ${1} | ${if1} | ${if2} | ${learn}

This KW has to be implemented, but for now let's have a look on what parameters we are going to pass to it. We know that we are going to create a BD, and we know that it will exists on a VPP, which is on DUT, which is represented by the ${node}. Further parameters come out of necessity to pass variables to the VPP API to create BD: 

bridge_domain_add_del bd_id {bd_id} flood 1 uu-flood 1 forward 1 learn {learn} arp-term 0

We do not care for flood, uu-flood, forward and arp-term parameters, because those are going to be mostly static in our tests. So for now we only need to be able to change bridge domain ID and learn settings.


  • tags
  • documentation

CSIT Code Structure

CSIT project consists of the following:

  • RobotFramework tests, resources, and libraries.
  • bash scripts – tools, and anything system-related (copying files, installing SW on nodes, ...).
  • Python libraries
    • the brains of the execution.
    • for different functionality there is a different module, i.e.
      • vpp
        • ipv4 utils.
        • ipv6 utils.
        • xconnect.
        • bdomain.
        • VAT (vpp_api_test) helpers.
        • Config generator.
      • ssh.
      • topology.
      • packet verifier – packet generator and validator.
      • v4/v6 ip network and host address generator.
  • vpp_api_test templates.

Each RF testsuite/case has TAGs associated with it that describe what environment that it can be run on: HW/VM, or what topology it requires. RobotFramework is executed with parameter that links to topology description file, we call it topology for simplicity. This file is parsed to variable “nodes” and later used in test cases and libraries.

In general test cases are written in readable English, so that even non-coders can understand it. These top level test cases should stay the same; in other words the testcase text should not represent “how” the test is done, but “what” the test case does.

Libraries to handle VPP functionality are written in Python and are separated on per-feature basis: v4, v6, interface (admin up, state status and so on), xconnect and bdomain. More modules are going to be implemented when needed.

Performance tests are executed using packet traffic generators external to servers running VPP code. Python APIs are used to control the traffic generators. Linux Foundation hosts physical infrastructure dedicated to FD.io, consisting of three of 3-compute-node performance testbeds (compute node = x86_64 multi-core server). Two of the compute nodes run VPP code, one runs a software traffic generator. Currently CSIT performance tests are executed using trex.

CSIT Test Code Guidelines

WORK IN PROGRESS

Here are some guidelines for writing reliable, maintainable, reusable and readable Robot Framework (RF) test code. There is used Robot Framework version 2.9.2 (user guide) in CSIT.

RobotFramework test case files and resource files

  • General
    • RobotFramework test case files and resource files use special extension .robot
    • Usage of pipe and space separated file format is strongly recommended. Tabs are invisible characters which is error prone.
    • Files should be encoded in ASCII. Non-ASCII characters are allowed but they must be encoded in UTF8 (the default Robot source file encoding).
    • Line length is limited to 80 characters.
    • There must be included licence (/csit/docs/licence.rst) at the begging of each file.
    • Copy-pasting of the code is unwanted practice, any code that could be re-used has to be put into RF keyword (KW) or python library instead of copy-pasted.
  • Test cases
    • Test cases are written in Behavior-driven style – i.e. in readable English so that even non-technical project stakeholders can understand it: 
  *** Test Cases ***
  | VPP can encapsulate L2 in VXLAN over IPv4 over Dot1Q
  | | Given Path for VXLAN testing is set
  | | ...   | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']}
  | | And   Interfaces in path are up
  | | And   Vlan interfaces for VXLAN are created | ${VLAN}
  | |       ...                                   | ${dut1} | ${dut1s_to_dut2}
  | |       ...                                   | ${dut2} | ${dut2s_to_dut1}
  | | And   IP addresses are set on interfaces
  | |       ...         | ${dut1} | ${dut1s_vlan_name} | ${dut1s_vlan_index}
  | |       ...         | ${dut2} | ${dut2s_vlan_name} | ${dut2s_vlan_index}
  | | ${dut1s_vxlan}= | When Create VXLAN interface     | ${dut1} | ${VNI}
  | |                 | ...  | ${dut1s_ip_address} | ${dut2s_ip_address}
  | |                   And  Interfaces are added to BD | ${dut1} | ${BID}
  | |                   ...  | ${dut1s_to_tg} | ${dut1s_vxlan}
  | | ${dut2s_vxlan}= | And  Create VXLAN interface     | ${dut2} | ${VNI}
  | |                 | ...  | ${dut2s_ip_address} | ${dut1s_ip_address}
  | |                   And  Interfaces are added to BD | ${dut2} | ${BID}
  | |                   ...  | ${dut2s_to_tg} | ${dut2s_vxlan}
  | | Then Send and receive ICMPv4 bidirectionally
  | | ... | ${tg} | ${tgs_to_dut1} | ${tgs_to_dut2}
    • Every test case should contain short documentation. (example will be added) This documentation will be used by testdoc tool - Robot Framework's built-in tool for generating high level documentation based on test cases.
    • Do not use hard-coded constants. It is recommended to use the variable table (***Variables***) to define test case specific values. Use the assignment sign = after the variable name to make assigning variables slightly more explicit: 
  *** Variables ***
  | ${VNI}= | 23
    • Common test case specific settings of the test environment should be done in Test Setup part of the Setting table ease on (***Settings***).
    • Post-test cleaning and processing actions should be done in Test Teardown part of the Setting table (e.g. download statistics from VPP nodes). This part is executed even if the test case has failed. On the other hand it is possible to disable the tear-down from command line, thus leaving the system in “broken” state for investigation.
    • Every TC must be correctly tagged. List of defined tags is in /csit/docs/tag_documentation.rst file.
    • User high-level keywords specific for the particular test case can be implemented in the keyword table of the test case to enable readability and code-reuse.
  • Resource files
    • Used to implement higher-level keywords that are used in test cases or other higher-level keywords.
    • Every keyword must contains Documentation where the purpose and arguments of the KW are described.
    • The best practice is that the KW usage example is the part of the Documentation. It is recommended to use pipe and space separated format for the example.
    • Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).


Python library files

  • General
    • Used to implement low-level keywords that are used in resource files (to create higher-level keywords) or in test cases.
    • Higher-level keywords can be implemented in python library file too, especially in the case that their implementation in resource file would be too difficult or impossible, e.g. nested FOR loops or branching.
    • Every keyword, Python module, class, method, enums has to contain documentation string with the short description and used input parameters and possible return value(s).
    • The best practice is that the KW usage example is the part of the Documentation. It should contains two parts – RobotFramework example and Python example. It is recommended to use pipe and space separated format in case of RobotFramework example.
    • KW usage examples can be grouped and used in the class documentation string to provide better overview of the usage and relationships between KWs.
    • Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).
    • There must be included licence (/csit/docs/licence.rst) at the begging of each file.
  • Coding
    • It is recommended to use some standard development tool (e.g. PyCharm Community Edition) and follow PEP-8 recommendations.
    • All python code (not only RF libraries) must adhere to PEP-8 standard. This is enforced by CSIT Jenkins verify job.
    • Indentation – do not use tab for indents! Indent is defined as four spaces.
    • Line length – limited to 80 characters.
    • Imports - use the full pathname location of the module, e.g. from resources.libraries.python.topology import Topology. Imports should be grouped in the following order: 1. standard library imports, 2. related third party imports, 3. local application/library specific imports. You should put a blank line between each group of imports.
    • Blank lines - Two blank lines between top-level definitions, one blank line between method definitions.
    • Do not use global variables inside library files.
    • Comparisons – should be in format 0 == ret_code not ret_code == 0 to avoid possible interchange of = (assignment) and == (equal to) that could be difficult to identify such error.
    • Constants – avoid to use hard-coded constants (e.g. numbers, paths without any description). Use configuration file(s), like /csit/resources/libraries/python/constants.py, with appropriate comments.
    • Logging – log at the lowest possible level of implementation (debugging purposes). Use same style for similar events. Keep logging as verbose as necessary.
    • Exceptions – use the most appropriate exception not general one („Exception“ ) if possible. Create your own exception if necessary and implement there logging, level debug.
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/Documentation&oldid=1133"