CSIT/Documentation

From fd.io
< CSIT
Revision as of 20:47, 15 April 2016 by Skobza (Talk | contribs)

Jump to: navigation, search


WORK IN PROGRESS

CSIT components

Code in CSIT project is built on two biggest components: RobotFramework (RF in further text) and Python.

RF is test automation tool, which allows to select and run test suites and test cases, on selected target topology (nodes) and provide us with output logs in readable and parse-able form. RF uses column-based formatting for its files. For clarity and to avoid problems, CSIT team has selected the "pipe and space separated" format (rather than tab-separated format). RF is case insensitive, but we strive to be consistent in how we use upper/lower case in RF files. The value of RF really is its readability; anyone who spends a little time reading RF source files can understand what is going on, and what is more important - even non-programmers can understand the essence of what given test case is doing. There are only two types of RF files stored in CSIT: resource (aka library) files, and test suites (any RF file containing test case). In resource files we store all Keywords (RF's name for functions/methods) that are generic, and could be re-used. For example NIC manipulation is going to be re-used in nearly all test-suites, hence it is placed in file for common reuse as a RF library. All tests in RF format are stored in tests/suites/ sub directories. RF interprets every directory here as test suite, as well as all files containing test cases are taken as test suites.


Python is a component that needs no introduction by itself. In CSIT, we use Python to perform tasks that are unsuitable for RF format. Since RF is written in Python, it integrates with Python extensions quite nicely. RF files (test suites, resource files) can import .py scripts directly, and reference classes and functions from python modules easily. So in general the Test case definitions are meant to be written in RF format in RF file (suffix of the file is typically .robot), and anything more sophisticated (e.g. SSH handling) is done in Python library. 

- RF tests, suites, libs
- Python libs
- Python scapy
- Python 
- topology

- pybot execution - tags & how it helps execution execution - setup parallel

- reading

Typical problems - vpp-csit-verify job failed, now what? - how to read console logs - reading robotframework log - timeout problems

CSIT Code Structure

CSIT project consists of the following:

  • RobotFramework tests, resources, and libraries.
  • bash scripts – tools, and anything system-related (copying files, installing SW on nodes, ...).
  • Python libraries
    • the brains of the execution.
    • for different functionality there is a different module, i.e.
      • vpp
        • ipv4 utils.
        • ipv6 utils.
        • xconnect.
        • bdomain.
        • VAT (vpp_api_test) helpers.
        • Config generator.
      • ssh.
      • topology.
      • packet verifier – packet generator and validator.
      • v4/v6 ip network and host address generator.
  • vpp_api_test templates.

Each RF testsuite/case has TAGs associated with it that describe what environment that it can be run on: HW/VM, or what topology it requires. RobotFramework is executed with parameter that links to topology description file, we call it topology for simplicity. This file is parsed to variable “nodes” and later used in test cases and libraries.

In general test cases are written in readable English, so that even non-coders can understand it. These top level test cases should stay the same; in other words the testcase text should not represent “how” the test is done, but “what” the test case does.

Libraries to handle VPP functionality are written in Python and are separated on per-feature basis: v4, v6, interface (admin up, state status and so on), xconnect and bdomain. More modules are going to be implemented when needed.

Performance tests are executed using packet traffic generators external to servers running VPP code. Python APIs are used to control the traffic generators. Linux Foundation hosts physical infrastructure dedicated to FD.io, consisting of three of 3-compute-node performance testbeds (compute node = x86_64 multi-core server). Two of the compute nodes run VPP code, one runs a software traffic generator. Currently CSIT performance tests are executed using trex.

CSIT Test Code Guidelines

WORK IN PROGRESS

Here are some guidelines for writing reliable, maintainable, reusable and readable Robot Framework (RF) test code. There is used Robot Framework version 2.9.2 (user guide) in CSIT.

RobotFramework test case files and resource files

  • General
    • RobotFramework test case files and resource files use special extension .robot
    • Usage of pipe and space separated file format is strongly recommended. Tabs are invisible characters which is error prone.
    • Files should be encoded in ASCII. Non-ASCII characters are allowed but they must be encoded in UTF8 (the default Robot source file encoding).
    • Line length is limited to 80 characters.
    • There must be included licence (/csit/docs/licence.rst) at the begging of each file.
    • Copy-pasting of the code is unwanted practice, any code that could be re-used has to be put into RF keyword (KW) or python library instead of copy-pasted.
  • Test cases
    • Test cases are written in Behavior-driven style – i.e. in readable English so that even non-technical project stakeholders can understand it: 
  *** Test Cases ***
  | VPP can encapsulate L2 in VXLAN over IPv4 over Dot1Q
  | | Given Path for VXLAN testing is set
  | | ...   | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']}
  | | And   Interfaces in path are up
  | | And   Vlan interfaces for VXLAN are created | ${VLAN}
  | |       ...                                   | ${dut1} | ${dut1s_to_dut2}
  | |       ...                                   | ${dut2} | ${dut2s_to_dut1}
  | | And   IP addresses are set on interfaces
  | |       ...         | ${dut1} | ${dut1s_vlan_name} | ${dut1s_vlan_index}
  | |       ...         | ${dut2} | ${dut2s_vlan_name} | ${dut2s_vlan_index}
  | | ${dut1s_vxlan}= | When Create VXLAN interface     | ${dut1} | ${VNI}
  | |                 | ...  | ${dut1s_ip_address} | ${dut2s_ip_address}
  | |                   And  Interfaces are added to BD | ${dut1} | ${BID}
  | |                   ...  | ${dut1s_to_tg} | ${dut1s_vxlan}
  | | ${dut2s_vxlan}= | And  Create VXLAN interface     | ${dut2} | ${VNI}
  | |                 | ...  | ${dut2s_ip_address} | ${dut1s_ip_address}
  | |                   And  Interfaces are added to BD | ${dut2} | ${BID}
  | |                   ...  | ${dut2s_to_tg} | ${dut2s_vxlan}
  | | Then Send and receive ICMPv4 bidirectionally
  | | ... | ${tg} | ${tgs_to_dut1} | ${tgs_to_dut2}
    • Every test case should contain short documentation. (example will be added) This documentation will be used by testdoc tool - Robot Framework's built-in tool for generating high level documentation based on test cases.
    • Do not use hard-coded constants. It is recommended to use the variable table (***Variables***) to define test case specific values. Use the assignment sign = after the variable name to make assigning variables slightly more explicit: 
  *** Variables ***
  | ${VNI}= | 23
    • Common test case specific settings of the test environment should be done in Test Setup part of the Setting table ease on (***Settings***).
    • Post-test cleaning and processing actions should be done in Test Teardown part of the Setting table (e.g. download statistics from VPP nodes). This part is executed even if the test case has failed. On the other hand it is possible to disable the tear-down from command line, thus leaving the system in “broken” state for investigation.
    • Every TC must be correctly tagged. List of defined tags is in /csit/docs/tag_documentation.rst file.
    • User high-level keywords specific for the particular test case can be implemented in the keyword table of the test case to enable readability and code-reuse.
  • Resource files
    • Used to implement higher-level keywords that are used in test cases or other higher-level keywords.
    • Every keyword must contains Documentation where the purpose and arguments of the KW are described.
    • The best practice is that the KW usage example is the part of the Documentation. It is recommended to use pipe and space separated format for the example.
    • Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).


Python library files

  • General
    • Used to implement low-level keywords that are used in resource files (to create higher-level keywords) or in test cases.
    • Higher-level keywords can be implemented in python library file too, especially in the case that their implementation in resource file would be too difficult or impossible, e.g. nested FOR loops or branching.
    • Every keyword, Python module, class, method, enums has to contain documentation string with the short description and used input parameters and possible return value(s).
    • The best practice is that the KW usage example is the part of the Documentation. It should contains two parts – RobotFramework example and Python example. It is recommended to use pipe and space separated format in case of RobotFramework example.
    • KW usage examples can be grouped and used in the class documentation string to provide better overview of the usage and relationships between KWs.
    • Keyword name should describe what the keyword does, specifically and in a reasonable length (“short sentence”).
    • There must be included licence (/csit/docs/licence.rst) at the begging of each file.
  • Coding
    • It is recommended to use some standard development tool (e.g. PyCharm Community Edition) and follow PEP-8 recommendations.
    • All python code (not only RF libraries) must adhere to PEP-8 standard. This is enforced by CSIT Jenkins verify job.
    • Indentation – do not use tab for indents! Indent is defined as four spaces.
    • Line length – limited to 80 characters.
    • Imports - use the full pathname location of the module, e.g. from resources.libraries.python.topology import Topology. Imports should be grouped in the following order: 1. standard library imports, 2. related third party imports, 3. local application/library specific imports. You should put a blank line between each group of imports.
    • Blank lines - Two blank lines between top-level definitions, one blank line between method definitions.
    • Do not use global variables inside library files.
    • Comparisons – should be in format 0 == ret_code not ret_code == 0 to avoid possible interchange of = (assignment) and == (equal to) that could be difficult to identify such error.
    • Constants – avoid to use hard-coded constants (e.g. numbers, paths without any description). Use configuration file(s), like /csit/resources/libraries/python/constants.py, with appropriate comments.
    • Logging – log at the lowest possible level of implementation (debugging purposes). Use same style for similar events. Keep logging as verbose as necessary.
    • Exceptions – use the most appropriate exception not general one („Exception“ ) if possible. Create your own exception if necessary and implement there logging, level debug.
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/Documentation&oldid=870"