CSIT/csit-perf-tc-naming-change

From fd.io
< CSIT
Revision as of 21:39, 5 January 2017 by Mackonstan (Talk | contribs)

Jump to: navigation, search

CSIT Performance TCs Naming

Background

Now that number and type of CSIT test cases (TCs) grew significantly, existing naming notation does not work anymore. Firstly, the existing practice does not follow any strict naming convention. Secondly current TC names do not clearly capture neither encapsulations nor the actual type and content of the tests. Thirdly HW configurations in terms of NICs and ports number and locality are not captured in the name. Bottom line, within the CSIT project we agreed to define a new stricter TC naming convention and update all existing test cases.

This wiki page captures this change.

FD.io Gerrit Patches

  1. Performance tests
  2. Functional tests
    • <to be added>

CSIT Test Files - Naming Scheme

Test topologies and functions encoded using tree notation with physical ports and NIC on the left (underlay), followed by outer most frame header, then other stacked headers up to the header processed by vSwitch-VPP, then VPP forwarding function, then encap on vhost interface, number of vhost interfaces, number of VMs. Some examples and legend here:

  1. IPv4 routed forwarding with Intel x520 2-port 10GE NICs - NDR throughput discovery
    • 10ge2p1x520-ethip4-ip4base-ndrdisc.robot
    • 10ge2p1x520-ethip4-ip4scale20k-ndrdisc.robot
    • 10ge2p1x520-ethip4-ip4scale200k-ndrdisc.robot
    • 10ge2p1x520-ethip4-ip4scale2m-ndrdisc.robot
  2. IPv4 routed forwarding with Intel x520 2-port 10GE NICs - NDR throughput check
    • 10ge2p1x520-ethip4-ip4base-ndrchk.robot
    • 10ge2p1x520-ethip4-ip4scale20k-ndrchk.robot
    • 10ge2p1x520-ethip4-ip4scale200k-ndrchk.robot
    • 10ge2p1x520-ethip4-ip4scale2m-ndrchk.robot
  3. IPv4 routed forwarding with different NICs - x710, xl710, vic1227, vic1385 - NDR throughput discovery
    • 10ge2p1x710-ethip4-ip4base-ndrdisc.robot
    • 10ge2p1vic1227-ethip4-ip4base-ndrdisc.robot
    • 40ge2p1xl710-ethip4-ip4base-ndrdisc.robot
    • 40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot
  4. L2 bridge-domain switching in different VLAN and vhost-user setups - NDR throughput discovery
    • 10ge2p1x520-eth-l2bdbase-eth-2vhost-1vm-ndrdisc.robot
    • 10ge2p1x520-dot1q-l2bdbase-eth-2vhost-1vm-ndrdisc.robot
    • 10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-ndrdisc.robot
    • 10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-ndrdisc.robot
    • 10ge2p1x520-dot1q-l2xcbase-eth-2vhost-1vm-ndrdisc.robot
    • 10ge2p1x520-ethip4vxlan-l2bdbase-eth-2vhost-1vm-ndrdisc.robot

Naming legend:

  1. <n>ge - Gigabit Ethernet port type, n=10|40|100|200|400
  2. <n>p - number of physical Ethernet ports
  3. <n><nic_model> - number of NICs and NIC model
  4. eth|dot1q|dot1ad
    • eth - untagged Ethernet on the physical wire
    • dot1q - single VLAN tag on physical wire
    • dot1ad - double VLAN tag on physical wire
  5. ip4|ip6 - outer IP header
    • ip4 - IPv4 header
    • ip6 - IPv6 header
  6. vxlan|lisp|gre - tunnel header
    • vxlan - VXLAN tunnel header
    • lisp - LISP tunnel header
    • gre - GRE tunnel header
    • gpe - GPE tunnel header
  7. l2bd - VPP L2 bridge-domain
    • base|scale
      • base - baseline test with one L2 MAC flow received per interface
      • scale - scale test with many L2 MAC flows received per interface
    • maclrn|macstc
      • maclrn - learning L2 MAC addresses or MAC flows
      • macstc - static L2 MAC addresses or MAC flows, programmed before the test
  8. l2xc - VPP L2 point-to-point crossconnect
    • base|scale
      • base - baseline test with one L2 MAC flow received per interface
      • scale - scale test with many L2 MAC flows received per interface
  9. ip4 - VPP IPv4 routed forwarding
    • base|scale
      • base - baseline test with one IPv4 flow received per interface
      • scale - scale tests with many IPv4 flows received per interface
  10. eth|dot1q|dot1ad
    • eth - untagged Ethernet on virtual interface
    • dot1q - single VLAN tag on virtual interface
    • dot1ad - double VLAN tag on virtual interface
  11. <n>vhost - vhost virtual interface, n=1,2, ...
  12. <n>vm - Virtual Machines, n=1,2, ...
  13. ndrdisc|ndrchk|pdrdisc|pdrchk
    • ndrdisc - NDR throughput discovery using RFC2544 binary search
    • ndrchk - NDR throughput check against a reference NDR throughput

Performance TCs

Throughput NDR Discovery

Sorted alphabetically by new name.

# New Test File Name Old Test File Name
1 10ge2p1x520-dot1ad-l2xcbase-ndrdisc.robot Long_Xconnect_Dot1ad_Intel-X520-DA2.robot
2 10ge2p1x520-dot1q-l2xcbase-ndrdisc.robot Long_Xconnect_Dot1q_Intel-X520-DA2.robot
3 10ge2p1x520-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot Long_Bridge_Domain_Vhost_Intel-X520-DA2.robot
4 10ge2p1x520-eth-l2bdbasemaclrn-ndrdisc.robot Bridge_Domain_Intel-X520-DA2.robot
5 10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-ndrdisc.robot Long_Xconnect_Vhost_Intel-X520-DA2.robot
6 10ge2p1x520-eth-l2xcbase-ndrdisc.robot Long_Xconnect_Intel-X520-DA2.robot
7 10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrdisc.robot Long_IPv4_Cop_Intel-X520-DA2.robot
8 10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-ndrdisc.robot Long_IPv4_Vhost_Intel-X520-DA2.robot
9 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot Long_IPv4_iAcl_Intel-X520-DA2.robot
10 10ge2p1x520-ethip4-ip4base-ndrdisc.robot Long_IPv4_Intel-X520-DA2.robot
11 10ge2p1x520-ethip4-ip4base-policermarkerbase-ndrdisc.robot Long_IPv4_Policer_Intel-X520-DA2.robot
12 10ge2p1x520-ethip4-ip4scale200k-ndrdisc.robot Long_IPv4_Fib_200k_Intel-X520-DA2.robot
13 10ge2p1x520-ethip4-ip4scale20k-ndrdisc.robot Long_IPv4_Fib_20k_Intel-X520-DA2.robot
14 10ge2p1x520-ethip4-ip4scale2m-ndrdisc.robot Long_IPv4_Fib_2M_Intel-X520-DA2.robot
15 10ge2p1x520-ethip4lispip4-ip4base-ndrdisc.robot Long_Lisp_Intel-X520-DA2.robot
16 10ge2p1x520-ethip4lispip6-ip4base-ndrdisc.robot Long_Lisp_Intel-X520-DA2.robot
17 10ge2p1x520-ethip4vxlan-l2xcbase-ndrdisc.robot Long_Xconnect_Vxlan_Ipv4_Intel-X520-DA2.robot
18 10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrdisc.robot Long_IPv6_Cop_Intel-X520-DA2.robot
19 10ge2p1x520-ethip6-ip6base-iacldstbase-ndrdisc.robot Long_IPv6_iAcl_Intel-X520-DA2.robot
20 10ge2p1x520-ethip6-ip6base-ndrdisc.robot Long_IPv6_Intel-X520-DA2.robot
21 10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot Long_IPv6_Fib_200k_Intel-X520-DA2.robot
22 10ge2p1x520-ethip6-ip6scale20k-ndrdisc.robot Long_IPv6_Fib_20k_Intel-X520-DA2.robot
23 10ge2p1x520-ethip6-ip6scale2m-ndrdisc.robot Long_IPv6_Fib_2M_Intel-X520-DA2.robot
24 10ge2p1x520-ethip6lispip4-ip6base-ndrdisc.robot Long_Lisp_Intel-X520-DA2.robot
25 10ge2p1x520-ethip6lispip6-ip6base-ndrdisc.robot Long_Lisp_Intel-X520-DA2.robot
26 10ge2p1x710-eth-l2bdbasemaclrn-ndrdisc.robot Bridge_Domain_Intel-X710-DA2.robot
27 40ge2p1xl710-eth-l2bdbasemaclrn-ndrdisc.robot Bridge_Domain_Intel-XL710.robot
28 40ge2p1xl710-eth-l2xcbase-ndrdisc.robot Long_Xconnect_Intel-XL710.robot
29 40ge2p1xl710-ethip4-ip4base-ndrdisc.robot Long_IPv4_Intel-XL710.robot
30 40ge2p1xl710-ethip6-ip6base-ndrdisc.robot Long_IPv6_Intel-XL710.robot

Throughput NDR Check

# New Test File Name Old Test File Name
1 10ge2p1x520-dot1ad-l2xcbase-ndrchk.robot Short_Xconnect_Dot1ad_Intel-X520-DA2.robot
2 10ge2p1x520-dot1q-l2xcbase-ndrchk.robot Short_Xconnect_Dot1q_Intel-X520-DA2.robot
3 10ge2p1x520-eth-l2bdbasemaclrn-ndrchk.robot Short_Bridge_Domain_Intel-X520-DA2.robot
4 10ge2p1x520-eth-l2xcbase-ndrchk.robot Short_Xconnect_Intel-X520-DA2.robot
5 10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrchk.robot Short_IPv4_Cop_Intel-X520-DA2.robot
6 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrchk.robot Short_IPv4_iAcl_Intel-X520-DA2.robot
7 10ge2p1x520-ethip4-ip4base-ndrchk.robot Short_IPv4_Intel-X520-DA2.robot
8 10ge2p1x520-ethip4-ip4base-policermarkerbase-ndrchk.robot Short_IPv4_Policer_Intel-X520-DA2.robot
9 10ge2p1x520-ethip4-ip4scale200k-ndrchk.robot Short_IPv4_Fib_200k_Intel-X520-DA2.robot
10 10ge2p1x520-ethip4-ip4scale20k-ndrchk.robot Short_IPv4_Fib_20k_Intel-X520-DA2.robot
11 10ge2p1x520-ethip4-ip4scale2m-ndrchk.robot Short_IPv4_Fib_2M_Intel-X520-DA2.robot
12 10ge2p1x520-ethip4lispip4-ip4base-ndrchk.robot Short_Lisp_Intel-X520-DA2.robot
13 10ge2p1x520-ethip4lispip6-ip4base-ndrchk.robot Short_Lisp_Intel-X520-DA2.robot
14 10ge2p1x520-ethip4vxlan-l2xcbase-ndrchk.robot Short_Xconnect_Vxlan_Ipv4_Intel-X520-DA2.robot
15 10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrchk.robot Short_IPv6_Cop_Intel-X520-DA2.robot
16 10ge2p1x520-ethip6-ip6base-iacldstbase-ndrchk.robot Short_IPv6_iAcl_Intel-X520-DA2.robot
17 10ge2p1x520-ethip6-ip6base-ndrchk.robot Short_IPv6_Intel-X520-DA2.robot
18 10ge2p1x520-ethip6-ip6scale200k-ndrchk.robot Short_IPv6_Fib_200k_Intel-X520-DA2.robot
19 10ge2p1x520-ethip6-ip6scale20k-ndrchk.robot Short_IPv6_Fib_20k_Intel-X520-DA2.robot
20 10ge2p1x520-ethip6-ip6scale2m-ndrchk.robot Short_IPv6_Fib_2M_Intel-X520-DA2.robot
21 10ge2p1x520-ethip6lispip4-ip6base-ndrchk.robot Short_Lisp_Intel-X520-DA2.robot
22 10ge2p1x520-ethip6lispip6-ip6base-ndrchk.robot Short_Lisp_Intel-X520-DA2.robot

Trending Graphs Tagging

In order to group performance TC results into performance trending graphs, CSIT is using RF tags. CSIT RF tags are described in `/docs/tag_documentation.rst'. RF tag changes related to performance TC naming change are captured as part of patch #3856.

Functional TCs

<to be added>

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/csit-perf-tc-naming-change&oldid=4032"