CSIT/csit-perf-tc-naming-change
From fd.io
CSIT Performance TCs Naming
Contents
Background
Now that the number and type of CSIT test cases (TCs) grew significantly, existing naming notation does not work anymore. Firstly, the existing practice does not always follow any strict naming convention. Secondly current TC names do not always clearly capture packet encapsulations, the actual type or content of the tests. Thirdly HW configurations in terms of NICs, ports and their locality are not captured in the name. Bottom line, to address these points within the CSIT project, we agreed to define a new more complete and stricter TC naming convention, and to apply this to all existing and new test cases.
This wiki page captures this change.
CSIT Tests - Naming
Overview
The approach is to use tree naming convention and to encode following testing information into test names:
- packet network port configuration
- physical port type;
- NIC model;
- port-NIC locality;
- packet encapsulations
- VPP packet processing
- packet forwarding mode
- packet processing function(s)
- packet forwarding path
- if present, network functions (processes, containers, VMs) and their topology within the computer;
- main measured variable, type of test;
Proposed convention is to encode physical ports and NIC on the left (underlay), followed by outer most frame header, then other stacked headers up to the header processed by vSwitch-VPP, then VPP forwarding function, then encap on vhost interface, number of vhost interfaces, number of VMs. If chained VMs present, they get added on the right. Test topology is expected to be symmetric, in other words packets enter and leave SUT through ports specified on the left of the test name. Here some examples to illustrate the convention followed by the complete legend, and tables mapping the new test filenames to old ones.
Convention
CSIT test naming convention for common tested VNF topologies:
- Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P
- PortNICConfig-WireEncapsulation-PacketForwardingFunction-PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType
- 10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with MAC learning, NDR throughput discovery.
- 10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot => 2 ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching with MAC learning, NDR throughput discovery.
- 10ge2p1x520-ethip4-ip4base-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
- 10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery.
- 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control Lists baseline matching on destination, NDR throughput discovery.
- 40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot => 2 ports of 40GE on Cisco vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
- virtio2p-ethip4-ip4base-func.robot => 2 ports of Virtio, IPv4 baseline routed forwarding, functional tests.
- Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, P2V2P, NIC2VMchain2NIC, P2V2V2P
- PortNICConfig-WireEncapsulation-PacketForwardingFunction-PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-VirtPortConfig-VMconfig-TestType
- 10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching to/from two vhost interfaces and one VM, NDR throughput discovery.
- 10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost interfaces and one VM, NDR throughput discovery.
- 10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot => 2 ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from four vhost interfaces and two VMs, NDR throughput discovery.
- virtio2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot => 2 ports of Virtio, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost interfaces and one VM, functional tests.
Legend
- PortNICConfig - packet network port configuration: physical port type, NIC model, port-NIC locality
- <n>ge | virtio - port type
- <n>ge - Gigabit Ethernet, n=10|40|100|200|400.
- virtio - virtio.
- <n>p - number of physical Ethernet ports, ports distributed evenly across tested NICs, n=1,2,...
- <n><nic_model> - number of NICs and NIC model, n=1,2,...
- <n>ge | virtio - port type
- WireEncapsulation
- eth | dot1q | dot1ad - Ethernet header
- eth - untagged Ethernet on the physical wire.
- dot1q - single VLAN tag on physical wire.
- dot1ad - double VLAN tag on physical wire.
- ip4 | ip6 | ip6e - outer IP header
- ip4 - IPv4 header.
- ip6 - IPv6 header.
- ip6e - IPv6 extended header.
- ipsectrt | ipsectnl - IPSec crypto
- ipsectrt - IPSec transport mode.
- ipsectnl - IPSec tunnel mode.
- vxlan | lisp | gre | gpe - tunnel header
- vxlan - VXLAN tunnel header.
- lisp - LISP tunnel header.
- gre - GRE tunnel header.
- gpe - GPE tunnel header.
- eth | dot1q | dot1ad - Ethernet header
- PacketForwardingFunction
- l2bd - VPP L2 bridge-domain
- base | scale
- base - baseline test with one L2 MAC flow received per interface.
- scale<n> - scale test with many L2 MAC flows received per interface, <n> - number of 48-bit MAC address entries present in L2 MAC lookup table.
- maclrn | macstc
- maclrn - learning L2 MAC addresses or MAC flows.
- macstc - static L2 MAC addresses or MAC flows, programmed before the test.
- base | scale
- l2xc - VPP L2 point-to-point crossconnect
- base | scale
- base - baseline test with one L2 cross-connect configured.
- scale<n> - scale test with <n> L2 cross-connects configured.
- base | scale
- ip4 - VPP IPv4 routed forwarding
- base | scale
- base - baseline test with one IPv4 flow received per interface.
- scale<n> - scale tests with many IPv4 flows received per interface, <n> - number of /32 IPv4 address entries present in IPv4 lookup table.
- base | scale
- ip6 - VPP IPv6 routed forwarding
- base | scale
- base - baseline test with one IPv6 flow received per interface.
- scale<n> - scale tests with many IPv6 flows received per interface, <n> - number of /128 IPv6 address entries present in IPv6 lookup table.
- base | scale
- l2bd - VPP L2 bridge-domain
- PacketProcessingFunction
- l2shg - L2 Split Horizon Group.
- iacldstbase - ingress Access Control Lists baseline matching on destination.
- ipolicemarkbase - ingress policer and marker baseline.
- copwhtlistbase - ingress IP Cop security whitelist filter entries, /24 for IPv4, /64 for IPv6.
- copblklistbase - ingress IP Cop security blacklist filter entries, /24 for IPv4, /64 for IPv6
- rpf - IP Reverse Path Forwarding check.
- ip4dhcpclient - IPv4 DHCP client.
- ip4dhcpcproxy - IPv4 DHCP proxy.
- VirtEncapsulation
- eth | dot1q | dot1ad
- eth - untagged Ethernet on virtual interface.
- dot1q - single VLAN tag on virtual interface.
- dot1ad - double VLAN tag on virtual interface.
- eth | dot1q | dot1ad
- VirtPortConfig
- <n>vhost - vhost virtual interface, n=1,2, ...
- VMconfig
- <n>vm - Virtual Machines, n=1,2, ...
- TestType
- ndrdisc | ndrchk | pdrdisc | pdrchk | func
- ndrdisc - NDR throughput discovery using RFC2544 binary search.
- ndrchk - NDR throughput check against a reference NDR throughput.
- pdrdisc - PDR throughput discovery using RFC2544 binary search.
- pdrchk - PDR throughput check against a reference NDR throughput.
- func - functional tests.
- ndrdisc | ndrchk | pdrdisc | pdrchk | func
CSIT Performance Tests
Performance tests executed in FD.io CSIT CPL (Continuous Performance Lab) physical testbeds. Naming change code in FD.io CSIT project by patch #3856.
Throughput NDR Discovery
Sorted alphabetically by new name.
| # | New Test Filename | Old Test Filename |
|---|---|---|
| 1 | 10ge2p1x520-dot1ad-l2xcbase-ndrdisc.robot | Long_Xconnect_Dot1ad_Intel-X520-DA2.robot |
| 2 | 10ge2p1x520-dot1q-l2xcbase-ndrdisc.robot | Long_Xconnect_Dot1q_Intel-X520-DA2.robot |
| 3 | 10ge2p1x520-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot | Long_Bridge_Domain_Vhost_Intel-X520-DA2.robot |
| 4 | 10ge2p1x520-eth-l2bdbasemaclrn-ndrdisc.robot | Bridge_Domain_Intel-X520-DA2.robot |
| 5 | 10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm-ndrdisc.robot | Long_Xconnect_Vhost_Intel-X520-DA2.robot |
| 6 | 10ge2p1x520-eth-l2xcbase-ndrdisc.robot | Long_Xconnect_Intel-X520-DA2.robot |
| 7 | 10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrdisc.robot | Long_IPv4_Cop_Intel-X520-DA2.robot |
| 8 | 10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm-ndrdisc.robot | Long_IPv4_Vhost_Intel-X520-DA2.robot |
| 9 | 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot | Long_IPv4_iAcl_Intel-X520-DA2.robot |
| 10 | 10ge2p1x520-ethip4-ip4base-ndrdisc.robot | Long_IPv4_Intel-X520-DA2.robot |
| 11 | 10ge2p1x520-ethip4-ip4base-ipolicemarkbase-ndrdisc.robot | Long_IPv4_Policer_Intel-X520-DA2.robot |
| 12 | 10ge2p1x520-ethip4-ip4scale200k-ndrdisc.robot | Long_IPv4_Fib_200k_Intel-X520-DA2.robot |
| 13 | 10ge2p1x520-ethip4-ip4scale20k-ndrdisc.robot | Long_IPv4_Fib_20k_Intel-X520-DA2.robot |
| 14 | 10ge2p1x520-ethip4-ip4scale2m-ndrdisc.robot | Long_IPv4_Fib_2M_Intel-X520-DA2.robot |
| 15 | 10ge2p1x520-ethip4lispip4-ip4base-ndrdisc.robot | Long_Lisp_Intel-X520-DA2.robot |
| 16 | 10ge2p1x520-ethip4lispip6-ip4base-ndrdisc.robot | Long_Lisp_Intel-X520-DA2.robot |
| 17 | 10ge2p1x520-ethip4vxlan-l2xcbase-ndrdisc.robot | Long_Xconnect_Vxlan_Ipv4_Intel-X520-DA2.robot |
| 18 | 10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrdisc.robot | Long_IPv6_Cop_Intel-X520-DA2.robot |
| 19 | 10ge2p1x520-ethip6-ip6base-iacldstbase-ndrdisc.robot | Long_IPv6_iAcl_Intel-X520-DA2.robot |
| 20 | 10ge2p1x520-ethip6-ip6base-ndrdisc.robot | Long_IPv6_Intel-X520-DA2.robot |
| 21 | 10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot | Long_IPv6_Fib_200k_Intel-X520-DA2.robot |
| 22 | 10ge2p1x520-ethip6-ip6scale20k-ndrdisc.robot | Long_IPv6_Fib_20k_Intel-X520-DA2.robot |
| 23 | 10ge2p1x520-ethip6-ip6scale2m-ndrdisc.robot | Long_IPv6_Fib_2M_Intel-X520-DA2.robot |
| 24 | 10ge2p1x520-ethip6lispip4-ip6base-ndrdisc.robot | Long_Lisp_Intel-X520-DA2.robot |
| 25 | 10ge2p1x520-ethip6lispip6-ip6base-ndrdisc.robot | Long_Lisp_Intel-X520-DA2.robot |
| 26 | 10ge2p1x710-eth-l2bdbasemaclrn-ndrdisc.robot | Bridge_Domain_Intel-X710-DA2.robot |
| 27 | 40ge2p1xl710-eth-l2bdbasemaclrn-ndrdisc.robot | Bridge_Domain_Intel-XL710.robot |
| 28 | 40ge2p1xl710-eth-l2xcbase-ndrdisc.robot | Long_Xconnect_Intel-XL710.robot |
| 29 | 40ge2p1xl710-ethip4-ip4base-ndrdisc.robot | Long_IPv4_Intel-XL710.robot |
| 30 | 40ge2p1xl710-ethip6-ip6base-ndrdisc.robot | Long_IPv6_Intel-XL710.robot |
Throughput NDR Check
| # | New Test Filename | Old Test Filename |
|---|---|---|
| 1 | 10ge2p1x520-dot1ad-l2xcbase-ndrchk.robot | Short_Xconnect_Dot1ad_Intel-X520-DA2.robot |
| 2 | 10ge2p1x520-dot1q-l2xcbase-ndrchk.robot | Short_Xconnect_Dot1q_Intel-X520-DA2.robot |
| 3 | 10ge2p1x520-eth-l2bdbasemaclrn-ndrchk.robot | Short_Bridge_Domain_Intel-X520-DA2.robot |
| 4 | 10ge2p1x520-eth-l2xcbase-ndrchk.robot | Short_Xconnect_Intel-X520-DA2.robot |
| 5 | 10ge2p1x520-ethip4-ip4base-copwhtlistbase-ndrchk.robot | Short_IPv4_Cop_Intel-X520-DA2.robot |
| 6 | 10ge2p1x520-ethip4-ip4base-iacldstbase-ndrchk.robot | Short_IPv4_iAcl_Intel-X520-DA2.robot |
| 7 | 10ge2p1x520-ethip4-ip4base-ndrchk.robot | Short_IPv4_Intel-X520-DA2.robot |
| 8 | 10ge2p1x520-ethip4-ip4base-ipolicemarkbase-ndrchk.robot | Short_IPv4_Policer_Intel-X520-DA2.robot |
| 9 | 10ge2p1x520-ethip4-ip4scale200k-ndrchk.robot | Short_IPv4_Fib_200k_Intel-X520-DA2.robot |
| 10 | 10ge2p1x520-ethip4-ip4scale20k-ndrchk.robot | Short_IPv4_Fib_20k_Intel-X520-DA2.robot |
| 11 | 10ge2p1x520-ethip4-ip4scale2m-ndrchk.robot | Short_IPv4_Fib_2M_Intel-X520-DA2.robot |
| 12 | 10ge2p1x520-ethip4lispip4-ip4base-ndrchk.robot | Short_Lisp_Intel-X520-DA2.robot |
| 13 | 10ge2p1x520-ethip4lispip6-ip4base-ndrchk.robot | Short_Lisp_Intel-X520-DA2.robot |
| 14 | 10ge2p1x520-ethip4vxlan-l2xcbase-ndrchk.robot | Short_Xconnect_Vxlan_Ipv4_Intel-X520-DA2.robot |
| 15 | 10ge2p1x520-ethip6-ip6base-copwhtlistbase-ndrchk.robot | Short_IPv6_Cop_Intel-X520-DA2.robot |
| 16 | 10ge2p1x520-ethip6-ip6base-iacldstbase-ndrchk.robot | Short_IPv6_iAcl_Intel-X520-DA2.robot |
| 17 | 10ge2p1x520-ethip6-ip6base-ndrchk.robot | Short_IPv6_Intel-X520-DA2.robot |
| 18 | 10ge2p1x520-ethip6-ip6scale200k-ndrchk.robot | Short_IPv6_Fib_200k_Intel-X520-DA2.robot |
| 19 | 10ge2p1x520-ethip6-ip6scale20k-ndrchk.robot | Short_IPv6_Fib_20k_Intel-X520-DA2.robot |
| 20 | 10ge2p1x520-ethip6-ip6scale2m-ndrchk.robot | Short_IPv6_Fib_2M_Intel-X520-DA2.robot |
| 21 | 10ge2p1x520-ethip6lispip4-ip6base-ndrchk.robot | Short_Lisp_Intel-X520-DA2.robot |
| 22 | 10ge2p1x520-ethip6lispip6-ip6base-ndrchk.robot | Short_Lisp_Intel-X520-DA2.robot |
Trending Graphs Tagging
In order to group performance TC results into performance trending graphs, CSIT is using RF tags. CSIT RF tags are described in `/docs/tag_documentation.rst'. RF tag changes related to performance TC naming change are captured as part of patch #3856.
CSIT Functional Tests
Functional tests executed in FD.io CSIT VIRL testbeds. Naming change coded in FD.io CSIT project by [<gerrit link to be added> patch #<gerrit patch id to be added>].
Sorted alphabetically by new name.
| # | New Test Filename | Old Test Filename |
|---|---|---|
| 1 | virtio2p-eth-l2bdbasemaclrn-func.robot | bridge_domain_untagged.robot |
| 2 | virtio2p-eth-l2bdbasemacstc-func.robot | bridge_domain_untagged.robot |
| 3 | virtio2p-eth-l2bdbasemaclrn-l2shg-func.robot | bridge_domain_untagged.robot |
| 4 | virtio2p-eth-l2bdbasemaclrn-eth-2vhost-1vm-func.robot | bridge_domain_untagged.robot |
| 5 | virtio2p-eth-l2bdbasemacstc-eth-2vhost-1vm-func.robot | bridge_domain_untagged.robot |
| 6 | virtio2p-ethip4-ip4base-copwhlistbase-func.robot | cop_whitelist_blacklist.robot |
| 7 | virtio2p-ethip4-ip4base-copblklistbase-func.robot | cop_whitelist_blacklist.robot |
| 8 | virtio2p-ethip6-ip6base-copwhlistbase-func.robot | cop_whitelist_blacklist_IPv6.robot |
| 9 | virtio2p-ethip6-ip6base-copblklistbase-func.robot | cop_whitelist_blacklist_IPv6.robot |
| 10 | virtio2p-ethip4-ip4base-ip4dhcpclient-func.robot | dhcp_client.robot |
| 11 | virtio2p-ethip4-ip4base-ip4dhcpproxy-func.robot | dhcp_proxy.robot |
| 12 | virtio2p-ethip4ipsectpt-ip4base-func.robot | ipsec_transport_mode_ipv4.robot |
| 13 | virtio2p-ethip4ipsectnl-ip4base-func.robot | ipsec_tunnel_mode_ipv4.robot |
| 15 | virtio2p-ethip6ipsectpt-ip6base-func.robot | ipsec_transport_mode_ipv6.robot |
| 16 | virtio2p-ethip6psectnl-ip6base-func.robot | ipsec_tunnel_mode_ipv6.robot |
| 17 | virtio2p-ethip4ipsectptgpe-ip4base-func.robot | ipsec_ipv4_lispgpe_ipv4.robot |
| 18 | virtio2p-ethip6ipsectptgpe-ip4base-func.robot | ipsec_ipv4_lispgpe_ipv6.robot |
| 19 | virtio2p-ethip4ipsectptgpe-ip6base-func.robot | ipsec_ipv6_lispgpe_ipv4.robot |
| 20 | virtio2p-ethip6ipsectptgpe-ip6base-func.robot | ipsec_ipv6_lispgpe_ipv6.robot |
| n | to be completed | to be completed |
