CSIT/csit-perf-tc-naming-changeLast updated 8 years ago

CSIT Performance TCs Naming

Background

Now that number and type of CSIT test cases (TCs) grew significantly, existing naming notation does not work anymore. Firstly, the existing practice does not follow any strict naming convention. Secondly current TC names do not clearly capture neither encapsulations nor the actual type and content of the tests. Thirdly HW configurations in terms of NICs and ports number and locality are not captured in the name. Bottom line, within the CSIT project we agreed to define a new stricter TC naming convention and update all existing test cases.

This wiki page captures this change.

FD.io Gerrit Patches

1. Performance tests
   • patch #3856
2. Functional tests
   • <to be added>

Performance TCs - Naming Scheme

Test topologies and functions encoded using tree notation with physical ports and NIC on the left (underlay), followed by outer most frame header, then other stacked headers up to the header processed by vSwitch-VPP, then VPP forwarding function, then encap on vhost interface, number of vhost interfaces, number of VMs. Some examples and legend here:

1. 10ge2p1x520-ethip4-ip4base2.robot
2. 10ge2p1x520-ethip4-ip4base20k.robot
3. 10ge2p1x520-ethip4-ip4base200k.robot
4. 10ge2p1x520-ethip4-ip4base2m.robot

1. 10ge2p1x710-ethip4-ip4base2.robot
2. 10ge2p1vic1227-ethip4-ip4base2.robot
3. 40ge2p1xl710-ethip4-ip4base2.robot
4. 40ge2p1vic1385-ethip4-ip4base2.robot

1. 10ge2p1x520-eth-l2bdbase-eth-2vhost-1vm
2. 10ge2p1x520-dot1q-l2bdbase-eth-2vhost-1vm
3. 10ge2p1x520-ethip4-ip4base-eth-2vhost-1vm
4. 10ge2p1x520-eth-l2xcbase-eth-2vhost-1vm
5. 10ge2p1x520-dot1q-l2xcbase-eth-2vhost-1vm
6. 10ge2p1x520-ethip4vxlan-l2bdbase-eth-2vhost-1vm

Legend:

• <n>ge - n=10|40|100
• <n>p - number of physical Ethernet ports
• <n><nic_model> - number of NICs and NIC model
• eth|dot1q|dot1ad
  • eth - untagged Ethernet on the physical wire
  • dot1q - single VLAN tag on physical wire
  • dot1ad - double VLAN tag on physical wire
• ip4|ip6
  • ip4 - IPv4 header
  • ip6 - IPv6 header
• vxlan - VXLAN header
• l2bdbase, l2bdscale - VPP L2 bridge-domain, L2 MAC learning&switching
  • base - baseline test with one L2 MAC flow received per interface
  • scale - scale tests with many L2 MAC flows received per interface
  • distinct L@ MAC flow - distinct (src-mac,dst-mac) tuple in MAC headers
• l2xcbase, l2xcscale - VPP L2 point-to-point crossconnect
  • base - baseline test with one L2 MAC flow received per interface
  • scale - scale tests with many L2 MAC flows received per interface
  • distinct L2 MAC flow - distinct (src-mac,dst-mac) tuple in MAC headers
• ip4base, ip4scale - VPP IPv4 routed forwarding
  • base - baseline test with one IPv4 flow received per interface
  • scale - scale tests with many IPv4 flows received per interface
  • distinct IPv4 flow - distinct (src-ip4,dst-ip4) tuple in IPv4 headers
• Short tests:
  • use suffix "-ndrchk"

Performance TCs - New Names

Listing order is the same as for old names, listed in next section:

1. 10ge2p1x520-eth-l2bdbasemaclrn-ndrchk.robot
2. 10ge2p1x710-eth-l2bdbasemaclrn-ndrdisc.robot
3. 40ge2p1xl710-eth-l2bdbasemaclrn-ndrdisc.robot

<list to be completed>

10ge2p1x520-dot1ad-l2xcbase-ndrchk.robot

Performance TCs - Old Names

Listing order is the same as for new names, listed in previous section:

1. Bridge_Domain_Intel-X520-DA2.robot
2. Bridge_Domain_Intel-X710-DA2.robot
3. Bridge_Domain_Intel-XL710.robot
4. Long_Bridge_Domain_Vhost_Intel-X520-DA2.robot
5. Long_IPv4_Cop_Intel-X520-DA2.robot
6. Long_IPv4_Fib_200k_Intel-X520-DA2.robot
7. Long_IPv4_Fib_20k_Intel-X520-DA2.robot
8. Long_IPv4_Fib_2M_Intel-X520-DA2.robot
9. Long_IPv4_Intel-X520-DA2.robot
10. Long_IPv4_Intel-XL710.robot
11. Long_IPv4_Policer_Intel-X520-DA2.robot
12. Long_IPv4_Vhost_Intel-X520-DA2.robot
13. Long_IPv4_iAcl_Intel-X520-DA2.robot
14. Long_IPv6_Cop_Intel-X520-DA2.robot
15. Long_IPv6_Fib_200k_Intel-X520-DA2.robot
16. Long_IPv6_Fib_20k_Intel-X520-DA2.robot
17. Long_IPv6_Fib_2M_Intel-X520-DA2.robot
18. Long_IPv6_Intel-X520-DA2.robot
19. Long_IPv6_Intel-XL710.robot
20. Long_IPv6_iAcl_Intel-X520-DA2.robot
21. Long_Lisp_Intel-X520-DA2.robot
22. Long_Xconnect_Dot1ad_Intel-X520-DA2.robot
23. Long_Xconnect_Dot1q_Intel-X520-DA2.robot
24. Long_Xconnect_Intel-X520-DA2.robot
25. Long_Xconnect_Intel-XL710.robot
26. Long_Xconnect_Vhost_Intel-X520-DA2.robot
27. Long_Xconnect_Vxlan_Ipv4_Intel-X520-DA2.robot
28. Short_Bridge_Domain_Intel-X520-DA2.robot
29. Short_IPv4_Cop_Intel-X520-DA2.robot
30. Short_IPv4_Fib_200k_Intel-X520-DA2.robot
31. Short_IPv4_Fib_20k_Intel-X520-DA2.robot
32. Short_IPv4_Fib_2M_Intel-X520-DA2.robot
33. Short_IPv4_Intel-X520-DA2.robot
34. Short_IPv4_Policer_Intel-X520-DA2.robot
35. Short_IPv4_iAcl_Intel-X520-DA2.robot
36. Short_IPv6_Cop_Intel-X520-DA2.robot
37. Short_IPv6_Fib_200k_Intel-X520-DA2.robot
38. Short_IPv6_Fib_20k_Intel-X520-DA2.robot
39. Short_IPv6_Fib_2M_Intel-X520-DA2.robot
40. Short_IPv6_Intel-X520-DA2.robot
41. Short_IPv6_iAcl_Intel-X520-DA2.robot
42. Short_Lisp_Intel-X520-DA2.robot
43. Short_Xconnect_Dot1ad_Intel-X520-DA2.robot
44. Short_Xconnect_Dot1q_Intel-X520-DA2.robot
45. Short_Xconnect_Intel-X520-DA2.robot
46. Short_Xconnect_Vxlan_Ipv4_Intel-X520-DA2.robot

Performance TCs - Tagging for Performance Trending Graphs

In order to group performance TC results into performance trending graphs, CSIT is using RF tags. CSIT RF tags are described in `/docs/tag_documentation.rst'. RF tag changes related to performance TC naming change are captured as part of patch #3856.

Functional TCs - Naming Scheme

<to be added>

Functional TCs - New Names

<to be added>

Functional TCs - Old Names

<to be added>

Miscellaneous Notes

Alternative Perf TC Name Proposal - Deprecated

1. short_x520_dot1ad-l2xc.robot
2. short_x520_dot1q-l2xc.robot
3. short_x520_eth-ip4-base.robot
4. short_x520_eth-ip4-cop-whitelist.robot
5. short_x520_eth-ip4-fib-200k.robot
6. short_x520_eth-ip4-fib-20k.robot
7. short_x520_eth-ip4-fib-2M.robot
8. short_x520_eth-ip4-iacl-dst.robot
9. short_x520_eth-ip4-lisp-ip4.robot
10. short_x520_eth-ip4-lisp-ip6.robot
11. short_x520_eth-ip4-policer-marker.robot
12. short_x520_eth-ip4-vxlan-l2xc.robot
13. short_x520_eth-ip6-base.robot
14. short_x520_eth-ip6-cop-whitelist.robot
15. short_x520_eth-ip6-fib-200k.robot
16. short_x520_eth-ip6-fib-20k.robot
17. short_x520_eth-ip6-fib-2M.robot
18. short_x520_eth-ip6-iacl-dst.robot
19. short_x520_eth-ip6-lisp-ip4.robot
20. short_x520_eth-ip6-lisp-ip6.robot
21. short_x520_eth-l2bd-base.robot
22. short_x520_eth-l2xc-base.robot
23. x520_dot1ad-l2xc.robot
24. x520_dot1q-l2xc.robot
25. x520_eth-ip4-base-vhost.robot
26. x520_eth-ip4-base.robot
27. x520_eth-ip4-cop-whitelist.robot
28. x520_eth-ip4-fib-200k.robot
29. x520_eth-ip4-fib-20k.robot
30. x520_eth-ip4-fib-2M.robot
31. x520_eth-ip4-iacl-dst.robot
32. x520_eth-ip4-lisp-ip4.robot
33. x520_eth-ip4-lisp-ip6.robot
34. x520_eth-ip4-policer-marker.robot
35. x520_eth-ip4-vxlan-l2xc.robot
36. x520_eth-ip6-base.robot
37. x520_eth-ip6-cop-whitelist.robot
38. x520_eth-ip6-fib-200k.robot
39. x520_eth-ip6-fib-20k.robot
40. x520_eth-ip6-fib-2M.robot
41. x520_eth-ip6-iacl-dst.robot
42. x520_eth-ip6-lisp-ip4.robot
43. x520_eth-ip6-lisp-ip6.robot
44. x520_eth-l2bd-base-vhost.robot
45. x520_eth-l2bd-base.robot
46. x520_eth-l2xc-base-vhost.robot
47. x520_eth-l2xc-base.robot
48. x710_eth-l2bd-base.robot
49. xl710_eth-ip4-base.robot
50. xl710_eth-ip6-base.robot
51. xl710_eth-l2bd-base.robot
52. xl710_eth-l2xc-base.robot