Difference between revisions of "CSIT/PerfTestPlan"

From fd.io
Jump to: navigation, search
Line 8: Line 8:
 
#** TODO items
 
#** TODO items
 
#*** name - rename to "csit-vpp-performance-hw" or better "csit-vpp-perf-hw"
 
#*** name - rename to "csit-vpp-performance-hw" or better "csit-vpp-perf-hw"
−
#**** [IN REVIEW, pmikus] [https://gerrit.fd.io/r/#/c/786/ gerrit.fd.io change]
+
#**** [IN REVIEW, pmikus] [https://gerrit.fd.io/r/#/c/786/ gerrit]
 
#*** scope - currently only tb2, need to add other testbeds: tb1, tb3
 
#*** scope - currently only tb2, need to add other testbeds: tb1, tb3
 
#*** scope - tbX reservation script and tbX scheduler code already done and tested, needs to be used
 
#*** scope - tbX reservation script and tbX scheduler code already done and tested, needs to be used
Line 14: Line 14:
 
#** TODO items
 
#** TODO items
 
#*** rename to "verify-perf-all"
 
#*** rename to "verify-perf-all"
−
#**** [IN REVIEW, pmikus] [https://gerrit.fd.io/r/#/c/786/ gerrit.fd.io change]
+
#**** [IN REVIEW, pmikus] [https://gerrit.fd.io/r/#/c/786/ gerrit]
 
#*** consider adding later on "verify-perf-short", "verify-perf-long"
 
#*** consider adding later on "verify-perf-short", "verify-perf-long"
 
#* Status: WORKING
 
#* Status: WORKING
Line 28: Line 28:
 
==CSIT performance test cases - working==
 
==CSIT performance test cases - working==
  
−
# Working performance test cases - 18 working in FD.io physical testbeds today.
+
# Performance test cases - 30 working in FD.io physical testbeds today.
 +
#* hw-tb2 - physical testbed 2 is live.
 +
#* hw-tb1, hw-tb2 - physical testbeds 1 and 2 are being onboarded.
 
# Last manual run results reports:
 
# Last manual run results reports:
 
#* https://jenkins.fd.io/view/csit/job/csit-vpp-functional-hw-tb2/
 
#* https://jenkins.fd.io/view/csit/job/csit-vpp-functional-hw-tb2/
Line 34: Line 36:
 
#* https://jenkins.fd.io/view/vpp/job/vpp-verify-performance-short/
 
#* https://jenkins.fd.io/view/vpp/job/vpp-verify-performance-short/
 
# List of working performance test cases
 
# List of working performance test cases
−
## ~/csit$ grep "^| [a-zA-Z1-9]" -nr tests | grep -vi " | "  | sort | grep performance
+
## ~/csit$ grep "^| [a-zA-Z1-9]" -nr tests | grep -vi " | "  | sort | grep performance
−
## tests/suites/performance/long_bridge_domain.robot:24:| Find NDR by using linear search and 64B frames through bridge domain in 3-node topology
+
## tests/suites/performance/long_bridge_domain.robot:103:| Find NDR with 4 cores and rss 2 by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology
−
##* // TODO items: rename to `Find NDR by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology`
+
## tests/suites/performance/long_bridge_domain.robot:25:| Find NDR by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology
−
## tests/suites/performance/long_bridge_domain.robot:34:| Find NDR by using linear search and 1518B frames through bridge domain in 3-node topology
+
## tests/suites/performance/long_bridge_domain.robot:44:| Find NDR by using RFC2544 linear search and 1518B frames through bridge domain in 3-node topology
−
## tests/suites/performance/long_bridge_domain.robot:44:| Find NDR by using linear search and 9000B frames through bridge domain in 3-node topology
+
## tests/suites/performance/long_bridge_domain.robot:64:| Find NDR by using RFC2544 linear search and 9000B frames through bridge domain in 3-node topology
−
## tests/suites/performance/long_ipv4.robot:28:| Find NDR by using linear search and 64B frames through IPv4 forwarding in 3-node topology
+
## tests/suites/performance/long_bridge_domain.robot:84:| Find NDR with 2 cores and rss 1 by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology
−
## tests/suites/performance/long_ipv4.robot:38:| Find NDR by using linear search and 1518B frames through IPv4 forwarding in 3-node topology
+
## tests/suites/performance/long_ipv4.robot:107:| Find NDR with 4 cores and rss 2 by using RFC2544 linear search and 64B frames through IPv4 forwarding in 3-node topology
−
## tests/suites/performance/long_ipv4.robot:48:| Find NDR by using linear search and 9000B frames through IPv4 forwarding in 3-node topology
+
## tests/suites/performance/long_ipv4.robot:29:| Find NDR by using RFC2544 linear search and 64B frames through IPv4 forwarding in 3-node topology
−
## tests/suites/performance/long_xconnect.robot:25:| Find NDR by using linear search and 64B frames through L2 cross connect in 3-node topology
+
## tests/suites/performance/long_ipv4.robot:48:| Find NDR by using RFC2544 linear search and 1518B frames through IPv4 forwarding in 3-node topology
−
## tests/suites/performance/long_xconnect.robot:35:| Find NDR by using linear search and 1518B frames through L2 cross connect in 3-node topology
+
## tests/suites/performance/long_ipv4.robot:68:| Find NDR by using RFC2544 linear search and 9000B frames through IPv4 forwarding in 3-node topology
−
## tests/suites/performance/long_xconnect.robot:45:| Find NDR by using linear search and 9000B frames through L2 cross connect in 3-node topology
+
## tests/suites/performance/long_ipv4.robot:88:| Find NDR with 2 cores and rss 1 by using RFC2544 linear search and 64B frames through IPv4 forwarding in 3-node topology
−
## tests/suites/performance/short_bridge_domain.robot:23:| 1core VPP passes 64B frames through bridge domain at 3.5mpps in 3-node topology
+
## tests/suites/performance/long_xconnect.robot:107:| Find NDR with 4 cores and rss 2 by using RFC2544 linear search and 64B frames through L2 cross connect in 3-node topology
−
## tests/suites/performance/short_bridge_domain.robot:30:| 1core VPP passes 1518B frames through bridge domain at 10gbps in 3-node topology
+
## tests/suites/performance/long_xconnect.robot:32:| Find NDR by using RFC2544 linear search and 64B frames through L2 cross connect in 3-node topology
−
## tests/suites/performance/short_bridge_domain.robot:37:| 1core VPP passes 9000B frames through bridge domain at 10gbps in 3-node topology
+
## tests/suites/performance/long_xconnect.robot:50:| Find NDR by using RFC2544 linear search and 1518B frames through L2 cross connect in 3-node topology
−
## tests/suites/performance/short_ipv4.robot:27:| 1core VPP passes 64B frames through IPv4 forwarding at 3.5mpps in 3-node topology
+
## tests/suites/performance/long_xconnect.robot:69:| Find NDR by using RFC2544 linear search and 9000B frames through L2 cross connect in 3-node topology
−
##* // TODO items: rename to `1core VPP passes 64B frames through IPv4 forwarding at 2x 3.5Mpps in 3-node topology`
+
## tests/suites/performance/long_xconnect.robot:88:| Find NDR with 2 cores and rss 1 by using RFC2544 linear search and 64B frames through L2 cross connect in 3-node topology
−
##* // TODO items: currently testing all functional combinations takes 10-to-15 minutes; going forward would need to split the functional combinations
+
## tests/suites/performance/short_bridge_domain.robot:25:| 1core VPP passes 64B frames through bridge domain at 2x 3.5Mpps in 3-node topology
−
## tests/suites/performance/short_ipv4.robot:34:| 1core VPP passes 1518B frames through IPv4 forwarding at 10gbps in 3-node topology
+
## tests/suites/performance/short_bridge_domain.robot:41:| 1core VPP passes 1518B frames through bridge domain at 2x 10Gbps in 3-node topology
−
## tests/suites/performance/short_ipv4.robot:41:| 1core VPP passes 9000B frames through IPv4 forwarding at 10gbps in 3-node topology
+
## tests/suites/performance/short_bridge_domain.robot:57:| 1core VPP passes 9000B frames through bridge domain at 2x 10Gbps in 3-node topology
−
## tests/suites/performance/short_xconnect.robot:24:| 1core VPP passes 64B frames through L2 cross connect at 3.5mpps in 3-node topology
+
## tests/suites/performance/short_bridge_domain.robot:73:| 2core VPP with rss 1 passes 64B frames through bridge domain at 2x 7.6Mpps in 3-node topology
−
## tests/suites/performance/short_xconnect.robot:31:| 1core VPP passes 1518B frames through L2 cross connect at 10gbps in 3-node topology
+
## tests/suites/performance/short_bridge_domain.robot:89:| 4core VPP with rss 2 passes 64B frames through bridge domain at 2x 8.6Mpps in 3-node topology
−
## tests/suites/performance/short_xconnect.robot:38:| 1core VPP passes 9000B frames through L2 cross connect at 10gbps in 3-node topology
+
## tests/suites/performance/short_ipv4.robot:29:| 1core VPP passes 64B frames through IPv4 forwarding at 2x 3.5mpps in 3-node topology
 +
## tests/suites/performance/short_ipv4.robot:45:| 1core VPP passes 1518B frames through IPv4 forwarding at 2x 10gbps in 3-node topology
 +
## tests/suites/performance/short_ipv4.robot:61:| 1core VPP passes 9000B frames through IPv4 forwarding at 2x 10gbps in 3-node topology
 +
## tests/suites/performance/short_ipv4.robot:77:| 2core VPP with rss 1 passes 64B frames through IPv4 forwarding at 2x 8.2mpps in 3-node topology
 +
## tests/suites/performance/short_ipv4.robot:93:| 4core VPP with rss 2 passes 64B frames through IPv4 forwarding at 2x 9.2mpps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:26:| 1core VPP passes 64B frames through L2 cross connect at 2x 3.5mpps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:42:| 1core VPP passes 1518B frames through L2 cross connect at 2x 10gbps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:58:| 1core VPP passes 9000B frames through L2 cross connect at 2x 10gbps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:74:| 2core VPP with rss 1 passes 64B frames through L2 cross connect at 2x 11.0mpps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:90:| 4core VPP with rss 2 passes 64B frames through L2 cross connect at 2x 11.6mpps in 3-node topology
  
 
==CSIT performance test cases - plan and status==
 
==CSIT performance test cases - plan and status==
  
−
# [DONE, mmiklus] DropRateSearch
+
For CSIT project to work and be useful, it is proposed that CSIT code and test cases development should be done in coordination with VPP Release Candidate content and release schedule.
−
#* [DONE, pmikus] binary search [https://gerrit.fd.io/r/#/c/564/ gerrit.fd.io change]
+
CSIT development work needs to be prioritized based on FD.io community needs and feedback, and csit-dev work distributed. To facilitate this, we compiled an initial list of proposed performance test cases that must, should, could be tested. Each csit-dev work item is prefixed with status and/or proposed priority relative to FD.io VPP release:
−
#* [DONE, pmikus] BESTofN/WORSTofN [https://gerrit.fd.io/r/#/c/603/ gerrit.fd.io change]
+
* [Pn-Rm] – Proposed coverage of VPP functionality by FD.io CSIT functional and performance test cases.
−
#* [DONE, pmikus] combined search [https://gerrit.fd.io/r/#/c/671/ gerrit.fd.io change]
+
** '''P'''n – CSIT development and integration '''priority'''; n=0,1,2, 0-for-MUST, 1-for-SHOULD, 2-for-COULD.
−
# [WIP, mmiklus] extend topology compute library with NIC type filtering
+
** '''R'''m – VPP major '''release''', R1 stands for the first FD.io VPP release, R2 for the 2nd FD.io VPP release.
−
# [IN REVIEW, pmikus] multicore tests [https://gerrit.fd.io/r/#/c/710/ gerrit.fd.io change]
+
* [DONE-Rm] - Done and working test cases, executed for VPP Rm code.
−
# [TBD] IPv6
+
* [TBD] - Development work scheduling priority to-be-defined.
−
# [TBD] Large-scale FIB tests
+
 
−
#* IPv4
+
Current list of identified csit-dev work items:
−
#* IPv6
+
 
−
#* mac
+
# Long throughput benchmarks - libraries
−
#* Various prefix lengths
+
## [DONE-R1] IPv4 NonDropRate search
−
# [TBD] Encapsulations
+
##* [DONE-R1] linear search [rfc2544] [gerrit-ref]
−
#* VxLAN
+
##* [DONE-R1] binary search [rfc2544] [https://gerrit.fd.io/r/#/c/564/ gerrit]
−
#* VLAN
+
##* [DONE-R1] BESTofN/WORSTofN search [https://gerrit.fd.io/r/#/c/603/ gerrit]
−
#* GRE
+
##* [DONE-R1] combined binary+linear search [https://gerrit.fd.io/r/#/c/671/ gerrit]
−
#* IPSEC
+
## [DONE-R1] L2 NonDropRate search [gerrit-ref]
−
# [TBD] Create topology files for tb1, tb3
+
## [P0-R1] IPv6 NonDropRate search [gerrit-ref]
−
# [TBD] Host, driver, VPP baseline performance
+
# Long throughput benchmarks - test cases
−
#* [TBD] User-mode app (L3FWD-xconnect) with DPDK driver per NIC
+
## [DONE-R1] IPv4 baseline NonDropRate search
−
#* [TBD] User-mode VPP baseline config with the same DPDK driver per NIC
+
##* [DONE-R1] linear search [rfc2544] [gerrit-ref]
−
# [TBD] vNet VPP topologies
+
##* [DONE-R1] binary search [rfc2544] [gerrit-ref]
−
#* [TBD] Phy1-VPP-Phy2
+
##* [DONE-R1] BESTofN/WORSTofN search [gerrit-ref]
−
#* [TBD] Phy1-VPP-VM-VPP-Phy2
+
##* [DONE-R1] combined binary+linear search [gerrit-ref]
−
# [TBD] T-REX VXLAN traffic profile & CSIT tests
+
## [DONE-R1] L2 NonDropRate search
−
# IxNetwork driver
+
##* [DONE-R1] linear search [rfc2544] [gerrit-ref]
−
#* [TBD] driver design
+
##* [DONE-R1] binary search [rfc2544] [gerrit-ref]
−
#* [TBD] driver implementation
+
##* [DONE-R1] BESTofN/WORSTofN search [gerrit-ref]
−
#** setup, teardown
+
##* [DONE-R1] combined binary+linear search [gerrit-ref]
−
#** traffic stream configuration
+
## [P0-R1] IPv6 NonDropRate search [gerrit-ref]
−
#** statistics retrieval
+
##* [P0-R1] linear search [rfc2544] [gerrit-ref]
−
#* [TBD] latency measurements
+
##* [P0-R1] binary search [rfc2544] [gerrit-ref]
−
#** feasibility
+
##* [P0-R1] BESTofN/WORSTofN search [gerrit-ref]
−
#** implementation
+
##* [P0-R1] combined binary+linear search [gerrit-ref]
−
#* [TBD] integration with CSIT tests
+
# Short throughput benchmarks - libraries
−
# Moongen driver
+
## [DONE-R1] short IPv4 throughput test against reference NDR [gerrit-ref]
−
#* [TBD] verify requirements and available HW against latest MoonGen codebase
+
## [DONE-R1] short L2 throughput test against reference NDR [gerrit-ref]
−
#* [TBD] integration
+
## [P0-R1] short IPv6 throughput test against reference NDR [gerrit-ref]
−
# T-REX
+
# Short throughput benchmarks - test cases
−
#* [TBD] latency (T-REX support to be confirmed)
+
## [DONE-R1] short IPv4 throughput - routed-forwarding [gerrit-ref]
 +
## [DONE-R1] short L2 throughput - bridge-domain, l2-xconnect [gerrit-ref]
 +
## [P0-R1] short IPv6 throughput - routed-forwarding [gerrit-ref]
 +
# 3-node physical testbed - libraries
 +
#* [DONE-R1] Create host topology files for tb2
 +
#* [P0-R1] Extend host topology with NIC type filtering
 +
#* [P0-R1] Create host topology files for tb1, tb3
 +
# Multi-core multi-thread tests - libraries
 +
#* [DONE-R1] multicore tests [https://gerrit.fd.io/r/#/c/710/ gerrit]
 +
# Large-scale FIB tests - libraries
 +
#* [P1-R1] IPv4 same prefix length
 +
#* [P2-R1] IPv4 varying prefix lengths
 +
#* [P1-R1] IPv6 same prefix length
 +
#* [P2-R1] IPv6 varying prefix lengths
 +
#* [P1-R1] MAC addresses
 +
# Packet encapsulations - test cases
 +
## IP4 encapsulations
 +
## [P1-R1] ip4-gre-ip4
 +
## [P1-R2] ip4-ipsec-ip4
 +
# IP6 encapsulations
 +
## [P1-R1] ip4-o-ip6 softwire
 +
##* [P1-R1] lightweight46 [rfc7596]
 +
##* [P2-R1] map-e [rfc7597]
 +
##* [P2-R1] map-t [rfc7599]
 +
## [TBD] ip6-segment-routing
 +
## [P1-R2] ip6-ipsec-ip6
 +
# L2 encapsulations
 +
## [P0-R1] l2-vlan-dot1q
 +
## [P2-R1] l2-vlan-dot1ad
 +
## [P1-R1] l2-vxlan-ipv4
 +
## [TBD] l2-eth-l2tpv3-ip4
 +
## [TBD] l2-eth-l2tpv3-ip6
 +
## [TBD] l2-eth-gre-ip4
 +
## [TBD] l2-eth-gre-ip6
 +
# VPP vNet topology baselines - libraries
 +
## [TBD] Host stack baseline - Phy1-App-Phy2
 +
##* [TBD] Reference host user-mode app, L3FWD-xconnect, DPDK driver per NIC type
 +
## [TBD] Host stack VPP baseline - Phy1-VPP-Phy2
 +
##* [TBD] Host VPP IPv4 baseline config, DPDK driver per NIC type
 +
## [TBD] Guest stack baseline - Phy1-pcipt-AppVM-pcipt-Phy2
 +
##* [TBD] Reference guest user-mode app, L3FWD-xconnect, DPDK driver per NIC type
 +
## [TBD] Guest stack VPP baseline - Phy1-pcipt-VPPVM-pcipt-Phy2
 +
##* [TBD] VPP IPv4 baseline config, DPDK driver per NIC type
 +
## [TBD] Host and guest stack baseline - Phy1-VPP-AppVM-VPP-Phy2
 +
##* [TBD] VPP IPv4 baseline config, vhost-user to AppVM
 +
# vNet VPP topologies - libraries
 +
#* [DONE-R1] Phy1-VPP-Phy2
 +
#* [P0-R1] Phy1-VPP-VM-VPP-Phy2
 +
# Traffic generator drivers
 +
## [DONE-R1] T-Rex
 +
##* [DONE-R1] T-Rex driver
 +
##* [TBD] TRex support for packet delay and packet delay variation (T-REX support to be confirmed)
 +
## [TBD] IXIA IxNetwork driver
 +
##* [TBD] driver design
 +
##* [TBD] driver implementation
 +
##** [TBD] test case setup, teardown
 +
##** [TBD] traffic stream configuration
 +
##** [TBD] statistics retrieval
 +
##* [TBD] latency measurements
 +
##** [TBD] feasibility
 +
##** [TBD] implementation
 +
##* [TBD] integration with CSIT tests
 +
## [TBD] Moongen driver
 +
##* [TBD] verify requirements and available HW against latest MoonGen codebase
 +
##* [TBD] CSIT integration
 
# Other work items
 
# Other work items
 
#* [TBD] integrate [https://pypi.python.org/pypi/sysinfosuite Python sysinfosuite] to CSIT
 
#* [TBD] integrate [https://pypi.python.org/pypi/sysinfosuite Python sysinfosuite] to CSIT
 
#* [TBD] unified performance trend dashboard
 
#* [TBD] unified performance trend dashboard

Revision as of 17:14, 18 April 2016

FD.io IT systems integration

Three Jenkins jobs for CSIT performance test cases are in place today:

  1. Job name: csit-vpp-functional-hw-tb2
    • Description: This job is testing csit project against performance test cases
      • TODO items
        • name - rename to "csit-vpp-performance-hw" or better "csit-vpp-perf-hw"
        • scope - currently only tb2, need to add other testbeds: tb1, tb3
        • scope - tbX reservation script and tbX scheduler code already done and tested, needs to be used
    • Trigger: Comment "hw-perf-check" in proposed patch in CSIT project
      • TODO items
        • rename to "verify-perf-all"
        • consider adding later on "verify-perf-short", "verify-perf-long"
    • Status: WORKING
  2. Job name: vpp-verify-performance-long
    • Description: This job is testing vpp proposed patch against long performance test cases
    • Trigger: Comment "verify-perf-long" in proposed patch in VPP project
    • Status: WORKING
  3. Job name: vpp-verify-performance-short
    • Description: This job is testing vpp proposed patch against short performance test cases
    • Trigger: Comment "verify-perf-short" in proposed patch in VPP project
    • Status: WORKING

CSIT performance test cases - working

  1. Performance test cases - 30 working in FD.io physical testbeds today.
    • hw-tb2 - physical testbed 2 is live.
    • hw-tb1, hw-tb2 - physical testbeds 1 and 2 are being onboarded.
  2. Last manual run results reports:
  3. List of working performance test cases
    1. ~/csit$ grep "^| [a-zA-Z1-9]" -nr tests | grep -vi " | " | sort | grep performance
    2. tests/suites/performance/long_bridge_domain.robot:103:| Find NDR with 4 cores and rss 2 by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology
    3. tests/suites/performance/long_bridge_domain.robot:25:| Find NDR by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology
    4. tests/suites/performance/long_bridge_domain.robot:44:| Find NDR by using RFC2544 linear search and 1518B frames through bridge domain in 3-node topology
    5. tests/suites/performance/long_bridge_domain.robot:64:| Find NDR by using RFC2544 linear search and 9000B frames through bridge domain in 3-node topology
    6. tests/suites/performance/long_bridge_domain.robot:84:| Find NDR with 2 cores and rss 1 by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology
    7. tests/suites/performance/long_ipv4.robot:107:| Find NDR with 4 cores and rss 2 by using RFC2544 linear search and 64B frames through IPv4 forwarding in 3-node topology
    8. tests/suites/performance/long_ipv4.robot:29:| Find NDR by using RFC2544 linear search and 64B frames through IPv4 forwarding in 3-node topology
    9. tests/suites/performance/long_ipv4.robot:48:| Find NDR by using RFC2544 linear search and 1518B frames through IPv4 forwarding in 3-node topology
    10. tests/suites/performance/long_ipv4.robot:68:| Find NDR by using RFC2544 linear search and 9000B frames through IPv4 forwarding in 3-node topology
    11. tests/suites/performance/long_ipv4.robot:88:| Find NDR with 2 cores and rss 1 by using RFC2544 linear search and 64B frames through IPv4 forwarding in 3-node topology
    12. tests/suites/performance/long_xconnect.robot:107:| Find NDR with 4 cores and rss 2 by using RFC2544 linear search and 64B frames through L2 cross connect in 3-node topology
    13. tests/suites/performance/long_xconnect.robot:32:| Find NDR by using RFC2544 linear search and 64B frames through L2 cross connect in 3-node topology
    14. tests/suites/performance/long_xconnect.robot:50:| Find NDR by using RFC2544 linear search and 1518B frames through L2 cross connect in 3-node topology
    15. tests/suites/performance/long_xconnect.robot:69:| Find NDR by using RFC2544 linear search and 9000B frames through L2 cross connect in 3-node topology
    16. tests/suites/performance/long_xconnect.robot:88:| Find NDR with 2 cores and rss 1 by using RFC2544 linear search and 64B frames through L2 cross connect in 3-node topology
    17. tests/suites/performance/short_bridge_domain.robot:25:| 1core VPP passes 64B frames through bridge domain at 2x 3.5Mpps in 3-node topology
    18. tests/suites/performance/short_bridge_domain.robot:41:| 1core VPP passes 1518B frames through bridge domain at 2x 10Gbps in 3-node topology
    19. tests/suites/performance/short_bridge_domain.robot:57:| 1core VPP passes 9000B frames through bridge domain at 2x 10Gbps in 3-node topology
    20. tests/suites/performance/short_bridge_domain.robot:73:| 2core VPP with rss 1 passes 64B frames through bridge domain at 2x 7.6Mpps in 3-node topology
    21. tests/suites/performance/short_bridge_domain.robot:89:| 4core VPP with rss 2 passes 64B frames through bridge domain at 2x 8.6Mpps in 3-node topology
    22. tests/suites/performance/short_ipv4.robot:29:| 1core VPP passes 64B frames through IPv4 forwarding at 2x 3.5mpps in 3-node topology
    23. tests/suites/performance/short_ipv4.robot:45:| 1core VPP passes 1518B frames through IPv4 forwarding at 2x 10gbps in 3-node topology
    24. tests/suites/performance/short_ipv4.robot:61:| 1core VPP passes 9000B frames through IPv4 forwarding at 2x 10gbps in 3-node topology
    25. tests/suites/performance/short_ipv4.robot:77:| 2core VPP with rss 1 passes 64B frames through IPv4 forwarding at 2x 8.2mpps in 3-node topology
    26. tests/suites/performance/short_ipv4.robot:93:| 4core VPP with rss 2 passes 64B frames through IPv4 forwarding at 2x 9.2mpps in 3-node topology
    27. tests/suites/performance/short_xconnect.robot:26:| 1core VPP passes 64B frames through L2 cross connect at 2x 3.5mpps in 3-node topology
    28. tests/suites/performance/short_xconnect.robot:42:| 1core VPP passes 1518B frames through L2 cross connect at 2x 10gbps in 3-node topology
    29. tests/suites/performance/short_xconnect.robot:58:| 1core VPP passes 9000B frames through L2 cross connect at 2x 10gbps in 3-node topology
    30. tests/suites/performance/short_xconnect.robot:74:| 2core VPP with rss 1 passes 64B frames through L2 cross connect at 2x 11.0mpps in 3-node topology
    31. tests/suites/performance/short_xconnect.robot:90:| 4core VPP with rss 2 passes 64B frames through L2 cross connect at 2x 11.6mpps in 3-node topology

CSIT performance test cases - plan and status

For CSIT project to work and be useful, it is proposed that CSIT code and test cases development should be done in coordination with VPP Release Candidate content and release schedule. CSIT development work needs to be prioritized based on FD.io community needs and feedback, and csit-dev work distributed. To facilitate this, we compiled an initial list of proposed performance test cases that must, should, could be tested. Each csit-dev work item is prefixed with status and/or proposed priority relative to FD.io VPP release:

  • [Pn-Rm] – Proposed coverage of VPP functionality by FD.io CSIT functional and performance test cases.
    • Pn – CSIT development and integration priority; n=0,1,2, 0-for-MUST, 1-for-SHOULD, 2-for-COULD.
    • Rm – VPP major release, R1 stands for the first FD.io VPP release, R2 for the 2nd FD.io VPP release.
  • [DONE-Rm] - Done and working test cases, executed for VPP Rm code.
  • [TBD] - Development work scheduling priority to-be-defined.

Current list of identified csit-dev work items:

  1. Long throughput benchmarks - libraries
    1. [DONE-R1] IPv4 NonDropRate search
      • [DONE-R1] linear search [rfc2544] [gerrit-ref]
      • [DONE-R1] binary search [rfc2544] gerrit
      • [DONE-R1] BESTofN/WORSTofN search gerrit
      • [DONE-R1] combined binary+linear search gerrit
    2. [DONE-R1] L2 NonDropRate search [gerrit-ref]
    3. [P0-R1] IPv6 NonDropRate search [gerrit-ref]
  2. Long throughput benchmarks - test cases
    1. [DONE-R1] IPv4 baseline NonDropRate search
      • [DONE-R1] linear search [rfc2544] [gerrit-ref]
      • [DONE-R1] binary search [rfc2544] [gerrit-ref]
      • [DONE-R1] BESTofN/WORSTofN search [gerrit-ref]
      • [DONE-R1] combined binary+linear search [gerrit-ref]
    2. [DONE-R1] L2 NonDropRate search
      • [DONE-R1] linear search [rfc2544] [gerrit-ref]
      • [DONE-R1] binary search [rfc2544] [gerrit-ref]
      • [DONE-R1] BESTofN/WORSTofN search [gerrit-ref]
      • [DONE-R1] combined binary+linear search [gerrit-ref]
    3. [P0-R1] IPv6 NonDropRate search [gerrit-ref]
      • [P0-R1] linear search [rfc2544] [gerrit-ref]
      • [P0-R1] binary search [rfc2544] [gerrit-ref]
      • [P0-R1] BESTofN/WORSTofN search [gerrit-ref]
      • [P0-R1] combined binary+linear search [gerrit-ref]
  3. Short throughput benchmarks - libraries
    1. [DONE-R1] short IPv4 throughput test against reference NDR [gerrit-ref]
    2. [DONE-R1] short L2 throughput test against reference NDR [gerrit-ref]
    3. [P0-R1] short IPv6 throughput test against reference NDR [gerrit-ref]
  4. Short throughput benchmarks - test cases
    1. [DONE-R1] short IPv4 throughput - routed-forwarding [gerrit-ref]
    2. [DONE-R1] short L2 throughput - bridge-domain, l2-xconnect [gerrit-ref]
    3. [P0-R1] short IPv6 throughput - routed-forwarding [gerrit-ref]
  5. 3-node physical testbed - libraries
    • [DONE-R1] Create host topology files for tb2
    • [P0-R1] Extend host topology with NIC type filtering
    • [P0-R1] Create host topology files for tb1, tb3
  6. Multi-core multi-thread tests - libraries
    • [DONE-R1] multicore tests gerrit
  7. Large-scale FIB tests - libraries
    • [P1-R1] IPv4 same prefix length
    • [P2-R1] IPv4 varying prefix lengths
    • [P1-R1] IPv6 same prefix length
    • [P2-R1] IPv6 varying prefix lengths
    • [P1-R1] MAC addresses
  8. Packet encapsulations - test cases
    1. IP4 encapsulations
    2. [P1-R1] ip4-gre-ip4
    3. [P1-R2] ip4-ipsec-ip4
  9. IP6 encapsulations
    1. [P1-R1] ip4-o-ip6 softwire
      • [P1-R1] lightweight46 [rfc7596]
      • [P2-R1] map-e [rfc7597]
      • [P2-R1] map-t [rfc7599]
    2. [TBD] ip6-segment-routing
    3. [P1-R2] ip6-ipsec-ip6
  10. L2 encapsulations
    1. [P0-R1] l2-vlan-dot1q
    2. [P2-R1] l2-vlan-dot1ad
    3. [P1-R1] l2-vxlan-ipv4
    4. [TBD] l2-eth-l2tpv3-ip4
    5. [TBD] l2-eth-l2tpv3-ip6
    6. [TBD] l2-eth-gre-ip4
    7. [TBD] l2-eth-gre-ip6
  11. VPP vNet topology baselines - libraries
    1. [TBD] Host stack baseline - Phy1-App-Phy2
      • [TBD] Reference host user-mode app, L3FWD-xconnect, DPDK driver per NIC type
    2. [TBD] Host stack VPP baseline - Phy1-VPP-Phy2
      • [TBD] Host VPP IPv4 baseline config, DPDK driver per NIC type
    3. [TBD] Guest stack baseline - Phy1-pcipt-AppVM-pcipt-Phy2
      • [TBD] Reference guest user-mode app, L3FWD-xconnect, DPDK driver per NIC type
    4. [TBD] Guest stack VPP baseline - Phy1-pcipt-VPPVM-pcipt-Phy2
      • [TBD] VPP IPv4 baseline config, DPDK driver per NIC type
    5. [TBD] Host and guest stack baseline - Phy1-VPP-AppVM-VPP-Phy2
      • [TBD] VPP IPv4 baseline config, vhost-user to AppVM
  12. vNet VPP topologies - libraries
    • [DONE-R1] Phy1-VPP-Phy2
    • [P0-R1] Phy1-VPP-VM-VPP-Phy2
  13. Traffic generator drivers
    1. [DONE-R1] T-Rex
      • [DONE-R1] T-Rex driver
      • [TBD] TRex support for packet delay and packet delay variation (T-REX support to be confirmed)
    2. [TBD] IXIA IxNetwork driver
      • [TBD] driver design
      • [TBD] driver implementation
        • [TBD] test case setup, teardown
        • [TBD] traffic stream configuration
        • [TBD] statistics retrieval
      • [TBD] latency measurements
        • [TBD] feasibility
        • [TBD] implementation
      • [TBD] integration with CSIT tests
    3. [TBD] Moongen driver
      • [TBD] verify requirements and available HW against latest MoonGen codebase
      • [TBD] CSIT integration
  14. Other work items
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/PerfTestPlan&oldid=889"