Difference between revisions of "CSIT/PerfTestPlan"

From fd.io
Jump to: navigation, search
(Created page with "==FD.io IT systems integration== Three Jenkins jobs for CSIT performance test cases are in place today: # Job name: '''csit-vpp-functional-hw-tb2''' #* Description: This job...")
 
Line 5: Line 5:
 
# Job name: '''csit-vpp-functional-hw-tb2'''
 
# Job name: '''csit-vpp-functional-hw-tb2'''
 
#* Description: This job is testing csit project against performance test cases
 
#* Description: This job is testing csit project against performance test cases
−
#** Work-In-Progress (WIP) items
+
#** TODO items
 
#*** name - rename to "csit-vpp-performance-hw" or better "csit-vpp-perf-hw"
 
#*** name - rename to "csit-vpp-performance-hw" or better "csit-vpp-perf-hw"
 
#*** scope - currently only tb2, need to add other testbeds: tb1, tb3
 
#*** scope - currently only tb2, need to add other testbeds: tb1, tb3
 
#*** scope - tbX reservation script and tbX scheduler code already done and tested, needs to be used
 
#*** scope - tbX reservation script and tbX scheduler code already done and tested, needs to be used
 
#* Trigger: Comment "hw-perf-check" in proposed patch in CSIT project
 
#* Trigger: Comment "hw-perf-check" in proposed patch in CSIT project
−
#** WIP items
+
#** TODO items
 
#*** rename to "verify-perf-all"
 
#*** rename to "verify-perf-all"
 
#*** consider adding later on "verify-perf-short", "verify-perf-long"
 
#*** consider adding later on "verify-perf-short", "verify-perf-long"
Line 22: Line 22:
 
#* Trigger: Comment "verify-perf-short" in proposed patch in VPP project
 
#* Trigger: Comment "verify-perf-short" in proposed patch in VPP project
 
#* Status: WORKING
 
#* Status: WORKING
 +
 +
==CSIT performance test cases - working==
 +
 +
# Working performance test cases - 18 working in FD.io physical testbeds today.
 +
# Last manual run results reports:
 +
#* https://jenkins.fd.io/view/csit/job/csit-vpp-functional-hw-tb2/
 +
#* https://jenkins.fd.io/view/vpp/job/vpp-verify-performance-long/
 +
#* https://jenkins.fd.io/view/vpp/job/vpp-verify-performance-short/
 +
# List of working performance test cases
 +
## ~/csit$  grep "^| [a-zA-Z1-9]" -nr tests | grep -vi " | "  | sort | grep performance
 +
## tests/suites/performance/long_bridge_domain.robot:24:| Find NDR by using linear search and 64B frames through bridge domain in 3-node topology
 +
##* // TODO items: rename to `Find NDR by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology`
 +
## tests/suites/performance/long_bridge_domain.robot:34:| Find NDR by using linear search and 1518B frames through bridge domain in 3-node topology
 +
## tests/suites/performance/long_bridge_domain.robot:44:| Find NDR by using linear search and 9000B frames through bridge domain in 3-node topology
 +
## tests/suites/performance/long_ipv4.robot:28:| Find NDR by using linear search and 64B frames through IPv4 forwarding in 3-node topology
 +
## tests/suites/performance/long_ipv4.robot:38:| Find NDR by using linear search and 1518B frames through IPv4 forwarding in 3-node topology
 +
## tests/suites/performance/long_ipv4.robot:48:| Find NDR by using linear search and 9000B frames through IPv4 forwarding in 3-node topology
 +
## tests/suites/performance/long_xconnect.robot:25:| Find NDR by using linear search and 64B frames through L2 cross connect in 3-node topology
 +
## tests/suites/performance/long_xconnect.robot:35:| Find NDR by using linear search and 1518B frames through L2 cross connect in 3-node topology
 +
## tests/suites/performance/long_xconnect.robot:45:| Find NDR by using linear search and 9000B frames through L2 cross connect in 3-node topology
 +
## tests/suites/performance/short_bridge_domain.robot:23:| 1core VPP passes 64B frames through bridge domain at 3.5mpps in 3-node topology
 +
## tests/suites/performance/short_bridge_domain.robot:30:| 1core VPP passes 1518B frames through bridge domain at 10gbps in 3-node topology
 +
## tests/suites/performance/short_bridge_domain.robot:37:| 1core VPP passes 9000B frames through bridge domain at 10gbps in 3-node topology
 +
## tests/suites/performance/short_ipv4.robot:27:| 1core VPP passes 64B frames through IPv4 forwarding at 3.5mpps in 3-node topology
 +
##* // TODO items: rename to `1core VPP passes 64B frames through IPv4 forwarding at 2x 3.5Mpps in 3-node topology`
 +
##* // TODO items: currently testing all functional combinations takes 10-to-15 minutes; going forward would need to split the functional combinations
 +
## tests/suites/performance/short_ipv4.robot:34:| 1core VPP passes 1518B frames through IPv4 forwarding at 10gbps in 3-node topology
 +
## tests/suites/performance/short_ipv4.robot:41:| 1core VPP passes 9000B frames through IPv4 forwarding at 10gbps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:24:| 1core VPP passes 64B frames through L2 cross connect at 3.5mpps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:31:| 1core VPP passes 1518B frames through L2 cross connect at 10gbps in 3-node topology
 +
## tests/suites/performance/short_xconnect.robot:38:| 1core VPP passes 9000B frames through L2 cross connect at 10gbps in 3-node topology
 +
 +
==CSIT performance test cases - plan and status==
 +
 +
# [DONE, mmiklus] DropRateSearch
 +
#* [DONE, pmikus] binary search [https://gerrit.fd.io/r/#/c/564/ gerrit.fd.io change]
 +
#* [DONE, pmikus] BESTofN/WORSTofN [https://gerrit.fd.io/r/#/c/603/ gerrit.fd.io change]
 +
#* [IN REVIEW, pmikus] combined search [https://gerrit.fd.io/r/#/c/671/ gerrit.fd.io change]
 +
# [WIP, mmiklus] extend topology compute library with NIC type filtering
 +
# [WIP, pmikus] multicore tests [https://gerrit.fd.io/r/#/c/710/ gerrit.fd.io change]
 +
# [TBD] IPv6
 +
# [TBD] Large-scale FIB tests
 +
#* IPv4
 +
#* IPv6
 +
#* mac
 +
#* Various prefix lengths
 +
# [TBD] Encapsulations
 +
#* VxLAN
 +
#* VLAN
 +
#* GRE
 +
#* IPSEC
 +
# [TBD] Create topology files for tb1, tb3
 +
# [TBD] T-REX VXLAN traffic profile & CSIT tests
 +
# IxNetwork driver
 +
#* [TBD] driver design
 +
#* [TBD] driver implementation
 +
#** setup, teardown
 +
#** traffic stream configuration
 +
#** statistics retrieval
 +
#* [TBD] latency measurements
 +
#** feasibility
 +
#** implementation
 +
#* [TBD] integration with CSIT tests
 +
# Moongen driver
 +
#* [TBD] verify requirements and available HW against latest MoonGen codebase
 +
#* [TBD] integration
 +
# T-REX
 +
#* [TBD] latency (T-REX support to be confirmed)
 +
# Other work items
 +
#* [TBD] integrate [https://pypi.python.org/pypi/sysinfosuite Python sysinfosuite] to CSIT
 +
#* [TBD] unified performance trend dashboard
 +
 +
 +
---
 +
end

Revision as of 13:06, 13 April 2016

FD.io IT systems integration

Three Jenkins jobs for CSIT performance test cases are in place today:

  1. Job name: csit-vpp-functional-hw-tb2
    • Description: This job is testing csit project against performance test cases
      • TODO items
        • name - rename to "csit-vpp-performance-hw" or better "csit-vpp-perf-hw"
        • scope - currently only tb2, need to add other testbeds: tb1, tb3
        • scope - tbX reservation script and tbX scheduler code already done and tested, needs to be used
    • Trigger: Comment "hw-perf-check" in proposed patch in CSIT project
      • TODO items
        • rename to "verify-perf-all"
        • consider adding later on "verify-perf-short", "verify-perf-long"
    • Status: WORKING
  2. Job name: vpp-verify-performance-long
    • Description: This job is testing vpp proposed patch against long performance test cases
    • Trigger: Comment "verify-perf-long" in proposed patch in VPP project
    • Status: WORKING
  3. Job name: vpp-verify-performance-short
    • Description: This job is testing vpp proposed patch against short performance test cases
    • Trigger: Comment "verify-perf-short" in proposed patch in VPP project
    • Status: WORKING

CSIT performance test cases - working

  1. Working performance test cases - 18 working in FD.io physical testbeds today.
  2. Last manual run results reports:
  3. List of working performance test cases
    1. ~/csit$ grep "^| [a-zA-Z1-9]" -nr tests | grep -vi " | " | sort | grep performance
    2. tests/suites/performance/long_bridge_domain.robot:24:| Find NDR by using linear search and 64B frames through bridge domain in 3-node topology
      • // TODO items: rename to `Find NDR by using RFC2544 linear search and 64B frames through bridge domain in 3-node topology`
    3. tests/suites/performance/long_bridge_domain.robot:34:| Find NDR by using linear search and 1518B frames through bridge domain in 3-node topology
    4. tests/suites/performance/long_bridge_domain.robot:44:| Find NDR by using linear search and 9000B frames through bridge domain in 3-node topology
    5. tests/suites/performance/long_ipv4.robot:28:| Find NDR by using linear search and 64B frames through IPv4 forwarding in 3-node topology
    6. tests/suites/performance/long_ipv4.robot:38:| Find NDR by using linear search and 1518B frames through IPv4 forwarding in 3-node topology
    7. tests/suites/performance/long_ipv4.robot:48:| Find NDR by using linear search and 9000B frames through IPv4 forwarding in 3-node topology
    8. tests/suites/performance/long_xconnect.robot:25:| Find NDR by using linear search and 64B frames through L2 cross connect in 3-node topology
    9. tests/suites/performance/long_xconnect.robot:35:| Find NDR by using linear search and 1518B frames through L2 cross connect in 3-node topology
    10. tests/suites/performance/long_xconnect.robot:45:| Find NDR by using linear search and 9000B frames through L2 cross connect in 3-node topology
    11. tests/suites/performance/short_bridge_domain.robot:23:| 1core VPP passes 64B frames through bridge domain at 3.5mpps in 3-node topology
    12. tests/suites/performance/short_bridge_domain.robot:30:| 1core VPP passes 1518B frames through bridge domain at 10gbps in 3-node topology
    13. tests/suites/performance/short_bridge_domain.robot:37:| 1core VPP passes 9000B frames through bridge domain at 10gbps in 3-node topology
    14. tests/suites/performance/short_ipv4.robot:27:| 1core VPP passes 64B frames through IPv4 forwarding at 3.5mpps in 3-node topology
      • // TODO items: rename to `1core VPP passes 64B frames through IPv4 forwarding at 2x 3.5Mpps in 3-node topology`
      • // TODO items: currently testing all functional combinations takes 10-to-15 minutes; going forward would need to split the functional combinations
    15. tests/suites/performance/short_ipv4.robot:34:| 1core VPP passes 1518B frames through IPv4 forwarding at 10gbps in 3-node topology
    16. tests/suites/performance/short_ipv4.robot:41:| 1core VPP passes 9000B frames through IPv4 forwarding at 10gbps in 3-node topology
    17. tests/suites/performance/short_xconnect.robot:24:| 1core VPP passes 64B frames through L2 cross connect at 3.5mpps in 3-node topology
    18. tests/suites/performance/short_xconnect.robot:31:| 1core VPP passes 1518B frames through L2 cross connect at 10gbps in 3-node topology
    19. tests/suites/performance/short_xconnect.robot:38:| 1core VPP passes 9000B frames through L2 cross connect at 10gbps in 3-node topology

CSIT performance test cases - plan and status

  1. [DONE, mmiklus] DropRateSearch
  2. [WIP, mmiklus] extend topology compute library with NIC type filtering
  3. [WIP, pmikus] multicore tests gerrit.fd.io change
  4. [TBD] IPv6
  5. [TBD] Large-scale FIB tests
    • IPv4
    • IPv6
    • mac
    • Various prefix lengths
  6. [TBD] Encapsulations
    • VxLAN
    • VLAN
    • GRE
    • IPSEC
  7. [TBD] Create topology files for tb1, tb3
  8. [TBD] T-REX VXLAN traffic profile & CSIT tests
  9. IxNetwork driver
    • [TBD] driver design
    • [TBD] driver implementation
      • setup, teardown
      • traffic stream configuration
      • statistics retrieval
    • [TBD] latency measurements
      • feasibility
      • implementation
    • [TBD] integration with CSIT tests
  10. Moongen driver
    • [TBD] verify requirements and available HW against latest MoonGen codebase
    • [TBD] integration
  11. T-REX
    • [TBD] latency (T-REX support to be confirmed)
  12. Other work items


--- end

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/PerfTestPlan&oldid=808"