Difference between revisions of "CSIT/VnetInfraPlan"

From fd.io
Jump to: navigation, search
(Undo revision 890 by Ckoester (talk))
 
Line 1: Line 1:
−
==CSIT Hardware testbeds - working==
+
__TOC__
  
−
# [DONE, ckoester] Initial installation
+
=CSIT development plan - background=
−
#* 3x 3-node-ucsc240m4 at LF
+
−
#* Operating System installation
+
−
#* Topology and connectivity tests completed
+
−
#** FD.io CSIT physical testbeds wiring [link to wiring file]
+
−
# [DONE, ckoester] All NICs installed and verified
+
−
#* 2p10GE 82599 Niantic, Intel
+
−
#* 2p10GE X710 Fortville, Intel
+
−
#* 2p40GE XL710 Fortville, Intel
+
−
#* 2p40GE VIC1385, Cisco
+
−
#* 2p10GE VIC1227, Cisco
+
  
 +
CSIT test development priorities are aligned with VPP releases timeline, with relative dev work prioritization for each VPP release driven by perceived importance of test verification for VPP functionality and performance. Specific CSIT dev work priorities are driven by rough consensus within the project and contributors and committers availability and willingness to do work.
  
−
==CSIT VIRL testbeds - working==
+
Following priority tagging scheme is used in this CSIT wiki and FD.io jira for CSIT:
  
−
# [DONE, ckoester] Initial setup
+
* [$VppRlsNo-$RelPrio]
−
#* 3-node topology (similar to physical testbeds)
+
** VppRlsNo - aligned VPP release number - 16.09, 16.12, 17.03.
−
#* Automatic spawning of VIRL topology, VPP installation, creation of topology file for CSIT testing
+
** RelPrio - CSIT release priority - P0 for Must Have, P1 for Should Have, P2 for Could Have.
−
# [DONE, ckoester] Nested VM [gerrit.fd.io change]
+
* CSIT dev work tagging examples:
−
#* [DONE, ckoester] KVM-in-KVM working on VIRL machines [https://gerrit.fd.io/r/#/c/665/ gerrit.fd.io change]
+
** [1609-P0] - CSIT Must Have for vpp-v1609.
 +
** [1609-P1] - CSIT Should Have for vpp-v1609.
 +
** [1609-P2] - CSIT Could Have for vpp-v1609.
  
 +
CSIT dev work plan execution tracked on [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=1&projectKey=CSIT CSIT jira], with priority tags encoded as Jira labels for ease of tracking and reporting.
  
−
==CSIT Hardware testbeds - plan==
+
Two different testbeds are used to execute CSIT tests:
 +
# CSIT Hardware testbeds - run performance tests, [https://wiki.fd.io/view/CSIT/CSIT_LF_testbed description of physical setup].
 +
# CSIT VIRL testbeds - run functional tests.
  
−
# [P0-R1] NIC onboarding into CSIT
+
=CSIT infra documentation=
−
#* [DONE] 2p10GE 82599 Niantic, Intel
+
−
#* [P0-R1] 2p10GE X710 Fortville, Intel
+
−
#* [P0-R1] 2p40GE XL710 Fortville, Intel
+
−
#* [P0-R1] 2p40GE VIC1385, Cisco
+
−
#* [P0-R1] 2p10GE VIC1227, Cisco
+
−
#* [RMK] Need to resolve NIC fancy features getting in the way
+
−
#* [RMK] pending functional testcases to utilize additional NIC
+
−
# [P0-R1] Establish HW performance boundaries
+
−
#* [RMK] Need to validate functional testing for all NICs first
+
−
#* [RMK] UCS servers with NICs, CPU, PCI and Memory sub-systems
+
−
# [P2-R1] OOB monitoring
+
−
#* Perform SUT health check
+
−
#* Detect and react if administrative access to the SUT is lost
+
−
# [P1-R2] Scripted UCS reinstallation
+
−
#* [RMK] switch between OS/distributions
+
−
#* [RMK] re-install after failure
+
  
−
==CSIT VIRL testbeds - plan==
+
JIRA [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=8&quickFilter=82&quickFilter=86 board view], [https://jira.fd.io/issues/?filter=10662 list view] - '''infra''', '''docs''' labels for CSIT work tracking.
  
 +
# [1609-p0] Document VIRL image build and upgrade process, CSIT-333.
 +
# [1609-p0] Document configuration of VIRL server and HW testbeds from operational pools., CSIT-334.
  
−
# [P0-R1, ckoester] Expand hardware - ETA week of 04/26
+
=CSIT infra operations=
−
#* Install 2x additional UCS c240m4
+
−
#** [P0-R1] 1x for VIRL testbed redundancy
+
−
#** [P0-R1] 1x for testing and staging
+
−
#* [RMK] Waiting for UCS servers to be delivered and installed, ETA 2 weeks
+
−
# [P2-R1] Implement additional topologies
+
−
#* [P2-R1] 2-node testbed
+
−
#* [P2-R1] Star topologies
+
−
#* [P2-R1] Larger rings
+
−
#* [RMK] To be discussed with CSIT functional testing. Requirement for topologies is driven by functional test cases.
+
−
# Host management and monitoring
+
−
#* [P0-R1] Reservations, load-balancing, redundancy
+
−
#* [P1-R1] Usage monitoring
+
−
# [P1-R1] OOB monitoring
+
−
#* [P1-R1] Perform SUT health check
+
−
#* [P1-R1] Detect and react if administrative access to the SUT is lost
+
  
 +
JIRA [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=8&quickFilter=82&quickFilter=85 board view], [https://jira.fd.io/issues/?filter=10665 list view] - '''infra''', '''oper''' labels for CSIT work tracking.
  
−
==CSIT LF VM cloud testbeds==
+
# [1609-p0] Migration of perf tests from semiweekly to new daily job. CSIT-311.
 +
# [1609-p1] Long-term storage for error/diagnostic logs. CSIT-146.
 +
# [1609-p1] Test cases to log/store diagnostic data to external storage. CSIT-144.
 +
# [1609-p1] Move bootstrap_* files from CSIT to ci-management. CSIT-3.
  
−
# [P1-R1] Develop nested virtualisation
+
=CSIT Hardware testbeds=
−
#* [DONE, ckoester] Prepare KVM-in-KVM environment suitable for initial POC
+
−
#* [P0-R1] Run POC KVM-in-KVM in LF environment; identify and eliminate any showstoppers
+
−
#* [P1-R1] Run 3-node topology inside LF hosted VM cloud
+
−
#** Once running, it will enable elastic scaling of funcational VPP tests in LF VM cloud
+
−
#* [P2-R2, experimental] Run VIRL inside LF hosted VM cloud
+
−
#** Would allow testing of all topologies (not only 3-node) in elastic LF VM cloud
+
−
#** but may not be feasible due to execution time/resource constraints
+
  
 +
JIRA [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=8&quickFilter=82&quickFilter=83 board view], [https://jira.fd.io/issues/?filter=10663 list view] - '''infra''', '''env-phy''' labels for CSIT work tracking.
  
−
==CSIT Testsuite Portability==
+
# [1609-p0] Improve Qemu affinity setting. CSIT-337.
 +
# [1609-p0] Hugepage configuration on physical LF testbeds. CSIT-217.
 +
# [1609-p0] HW testbed installation scripts to include PIP requirements. CSIT-157.
 +
# [1609-p1] Out-of-band access to SUTs. CSIT-145.
 +
# [1609-p1] Improve test case logging. CSIT-149.
 +
# [1609-p1] Nested VM to include l3fwd startup script. CSIT-210.
 +
# [1609-p1] Create separate test case for VPP start. CSIT-147.
 +
# [1609-p2] Analyze usage of Python VPP API in CSIT instead of VAT. CSIT-7.
 +
# [1609-p2] dut_setup.sh enables trace on dpdk-input. CSIT-122.
 +
# [1609-p2] Fix keyword for VPP threads setup. CSIT-339.
 +
# [1609-p2] Detect CPU on all nodes. CSIT-338.
  
−
# [P2-R1] Distributable VIRL
+
=CSIT VIRL testbeds=
−
#* [RMK] Small VIRL topology for portable/laptop use
+
−
#* [RMK] Public repository of topologies
+
−
#* [RMK] Allow VPP code developers to test their code in their environment before committing
+
  
 +
JIRA [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=8&quickFilter=82&quickFilter=84 board view], [https://jira.fd.io/issues/?filter=10664 list view] - '''infra''', '''env-virl''' labels for CSIT work tracking.
  
−
==Multiple Operating System testing==
+
# [1609-p0] Include bridge-utils, docker and java in VM images. CSIT-192.
 +
# [1609-p1] Out-of-band access to SUTs. CSIT-145.
 +
# [1609-p1] Update nested VM qemu library to use 3rd serial console. CSIT-161.
 +
# [1609-p1] VIRL infrastructure periodic creation and distribution of images. CSIT-112.
 +
# [1609-p1] Health-check to capture TG/SUT environment after failed test case. CSIT-150.
 +
# [1609-p1] Execute start/stop-testcase scripts from git repository. CSIT-202.
 +
# [1609-p1] Do not destroy VM in case of test failure due to infrastructure issue. CSIT-151.
 +
# [1609-p1] Usage and status monitoring of VIRL hosts. CSIT-115.
 +
# [1609-p1] Improve test case logging. CSIT-149.
 +
# [1609-p1] Fix path creation KWs in to take *the whole path* in parameter list. CSIT-4.
 +
# [1609-p1] Create separate test case for VPP package installation. CSIT-148.
 +
# [1609-p1] Create separate test case for VPP start. CSIT-147.
 +
# [1609-p1] Add one general post-suite test to show vpp console output. CSIT-95.
 +
# [1609-p2] Nested VM: Replace cisco/cisco credentials with csit/csit. CSIT-159.
 +
# [1609-p2] Ubuntu VM: Replace cisco login with csit. CSIT-160.
 +
# [1609-p2] Modify VIRL and nested-VM username/password. CSIT-116.
 +
# [1609-p2] Nested-VM boot-up failed. CSIT-90.
 +
# [1609-p2] Use gcov for verifying VPP code test coverage by functional tests. CSIT-108.
  
−
(tbd: Hardware and/or virtual)
+
=Other CSIT infra work=
  
−
# [P2-R1] Be able to switch between various OS [distributions]x[versions]
+
# [1609-p1] Move bootstrap_* files from CSIT to ci-management., CSIT-3.
 +
# [1609-p1] Nested VM to include l3fwd startup script, CSIT-210.
 +
# [1609-p1] Test cases to log/store diagnostic data to external storage, CSIT-144.
 +
# [1609-p1] Long-term storage for error/diagnostic logs, CSIT-146.

Latest revision as of 18:37, 2 August 2016

CSIT development plan - background

CSIT test development priorities are aligned with VPP releases timeline, with relative dev work prioritization for each VPP release driven by perceived importance of test verification for VPP functionality and performance. Specific CSIT dev work priorities are driven by rough consensus within the project and contributors and committers availability and willingness to do work.

Following priority tagging scheme is used in this CSIT wiki and FD.io jira for CSIT:

  • [$VppRlsNo-$RelPrio]
    • VppRlsNo - aligned VPP release number - 16.09, 16.12, 17.03.
    • RelPrio - CSIT release priority - P0 for Must Have, P1 for Should Have, P2 for Could Have.
  • CSIT dev work tagging examples:
    • [1609-P0] - CSIT Must Have for vpp-v1609.
    • [1609-P1] - CSIT Should Have for vpp-v1609.
    • [1609-P2] - CSIT Could Have for vpp-v1609.

CSIT dev work plan execution tracked on CSIT jira, with priority tags encoded as Jira labels for ease of tracking and reporting.

Two different testbeds are used to execute CSIT tests:

  1. CSIT Hardware testbeds - run performance tests, description of physical setup.
  2. CSIT VIRL testbeds - run functional tests.

CSIT infra documentation

JIRA board view, list view - infra, docs labels for CSIT work tracking.

  1. [1609-p0] Document VIRL image build and upgrade process, CSIT-333.
  2. [1609-p0] Document configuration of VIRL server and HW testbeds from operational pools., CSIT-334.

CSIT infra operations

JIRA board view, list view - infra, oper labels for CSIT work tracking.

  1. [1609-p0] Migration of perf tests from semiweekly to new daily job. CSIT-311.
  2. [1609-p1] Long-term storage for error/diagnostic logs. CSIT-146.
  3. [1609-p1] Test cases to log/store diagnostic data to external storage. CSIT-144.
  4. [1609-p1] Move bootstrap_* files from CSIT to ci-management. CSIT-3.

CSIT Hardware testbeds

JIRA board view, list view - infra, env-phy labels for CSIT work tracking.

  1. [1609-p0] Improve Qemu affinity setting. CSIT-337.
  2. [1609-p0] Hugepage configuration on physical LF testbeds. CSIT-217.
  3. [1609-p0] HW testbed installation scripts to include PIP requirements. CSIT-157.
  4. [1609-p1] Out-of-band access to SUTs. CSIT-145.
  5. [1609-p1] Improve test case logging. CSIT-149.
  6. [1609-p1] Nested VM to include l3fwd startup script. CSIT-210.
  7. [1609-p1] Create separate test case for VPP start. CSIT-147.
  8. [1609-p2] Analyze usage of Python VPP API in CSIT instead of VAT. CSIT-7.
  9. [1609-p2] dut_setup.sh enables trace on dpdk-input. CSIT-122.
  10. [1609-p2] Fix keyword for VPP threads setup. CSIT-339.
  11. [1609-p2] Detect CPU on all nodes. CSIT-338.

CSIT VIRL testbeds

JIRA board view, list view - infra, env-virl labels for CSIT work tracking.

  1. [1609-p0] Include bridge-utils, docker and java in VM images. CSIT-192.
  2. [1609-p1] Out-of-band access to SUTs. CSIT-145.
  3. [1609-p1] Update nested VM qemu library to use 3rd serial console. CSIT-161.
  4. [1609-p1] VIRL infrastructure periodic creation and distribution of images. CSIT-112.
  5. [1609-p1] Health-check to capture TG/SUT environment after failed test case. CSIT-150.
  6. [1609-p1] Execute start/stop-testcase scripts from git repository. CSIT-202.
  7. [1609-p1] Do not destroy VM in case of test failure due to infrastructure issue. CSIT-151.
  8. [1609-p1] Usage and status monitoring of VIRL hosts. CSIT-115.
  9. [1609-p1] Improve test case logging. CSIT-149.
  10. [1609-p1] Fix path creation KWs in to take *the whole path* in parameter list. CSIT-4.
  11. [1609-p1] Create separate test case for VPP package installation. CSIT-148.
  12. [1609-p1] Create separate test case for VPP start. CSIT-147.
  13. [1609-p1] Add one general post-suite test to show vpp console output. CSIT-95.
  14. [1609-p2] Nested VM: Replace cisco/cisco credentials with csit/csit. CSIT-159.
  15. [1609-p2] Ubuntu VM: Replace cisco login with csit. CSIT-160.
  16. [1609-p2] Modify VIRL and nested-VM username/password. CSIT-116.
  17. [1609-p2] Nested-VM boot-up failed. CSIT-90.
  18. [1609-p2] Use gcov for verifying VPP code test coverage by functional tests. CSIT-108.

Other CSIT infra work

  1. [1609-p1] Move bootstrap_* files from CSIT to ci-management., CSIT-3.
  2. [1609-p1] Nested VM to include l3fwd startup script, CSIT-210.
  3. [1609-p1] Test cases to log/store diagnostic data to external storage, CSIT-144.
  4. [1609-p1] Long-term storage for error/diagnostic logs, CSIT-146.
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/VnetInfraPlan&oldid=2380"