Difference between revisions of "CSIT/csit1810 plan"

From fd.io
Jump to: navigation, search
Line 3: Line 3:
 
'''<span style="color: red">Please be aware that this page is work-in-progress and more info is being added continuously.</span>'''
 
'''<span style="color: red">Please be aware that this page is work-in-progress and more info is being added continuously.</span>'''
  
−
== Introduction ==
+
== Deliverables Tracking ==
  
−
All CSIT release deliverables should be tracked in FDio CSIT Jira .using one of the following Jira Epic categories:
+
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
  
−
...
+
# [CSIT Framework]
 +
# [Operations]
 +
# [Test]
 +
# [PAL]
 +
# [VIRL]
 +
# [HoneyComb]
 +
# [DMM]
 +
 
 +
== Focus Areas ==
 +
 
 +
{|class="wikitable"
 +
! Work Area
 +
! Description
 +
|-
 +
| Xeon Skylake tests
 +
| Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
 +
|-
 +
| Arm tests
 +
| Introduce Arm performance tests.
 +
|-
 +
| Atom tests
 +
| Introduce Atom performance tests.
 +
|-
 +
| Better vhost, memif coverage
 +
| Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
 +
|-
 +
| VPP per patch performance tests
 +
| Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
 +
|-
 +
| Evolve throughput search
 +
| Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
 +
|-
 +
| General enhancements
 +
| General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.
 +
|}
  
 
== Release Deliverables ==
 
== Release Deliverables ==
  
−
...
+
{|class="wikitable"
 +
! Name
 +
! Status
 +
! Jira Category
 +
! Description
 +
|-
 +
| 2n-Skx tests
 +
| WIP
 +
| 1810-Test
 +
| Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds.
 +
|-
 +
| VPP_Device
 +
| WIP
 +
| 1810-Framework
 +
| Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm.
 +
|-
 +
| VPP_Path
 +
| Open
 +
| 1810-Framework
 +
| Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing VPP_Path: CSIT_VIRL migration progress].
 +
|-
 +
| Performance trending PAL evolution
 +
| WIP
 +
| 1810-PAL
 +
| Make PAL use new Burst MRR trending tests once productized for better anomaly detection.
 +
|-
 +
| Performance change detection
 +
| WIP
 +
| 1810-Framework
 +
| Introduction of Burst MRR tests to trending.
 +
|-
 +
| Per VPP patch performance tests
 +
| WIP
 +
| 1810-Framework
 +
| Per VPP patch vs. parent performance tests and voting (-1/0/+1).
 +
|-
 +
| Clock cycles per VPP node into CSIT-PAL
 +
| Open
 +
| 1810-PAL
 +
| Use the new VPP stats infra to retrieve runtime counters instead of using "show run".
 +
|}
  
 
== Release Milestones ==
 
== Release Milestones ==

Revision as of 13:18, 5 September 2018


Please be aware that this page is work-in-progress and more info is being added continuously.

Deliverables Tracking

All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:

  1. [CSIT Framework]
  2. [Operations]
  3. [Test]
  4. [PAL]
  5. [VIRL]
  6. [HoneyComb]
  7. [DMM]

Focus Areas

Work Area Description
Xeon Skylake tests Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
Arm tests Introduce Arm performance tests.
Atom tests Introduce Atom performance tests.
Better vhost, memif coverage Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
VPP per patch performance tests Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
Evolve throughput search Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
General enhancements General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.

Release Deliverables

Name Status Jira Category Description
2n-Skx tests WIP 1810-Test Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds.
VPP_Device WIP 1810-Framework Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm.
VPP_Path Open 1810-Framework Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in VPP_Path: CSIT_VIRL migration progress.
Performance trending PAL evolution WIP 1810-PAL Make PAL use new Burst MRR trending tests once productized for better anomaly detection.
Performance change detection WIP 1810-Framework Introduction of Burst MRR tests to trending.
Per VPP patch performance tests WIP 1810-Framework Per VPP patch vs. parent performance tests and voting (-1/0/+1).
Clock cycles per VPP node into CSIT-PAL Open 1810-PAL Use the new VPP stats infra to retrieve runtime counters instead of using "show run".

Release Milestones

Milestone Date Deliverables
F0 2018-10-03 Test case keywords code complete. Only low-risk changes accepted.
RC1 2018-10-10 (F0+7) Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1.
RC2 2018-10-17 (RC1+7) Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2.
CSIT Release 2018-10-24 (RC2+7) CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release.
Report Publish 2018-11-07 (Rls+14) CSIT report published for VPP release.

Expected Dependencies on Other Projects

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/csit1810_plan&oldid=8354"