Difference between revisions of "CSIT/csit1810 plan"

From fd.io
Jump to: navigation, search
Line 1: Line 1:
 
[[Category:Release Plan]]
 
[[Category:Release Plan]]
  
−
'''<span style="color: red">Please be aware that this page is work-in-progress and more info is being added continuously.</span>'''
+
'''<span style="color: red">This page contains a snapshot in time of FD.io CSIT project work and is updated regularly by hand but not in real-time. Near real-time information is available in FD.io CSIT code repository and auto-generated docs.</span>'''
  
−
== Deliverables Tracking ==
+
== Introduction ==
  
−
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
+
This page tracks release information for FD.io CSIT-18.10.
  
−
# [https://jira.fd.io/browse/CSIT-1240 CSIT Framework]
+
== Release Milestones ==
−
# [https://jira.fd.io/browse/CSIT-1238 Operations]
+
−
# [https://jira.fd.io/browse/CSIT-1236 Test]
+
−
# [https://jira.fd.io/browse/CSIT-1237 PAL]
+
−
# [https://jira.fd.io/browse/CSIT-929 VIRL]
+
−
# [https://jira.fd.io/browse/CSIT-1239 HoneyComb]
+
−
# [DMM]
+
  
−
== Focus Areas ==
+
<!-- It is not uncommon for a project as part of its release process to have Milestones (M0, M1, M2, RC0, RC1) with corresponding events which occur at various milestones (API Freeze, Code Freeze, etc).  This is the place to list Milestones, their expected dates, and their expected events (whether process events, API Freeze, or noting a deliverable is coming in at a particular event. -->
  
−
{|class="wikitable"
+
{| class="wikitable"
−
! Work Area
+
! Milestone
−
! Description
+
! Date
 +
! Deliverables
 
|-
 
|-
−
| Xeon Skylake tests
+
| F0
−
| Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
+
| 2018-10-03
 +
| Test case keywords code complete. Only low-risk changes accepted.
 
|-
 
|-
−
| Arm tests
+
| RC1
−
| Introduce Arm performance tests.
+
| 2018-10-10 (F0+7)
 +
| Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with [https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.10#Release_Milestones VPP RC1].
 
|-
 
|-
−
| Atom tests
+
| RC2
−
| Introduce Atom performance tests.
+
| 2018-10-17 (RC1+7)
 +
| Dry-run testing begins of VPP RC2, performance and functional. Date aligned with [https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.10#Release_Milestones VPP RC2].
 
|-
 
|-
−
| Better vhost, memif coverage
+
| CSIT Release
−
| Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
+
| 2018-10-24 (RC2+7)
−
|-
+
| CSIT release complete. VPP release testing starts. Date aligned with [https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.10#Release_Milestones VPP Formal Release].
−
| VPP per patch performance tests
+
−
| Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
+
 
|-
 
|-
−
| Evolve throughput search
+
| Report Publish
−
| Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
+
| 2018-11-07 (Rls+14)
−
|-
+
| CSIT report published for VPP release.
−
| General enhancements
+
−
| General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.
+
 
|}
 
|}
  
Line 51: Line 45:
 
! Description
 
! Description
 
|-
 
|-
−
| 2n-Skx tests
+
| 2n-skx Tests
 
| WIP
 
| WIP
 
| 1810-Test
 
| 1810-Test
−
| Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds.
+
| Adding performamce tests to run on 2n-skx (2-Node Xeon Skylake) testbeds, reach coverage parity with 3n-skx (3-Node Xeon Skylake).
 
|-
 
|-
 
| VPP_Device
 
| VPP_Device
 
| WIP
 
| WIP
 
| 1810-Framework
 
| 1810-Framework
−
| Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm.
+
| New CI/CD system design and implementation of Container based VPP and Hardware device integration tests running on 1-Node testbeds: 1n-skx, 1n-arm.
 
|-
 
|-
 
| VPP_Path
 
| VPP_Path
Line 87: Line 81:
 
|}
 
|}
  
−
== Release Milestones ==
+
== Jira Task Tracking ==
  
−
<!-- It is not uncommon for a project as part of its release process to have Milestones (M0, M1, M2, RC0, RC1) with corresponding events which occur at various milestones (API Freeze, Code Freeze, etc).  This is the place to list Milestones, their expected dates, and their expected events (whether process events, API Freeze, or noting a deliverable is coming in at a particular event. -->
+
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
  
−
{| class="wikitable"
+
# [https://jira.fd.io/browse/CSIT-1240 CSIT Framework]
−
! Milestone
+
# [https://jira.fd.io/browse/CSIT-1238 Operations]
−
! Date
+
# [https://jira.fd.io/browse/CSIT-1236 Test]
−
! Deliverables
+
# [https://jira.fd.io/browse/CSIT-1237 PAL]
 +
# [https://jira.fd.io/browse/CSIT-929 VIRL]
 +
# [https://jira.fd.io/browse/CSIT-1239 HoneyComb]
 +
# [DMM]
 +
 
 +
== Multi-Release Work Areas ==
 +
 
 +
{|class="wikitable"
 +
! Work Area
 +
! Description
 
|-
 
|-
−
| F0
+
| Xeon Skx testbeds
−
| 2018-10-03
+
| Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
−
| Test case keywords code complete. Only low-risk changes accepted.
+
 
|-
 
|-
−
| RC1
+
| Arm testbeds
−
| 2018-10-10 (F0+7)
+
| Introduce Arm performance tests. Machines
−
| Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with [https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.10#Release_Milestones VPP RC1].
+
 
|-
 
|-
−
| RC2
+
| Atom testbeds
−
| 2018-10-17 (RC1+7)
+
| Introduce Atom performance tests.
−
| Dry-run testing begins of VPP RC2, performance and functional. Date aligned with [https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.10#Release_Milestones VPP RC2].
+
 
|-
 
|-
−
| CSIT Release
+
| Better vhost, memif coverage
−
| 2018-10-24 (RC2+7)
+
| Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
−
| CSIT release complete. VPP release testing starts. Date aligned with [https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.10#Release_Milestones VPP Formal Release].
+
 
|-
 
|-
−
| Report Publish
+
| VPP per patch performance tests
−
| 2018-11-07 (Rls+14)
+
| Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
−
| CSIT report published for VPP release.
+
|-
 +
| Evolve throughput search
 +
| Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
 +
|-
 +
| General enhancements
 +
| General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.
 
|}
 
|}
  
−
== Expected Dependencies on Other Projects ==
+
== External Dependencies ==
 +
 
 +
No dependencies identified.

Revision as of 13:18, 10 October 2018


This page contains a snapshot in time of FD.io CSIT project work and is updated regularly by hand but not in real-time. Near real-time information is available in FD.io CSIT code repository and auto-generated docs.

Introduction

This page tracks release information for FD.io CSIT-18.10.

Release Milestones

Milestone Date Deliverables
F0 2018-10-03 Test case keywords code complete. Only low-risk changes accepted.
RC1 2018-10-10 (F0+7) Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1.
RC2 2018-10-17 (RC1+7) Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2.
CSIT Release 2018-10-24 (RC2+7) CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release.
Report Publish 2018-11-07 (Rls+14) CSIT report published for VPP release.

Release Deliverables

Name Status Jira Category Description
2n-skx Tests WIP 1810-Test Adding performamce tests to run on 2n-skx (2-Node Xeon Skylake) testbeds, reach coverage parity with 3n-skx (3-Node Xeon Skylake).
VPP_Device WIP 1810-Framework New CI/CD system design and implementation of Container based VPP and Hardware device integration tests running on 1-Node testbeds: 1n-skx, 1n-arm.
VPP_Path Open 1810-Framework Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in VPP_Path: CSIT_VIRL migration progress.
Performance trending PAL evolution WIP 1810-PAL Make PAL use new Burst MRR trending tests once productized for better anomaly detection.
Performance change detection WIP 1810-Framework Introduction of Burst MRR tests to trending.
Per VPP patch performance tests WIP 1810-Framework Per VPP patch vs. parent performance tests and voting (-1/0/+1).
Clock cycles per VPP node into CSIT-PAL Open 1810-PAL Use the new VPP stats infra to retrieve runtime counters instead of using "show run".

Jira Task Tracking

All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:

  1. CSIT Framework
  2. Operations
  3. Test
  4. PAL
  5. VIRL
  6. HoneyComb
  7. [DMM]

Multi-Release Work Areas

Work Area Description
Xeon Skx testbeds Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
Arm testbeds Introduce Arm performance tests. Machines
Atom testbeds Introduce Atom performance tests.
Better vhost, memif coverage Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
VPP per patch performance tests Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
Evolve throughput search Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
General enhancements General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.

External Dependencies

No dependencies identified.

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/csit1810_plan&oldid=8468"