Difference between revisions of "CSIT/csit1810 plan"
From fd.io
< CSIT
Mackonstan (Talk | contribs) |
Mackonstan (Talk | contribs) |
||
Line 3: | Line 3: | ||
'''<span style="color: red">Please be aware that this page is work-in-progress and more info is being added continuously.</span>''' | '''<span style="color: red">Please be aware that this page is work-in-progress and more info is being added continuously.</span>''' | ||
− | == | + | == Deliverables Tracking == |
− | All CSIT release deliverables should be tracked in FDio CSIT Jira | + | All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories: |
− | ... | + | # [CSIT Framework] |
+ | # [Operations] | ||
+ | # [Test] | ||
+ | # [PAL] | ||
+ | # [VIRL] | ||
+ | # [HoneyComb] | ||
+ | # [DMM] | ||
+ | |||
+ | == Focus Areas == | ||
+ | |||
+ | {|class="wikitable" | ||
+ | ! Work Area | ||
+ | ! Description | ||
+ | |- | ||
+ | | Xeon Skylake tests | ||
+ | | Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04. | ||
+ | |- | ||
+ | | Arm tests | ||
+ | | Introduce Arm performance tests. | ||
+ | |- | ||
+ | | Atom tests | ||
+ | | Introduce Atom performance tests. | ||
+ | |- | ||
+ | | Better vhost, memif coverage | ||
+ | | Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. | ||
+ | |- | ||
+ | | VPP per patch performance tests | ||
+ | | Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. | ||
+ | |- | ||
+ | | Evolve throughput search | ||
+ | | Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR). | ||
+ | |- | ||
+ | | General enhancements | ||
+ | | General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data. | ||
+ | |} | ||
== Release Deliverables == | == Release Deliverables == | ||
− | ... | + | {|class="wikitable" |
+ | ! Name | ||
+ | ! Status | ||
+ | ! Jira Category | ||
+ | ! Description | ||
+ | |- | ||
+ | | 2n-Skx tests | ||
+ | | WIP | ||
+ | | 1810-Test | ||
+ | | Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds. | ||
+ | |- | ||
+ | | VPP_Device | ||
+ | | WIP | ||
+ | | 1810-Framework | ||
+ | | Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm. | ||
+ | |- | ||
+ | | VPP_Path | ||
+ | | Open | ||
+ | | 1810-Framework | ||
+ | | Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing VPP_Path: CSIT_VIRL migration progress]. | ||
+ | |- | ||
+ | | Performance trending PAL evolution | ||
+ | | WIP | ||
+ | | 1810-PAL | ||
+ | | Make PAL use new Burst MRR trending tests once productized for better anomaly detection. | ||
+ | |- | ||
+ | | Performance change detection | ||
+ | | WIP | ||
+ | | 1810-Framework | ||
+ | | Introduction of Burst MRR tests to trending. | ||
+ | |- | ||
+ | | Per VPP patch performance tests | ||
+ | | WIP | ||
+ | | 1810-Framework | ||
+ | | Per VPP patch vs. parent performance tests and voting (-1/0/+1). | ||
+ | |- | ||
+ | | Clock cycles per VPP node into CSIT-PAL | ||
+ | | Open | ||
+ | | 1810-PAL | ||
+ | | Use the new VPP stats infra to retrieve runtime counters instead of using "show run". | ||
+ | |} | ||
== Release Milestones == | == Release Milestones == |
Revision as of 13:18, 5 September 2018
Please be aware that this page is work-in-progress and more info is being added continuously.
Contents
[hide]Deliverables Tracking
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
- [CSIT Framework]
- [Operations]
- [Test]
- [PAL]
- [VIRL]
- [HoneyComb]
- [DMM]
Focus Areas
Work Area | Description |
---|---|
Xeon Skylake tests | Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04. |
Arm tests | Introduce Arm performance tests. |
Atom tests | Introduce Atom performance tests. |
Better vhost, memif coverage | Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. |
VPP per patch performance tests | Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. |
Evolve throughput search | Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR). |
General enhancements | General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data. |
Release Deliverables
Name | Status | Jira Category | Description |
---|---|---|---|
2n-Skx tests | WIP | 1810-Test | Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds. |
VPP_Device | WIP | 1810-Framework | Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm. |
VPP_Path | Open | 1810-Framework | Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in VPP_Path: CSIT_VIRL migration progress. |
Performance trending PAL evolution | WIP | 1810-PAL | Make PAL use new Burst MRR trending tests once productized for better anomaly detection. |
Performance change detection | WIP | 1810-Framework | Introduction of Burst MRR tests to trending. |
Per VPP patch performance tests | WIP | 1810-Framework | Per VPP patch vs. parent performance tests and voting (-1/0/+1). |
Clock cycles per VPP node into CSIT-PAL | Open | 1810-PAL | Use the new VPP stats infra to retrieve runtime counters instead of using "show run". |
Release Milestones
Milestone | Date | Deliverables |
---|---|---|
F0 | 2018-10-03 | Test case keywords code complete. Only low-risk changes accepted. |
RC1 | 2018-10-10 (F0+7) | Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. |
RC2 | 2018-10-17 (RC1+7) | Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2. |
CSIT Release | 2018-10-24 (RC2+7) | CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release. |
Report Publish | 2018-11-07 (Rls+14) | CSIT report published for VPP release. |