Difference between revisions of "CSIT/csit1810 plan"
From fd.io
< CSIT
Mackonstan (Talk | contribs) |
|||
| Line 7: | Line 7: | ||
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories: | All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories: | ||
| − | # [CSIT Framework] | + | # [https://jira.fd.io/browse/CSIT-1240 CSIT Framework] |
| − | # [Operations] | + | # [https://jira.fd.io/browse/CSIT-1238 Operations] |
| − | # [Test] | + | # [https://jira.fd.io/browse/CSIT-1236 Test] |
| − | # [PAL] | + | # [https://jira.fd.io/browse/CSIT-1237 PAL] |
| − | # [VIRL] | + | # [https://jira.fd.io/browse/CSIT-929 VIRL] |
| − | # [HoneyComb] | + | # [https://jira.fd.io/browse/CSIT-1239 HoneyComb] |
# [DMM] | # [DMM] | ||
Revision as of 11:44, 27 September 2018
Please be aware that this page is work-in-progress and more info is being added continuously.
Contents
Deliverables Tracking
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
- CSIT Framework
- Operations
- Test
- PAL
- VIRL
- HoneyComb
- [DMM]
Focus Areas
| Work Area | Description |
|---|---|
| Xeon Skylake tests | Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04. |
| Arm tests | Introduce Arm performance tests. |
| Atom tests | Introduce Atom performance tests. |
| Better vhost, memif coverage | Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. |
| VPP per patch performance tests | Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. |
| Evolve throughput search | Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR). |
| General enhancements | General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data. |
Release Deliverables
| Name | Status | Jira Category | Description |
|---|---|---|---|
| 2n-Skx tests | WIP | 1810-Test | Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds. |
| VPP_Device | WIP | 1810-Framework | Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm. |
| VPP_Path | Open | 1810-Framework | Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in VPP_Path: CSIT_VIRL migration progress. |
| Performance trending PAL evolution | WIP | 1810-PAL | Make PAL use new Burst MRR trending tests once productized for better anomaly detection. |
| Performance change detection | WIP | 1810-Framework | Introduction of Burst MRR tests to trending. |
| Per VPP patch performance tests | WIP | 1810-Framework | Per VPP patch vs. parent performance tests and voting (-1/0/+1). |
| Clock cycles per VPP node into CSIT-PAL | Open | 1810-PAL | Use the new VPP stats infra to retrieve runtime counters instead of using "show run". |
Release Milestones
| Milestone | Date | Deliverables |
|---|---|---|
| F0 | 2018-10-03 | Test case keywords code complete. Only low-risk changes accepted. |
| RC1 | 2018-10-10 (F0+7) | Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. |
| RC2 | 2018-10-17 (RC1+7) | Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2. |
| CSIT Release | 2018-10-24 (RC2+7) | CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release. |
| Report Publish | 2018-11-07 (Rls+14) | CSIT report published for VPP release. |
