Difference between revisions of "CSIT/csit1810 plan"
From fd.io
< CSIT
Mackonstan (Talk | contribs) |
Mackonstan (Talk | contribs) |
||
| Line 151: | Line 151: | ||
|- | |- | ||
| Arm testbeds | | Arm testbeds | ||
| − | | Introduce Arm performance tests. | + | | Introduce Arm performance tests. |
|- | |- | ||
| Atom testbeds | | Atom testbeds | ||
| Line 161: | Line 161: | ||
| VPP per patch performance tests | | VPP per patch performance tests | ||
| Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. | | Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. | ||
| + | |- | ||
| + | | More VPP telemetry reported and analysed | ||
| + | | API based consumption of VPP telemetry including existing general counters, and future extended per node counters. | ||
|- | |- | ||
| Evolve throughput search | | Evolve throughput search | ||
Revision as of 12:13, 15 October 2018
Contents
Introduction
This page tracks release information for FD.io CSIT-18.10. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.
Release Milestones
| Milestone | Date | Deliverables |
|---|---|---|
| F0 | 2018-10-03 | Test case keywords code complete. Only low-risk changes accepted. |
| RC1 | 2018-10-10 (F0+7) | Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. |
| RC2 | 2018-10-17 (RC1+7) | Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2. |
| CSIT Release | 2018-10-24 (RC2+7) | CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release. |
| Report Publish | 2018-11-07 (Rls+14) | CSIT report published for VPP release. |
Release Deliverables
| Name | Status | Jira Category | Description |
|---|---|---|---|
| 3n-skx Tests | Done | 1810-Test | Add performamce tests for 3n-skx (3-Node Xeon Skylake) testbeds: VM vhost and Container memif tests. |
| 2n-skx Tests | Done | 1810-Test | Add performamce tests for 2n-skx (2-Node Xeon Skylake) testbeds: focus on baseline and scale tests. |
| VXLAN Scale Tests | Done | 1810-Test | Add performamce tests for VXLAN scale with dot1q and l2bd. |
| AVF Driver Tests | Done | 1810-Test | Add performamce tests for i40e AVF driver on VPP, DPDK-free. |
| QAT | Done | 1810-Test | Fix reoccuring issues with QAT crypto accelerator cards. |
| VM Vhost Virtio Params Combinations | Done | 1810-Test | Add performance tests for VM vhost with different virtio parameters combinations: indirect buffers, mergeable buffers. |
| Soak Tests Experimental | WIP | 1810-Framework | Add soak performamce tests infrastructure for extended test duration and throughput discovery at given PLR and total long soak time e.g. minutes, hours, days. |
| VPP_Device | WIP | 1810-Framework | Container based functional VPP device tests: CI/CD system design integrated into LF, running on 1-Node testbeds (1n-skx, 1n-arm). Relying on VF+dot1q for external loopback cable packet passing. Focus on few baseline tests. |
| VPP_Path | Open | 1810-Framework | Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in CSIT_VIRL migration progress. |
| Trending Improved Detection | Done | 1810-PAL | Make trending job use new Burst MRR trending tests for better anomaly detection. |
| Trending Tests BMRR | Done | 1810-Framework | Use new Burst MRR tests for daily trending. |
| K8s/Ligato in Trending | Done | 1810-Test | Add K8s/Ligato Container memif tests to daily trending. |
| Per VPP Patch Performance Checks | WIP | 1810-Framework | Per VPP gerrit patch vs. parent performance tests, anomaly detection and voting (-1/0/+1). |
| Patch-on-Patch Infra | Done | 1810-Framework | Add capability to run performance tests using CSIT gerrit patch code testing VPP gerrit patch code, i.e. before any code is merged into git branch. |
| CSIT PAPI Support | WIP | 1810-Framework | Implementation of PAPI L1 KWs in CSIT. Required for migrating away from VAT. |
| Graphs Layout Improvements | WIP | 1810-PAL | Improve performance graphs layout for better readibility and maintenance: test grouping, axis labels, descriptions, other informative decoration. |
| Clock cycles per VPP node into CSIT-PAL | Open | 1810-PAL | Use the new VPP stats infra available via PAPI to retrieve runtime counters instead of using "show run". Blocked by no PAPI support in FD.io CSIT. |
Jira Task Tracking
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
- CSIT Framework
- Operations
- Test
- PAL
- VIRL
- HoneyComb
- [DMM]
Multi-Release Work Areas
| Work Area | Description |
|---|---|
| Xeon Skx testbeds | Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04. |
| Arm testbeds | Introduce Arm performance tests. |
| Atom testbeds | Introduce Atom performance tests. |
| Better vhost, memif coverage | Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. |
| VPP per patch performance tests | Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. |
| More VPP telemetry reported and analysed | API based consumption of VPP telemetry including existing general counters, and future extended per node counters. |
| Evolve throughput search | Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR). |
| General enhancements | General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data. |
External Dependencies
No dependencies identified.
