Difference between revisions of "CSIT/csit1904 plan"
| Line 46: | Line 46: | ||
| Done | | Done | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Fix integrator bugs, apply convergence workarounds. Fine tune soak throughput parameters. Add comparison vs. NDR/PDR rates. | + | | Fix integrator bugs, apply convergence workarounds. Fine tune soak throughput parameters. Add comparison vs. NDR/PDR rates. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| Suite Autogeneration | | Suite Autogeneration | ||
| Done | | Done | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Generate all NIC and search type combination at runtime, from suites in repo. | + | | Generate all NIC and search type combination at runtime, from suites in repo. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| VPP Patch Performance Tests | | VPP Patch Performance Tests | ||
| Done | | Done | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Fine tune VPP patch vs. parent performance tests, degradation detection, gerrit voting. Add auto-trigger subject to physical testbeds' capacity. Improve documentation. | + | | Fine tune VPP patch vs. parent performance tests, degradation detection, gerrit voting. Add auto-trigger subject to physical testbeds' capacity. Improve documentation. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| CSIT PAPI Support | | CSIT PAPI Support | ||
| WIP | | WIP | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Move CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords. | + | | Move CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| NFV Density | | NFV Density | ||
| Done | | Done | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Implement "thin-VM" with selectable NF workloads. Increase density matrix combination coverage. | + | | Implement "thin-VM" with selectable NF workloads. Increase density matrix combination coverage. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| 1n-arm VPP_Device | | 1n-arm VPP_Device | ||
| WIP | | WIP | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Bring up VPP_Device test environment on 1n-arm testbed(s). | + | | Bring up VPP_Device test environment on 1n-arm testbed(s). https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| 1n-dnv VPP_Device | | 1n-dnv VPP_Device | ||
| Next rls | | Next rls | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Bring up VPP_Device test environment on 1n-dnv testbed(s). | + | | Bring up VPP_Device test environment on 1n-dnv testbed(s). https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| VPP_Path | | VPP_Path | ||
| Line 84: | Line 84: | ||
i) Required for functional acceptance of VPP feature path(s) driven by use case(s); | i) Required for functional acceptance of VPP feature path(s) driven by use case(s); | ||
| − | ii) See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress]. | + | ii) See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress]. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| Per VPP node Perfmon | | Per VPP node Perfmon | ||
| Next rls | | Next rls | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Use new VPP perfmon plugin for per node compute efficiency telemetry. | + | | Use new VPP perfmon plugin for per node compute efficiency telemetry. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| 3n-arm Environment | | 3n-arm Environment | ||
| WIP | | WIP | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Bring up 2n-arm testbeds in LFN FD.io lab. | + | | Bring up 2n-arm testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| 2n-dnv Environment | | 2n-dnv Environment | ||
| WIP | | WIP | ||
| 1904-Optimizations | | 1904-Optimizations | ||
| − | | Bring up 2n-dnv testbeds in LFN FD.io lab. | + | | Bring up 2n-dnv testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446 |
|- | |- | ||
| 3n-arm Tests | | 3n-arm Tests | ||
Latest revision as of 07:26, 5 June 2019
Contents
Introduction
This page tracks release information for FD.io CSIT-1904. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.
Release Milestones
| Milestone | Date | Deliverables |
|---|---|---|
| F0 | 2019-04-03 | Test case keywords code complete. Only low-risk changes accepted. |
| RC1 | 2019-04-10 (F0+7) | Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. |
| RC2 | 2019-04-17 (RC1+7) | Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2. |
| CSIT Release | 2019-04-24 (RC2+7) | CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release. |
| Report Publish | 2019-05-08 (Rls+14) | CSIT report published for VPP release. |
Release Deliverables
| Name | Status | Jira Category | Description |
|---|---|---|---|
| Soak Tests | Done | 1904-Optimizations | Fix integrator bugs, apply convergence workarounds. Fine tune soak throughput parameters. Add comparison vs. NDR/PDR rates. https://jira.fd.io/browse/CSIT-1446 |
| Suite Autogeneration | Done | 1904-Optimizations | Generate all NIC and search type combination at runtime, from suites in repo. https://jira.fd.io/browse/CSIT-1446 |
| VPP Patch Performance Tests | Done | 1904-Optimizations | Fine tune VPP patch vs. parent performance tests, degradation detection, gerrit voting. Add auto-trigger subject to physical testbeds' capacity. Improve documentation. https://jira.fd.io/browse/CSIT-1446 |
| CSIT PAPI Support | WIP | 1904-Optimizations | Move CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords. https://jira.fd.io/browse/CSIT-1446 |
| NFV Density | Done | 1904-Optimizations | Implement "thin-VM" with selectable NF workloads. Increase density matrix combination coverage. https://jira.fd.io/browse/CSIT-1446 |
| 1n-arm VPP_Device | WIP | 1904-Optimizations | Bring up VPP_Device test environment on 1n-arm testbed(s). https://jira.fd.io/browse/CSIT-1446 |
| 1n-dnv VPP_Device | Next rls | 1904-Optimizations | Bring up VPP_Device test environment on 1n-dnv testbed(s). https://jira.fd.io/browse/CSIT-1446 |
| VPP_Path | Open | 1904-Optimizations | Complete moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:
i) Required for functional acceptance of VPP feature path(s) driven by use case(s); ii) See P1 and P2 markup in CSIT_VIRL migration progress. https://jira.fd.io/browse/CSIT-1446 |
| Per VPP node Perfmon | Next rls | 1904-Optimizations | Use new VPP perfmon plugin for per node compute efficiency telemetry. https://jira.fd.io/browse/CSIT-1446 |
| 3n-arm Environment | WIP | 1904-Optimizations | Bring up 2n-arm testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446 |
| 2n-dnv Environment | WIP | 1904-Optimizations | Bring up 2n-dnv testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446 |
| 3n-arm Tests | WIP | 1904-Test | Port existing 2n-skx tests to 2n-arm environment. https://jira.fd.io/browse/CSIT-1445 |
| 2n-dnv Tests | WIP | 1904-Test | Port existing 2n-skx tests to 2n-dnv environment. https://jira.fd.io/browse/CSIT-1445 |
| AVF Driver Tests | WIP | 1904-Test | 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests. https://jira.fd.io/browse/CSIT-1445 |
| VPP_Device Arm | WIP | 1904-Test | 1n-arm: add device-driver focused tests (migration away from VIRL). https://jira.fd.io/browse/CSIT-1445 |
| VPP_Device Denverton | Open | 1904-Test | 1n-dnv: add device-driver focused tests (migration away from VIRL). https://jira.fd.io/browse/CSIT-1445 |
| HoneyComb Tests | Done | 1904-HoneyComb | HoneyComb tests maintenance. (More detail to be added.) https://jira.fd.io/browse/CSIT-1517 |
Jira Task Tracking
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
- to be added
Multi-Release Work Areas
| Work Area | Description |
|---|---|
| Xeon Skx testbeds | Make Skylake performance test coverage complete:
i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest QEMU; iii) Push vpp-dev to Ubuntu 18.04. |
| Arm testbeds | Introduce Arm performance tests. |
| Atom testbeds | Introduce Denverton and Rangeley performance tests. |
| Better vhost, memif coverage | Produce more complete test data for NFV service density:
i) Scaled-out Vhost-user/VM and Memif/Container tests; ii) Test the same packet paths and NF topologies: service chains, service pipelines; iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with. v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth). |
| VPP per patch performance tests | Productise per VPP patch performance tests with change detection, prepare for voting:
i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. |
| Trending Improved Detection | Make trending job use new Burst MRR trending tests for better anomaly detection:
i) Currently postponed, as the algorithm detects performance changes not related to VPP code. ii) We need heavy workarounds or way more predictable SUT behavior. |
| More VPP telemetry reported and analysed | API based consumption of VPP telemetry including existing general counters, and future extended per node counters. |
| Evolve throughput search | Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:
i) Compare MLRsearch with PLRsearch soak test results. |
| General enhancements | General CSIT and VPP performance test and infrastructure enhancements:
i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data. |
External Dependencies
- No known external dependencies.
