Difference between revisions of "CSIT/csit1901 plan"
Mackonstan (Talk | contribs) |
Mackonstan (Talk | contribs) |
||
Line 42: | Line 42: | ||
! Jira Category | ! Jira Category | ||
! Description | ! Description | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
|- | |- | ||
| Soak Tests | | Soak Tests | ||
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | Productize soak | + | | Productize soak throughput tests for given PLR and extended test duration. |
|- | |- | ||
− | | | + | | VPP Patch Performance Tests |
| Open | | Open | ||
− | | 1901- | + | | 1901-Framework |
− | | | + | | VPP patch vs. parent performance tests, degradation detection, gerrit voting. Auto-trigger subject to physical testbeds' capacity. |
|- | |- | ||
− | | | + | | CSIT PAPI Support |
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | | + | | Move CSIT code from VAT to PAPI, rewrite of Level-1 Keywords. |
|- | |- | ||
− | | | + | | NFV Density |
| Open | | Open | ||
− | | 1901- | + | | 1901-Framework |
− | | | + | | Experimental: infrastructure code for VNF and CNF density performance tests. |
|- | |- | ||
− | | VPP_Device | + | | 1n-arm VPP_Device |
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | | + | | Bring up VPP_Device test environment on 1n-arm testbed(s). |
|- | |- | ||
− | | VPP_Device | + | | 1n-dnv VPP_Device |
| Open | | Open | ||
− | | 1901- | + | | 1901-Framework |
− | | | + | | Bring up VPP_Device test environment on 1n-dnv testbed(s). |
|- | |- | ||
| VPP_Path | | VPP_Path | ||
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | | + | | Continue moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests: |
+ | i) Required for functional acceptance of VPP feature path(s) driven by use case(s); | ||
+ | |||
+ | ii) See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress]. | ||
|- | |- | ||
− | | | + | | Clock cycles per VPP node into CSIT-PAL |
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | | + | | Use the new VPP stats infra to retrieve runtime counters, requires PAPI support in FD.io CSIT. |
|- | |- | ||
− | | | + | | 2n-arm Environment |
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | | + | | Bring up 2n-arm testbeds in LFN FD.io lab. |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
|- | |- | ||
| 2n-dnv Environment | | 2n-dnv Environment | ||
| Open | | Open | ||
| 1901-Framework | | 1901-Framework | ||
− | | | + | | Bring up 2n-dnv testbeds in LFN FD.io lab. |
+ | |- | ||
+ | | 2n-arm Tests | ||
+ | | Open | ||
+ | | 1901-Test | ||
+ | | Port existing 2n-skx tests to 2n-arm environment. | ||
|- | |- | ||
| 2n-dnv Tests | | 2n-dnv Tests | ||
Line 118: | Line 106: | ||
| Port existing 2n-skx tests to 2n-dnv environment. | | Port existing 2n-skx tests to 2n-dnv environment. | ||
|- | |- | ||
− | | | + | | AVF Driver Tests |
| Open | | Open | ||
− | | 1901- | + | | 1901-Test |
− | | | + | | 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests. |
|- | |- | ||
− | | 2n- | + | | 2n-skx Tests |
| Open | | Open | ||
| 1901-Test | | 1901-Test | ||
− | | | + | | 2n-skx: drive test parity with 3n-skx, focusing on non-tunnel encapsulations. |
+ | |- | ||
+ | | 3n-skx Tests | ||
+ | | Open | ||
+ | | 1901-Test | ||
+ | | If new NICs added, add baseline tests for these. | ||
+ | |- | ||
+ | | VPP_Device Xeon | ||
+ | | Open | ||
+ | | 1901-Test | ||
+ | | 1n-skx: add device-driver focused tests (migration away from VIRL). | ||
+ | |- | ||
+ | | VPP_Device Arm | ||
+ | | Open | ||
+ | | 1901-Test | ||
+ | | 1n-arm: add device-driver focused tests (migration away from VIRL). | ||
+ | |- | ||
+ | | VPP_Device Denverton | ||
+ | | Open | ||
+ | | 1901-Test | ||
+ | | 1n-dnv: add device-driver focused tests (migration away from VIRL). | ||
+ | |- | ||
+ | | NFV Density Tests | ||
+ | | Open | ||
+ | | 1901-Test | ||
+ | | Experimental: Density performance tests with VNF Service Chains, CNF Service Chains and CNF Service Pipelines. | ||
|} | |} | ||
Line 133: | Line 146: | ||
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories: | All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories: | ||
− | # [https://jira.fd.io/browse/CSIT-1240 | + | # [https://jira.fd.io/browse/CSIT-1240 Framework] |
# [https://jira.fd.io/browse/CSIT-1238 Operations] | # [https://jira.fd.io/browse/CSIT-1238 Operations] | ||
# [https://jira.fd.io/browse/CSIT-1236 Test] | # [https://jira.fd.io/browse/CSIT-1236 Test] | ||
Line 148: | Line 161: | ||
|- | |- | ||
| Xeon Skx testbeds | | Xeon Skx testbeds | ||
− | | Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest | + | | Make Skylake performance test coverage complete: |
+ | i) Boost tests in 2-Node setups, complete 3-Node setups; | ||
+ | |||
+ | ii) Complete Memif/Container and Vhost-user/VM with latest QEMU; | ||
+ | |||
+ | iii) Push vpp-dev to Ubuntu 18.04. | ||
|- | |- | ||
| Arm testbeds | | Arm testbeds | ||
Line 154: | Line 172: | ||
|- | |- | ||
| Atom testbeds | | Atom testbeds | ||
− | | Introduce | + | | Introduce Denverton and Rangeley performance tests. |
|- | |- | ||
| Better vhost, memif coverage | | Better vhost, memif coverage | ||
− | | | + | | Produce more complete test data for NFV service density: |
+ | i) Scaled-out Vhost-user/VM and Memif/Container tests; | ||
+ | ii) Test the same packet paths and NF topologies: service chains, service pipelines; | ||
+ | |||
+ | iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. | ||
+ | |||
+ | iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with. | ||
+ | |||
+ | v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth). | ||
|- | |- | ||
| VPP per patch performance tests | | VPP per patch performance tests | ||
− | | Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. | + | | Productise per VPP patch performance tests with change detection, prepare for voting: |
+ | i) Improve detection accuracy and precision; | ||
+ | |||
+ | ii) Nail down current results variance; | ||
+ | |||
+ | iii) Apply improvements to continuous trending and (future) git auto-bisection. | ||
|- | |- | ||
| Trending Improved Detection | | Trending Improved Detection | ||
− | | Make trending job use new Burst MRR trending tests for better anomaly detection | + | | Make trending job use new Burst MRR trending tests for better anomaly detection: |
+ | i) Currently postponed, as the algorithm detects performance changes not related to VPP code. | ||
+ | |||
+ | ii) We need heavy workarounds or way more predictable SUT behavior. | ||
|- | |- | ||
| More VPP telemetry reported and analysed | | More VPP telemetry reported and analysed | ||
Line 169: | Line 203: | ||
|- | |- | ||
| Evolve throughput search | | Evolve throughput search | ||
− | | Build upon MLRsearch experience vs. ordinary binary search: i) | + | | Build upon MLRsearch and PLRsearch experience vs. ordinary binary search: |
+ | i) Compare MLRsearch with PLRsearch soak test results. | ||
|- | |- | ||
| General enhancements | | General enhancements | ||
− | | General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18. | + | | General CSIT and VPP performance test and infrastructure enhancements: |
+ | i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; | ||
+ | |||
+ | ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; | ||
+ | |||
+ | iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; | ||
+ | |||
+ | iv) Start migration from VAT to VPP Python API; | ||
+ | |||
+ | v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data. | ||
|} | |} | ||
Revision as of 14:13, 19 December 2018
Contents
[hide]Introduction
This page tracks release information for FD.io CSIT 1901. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.
Release Milestones
Milestone | Date | Deliverables |
---|---|---|
F0 | 2019-01-09 | Test case keywords code complete. Only low-risk changes accepted. |
RC1 | 2019-01-16 (F0+7) | Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. |
RC2 | 2019-01-23 (RC1+7) | Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2. |
CSIT Release | 2019-01-30 (RC2+7) | CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release. |
Report Publish | 2019-02-13 (Rls+14) | CSIT report published for VPP release. |
Release Deliverables
Name | Status | Jira Category | Description |
---|---|---|---|
Soak Tests | Open | 1901-Framework | Productize soak throughput tests for given PLR and extended test duration. |
VPP Patch Performance Tests | Open | 1901-Framework | VPP patch vs. parent performance tests, degradation detection, gerrit voting. Auto-trigger subject to physical testbeds' capacity. |
CSIT PAPI Support | Open | 1901-Framework | Move CSIT code from VAT to PAPI, rewrite of Level-1 Keywords. |
NFV Density | Open | 1901-Framework | Experimental: infrastructure code for VNF and CNF density performance tests. |
1n-arm VPP_Device | Open | 1901-Framework | Bring up VPP_Device test environment on 1n-arm testbed(s). |
1n-dnv VPP_Device | Open | 1901-Framework | Bring up VPP_Device test environment on 1n-dnv testbed(s). |
VPP_Path | Open | 1901-Framework | Continue moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:
i) Required for functional acceptance of VPP feature path(s) driven by use case(s); ii) See P1 and P2 markup in CSIT_VIRL migration progress. |
Clock cycles per VPP node into CSIT-PAL | Open | 1901-Framework | Use the new VPP stats infra to retrieve runtime counters, requires PAPI support in FD.io CSIT. |
2n-arm Environment | Open | 1901-Framework | Bring up 2n-arm testbeds in LFN FD.io lab. |
2n-dnv Environment | Open | 1901-Framework | Bring up 2n-dnv testbeds in LFN FD.io lab. |
2n-arm Tests | Open | 1901-Test | Port existing 2n-skx tests to 2n-arm environment. |
2n-dnv Tests | Open | 1901-Test | Port existing 2n-skx tests to 2n-dnv environment. |
AVF Driver Tests | Open | 1901-Test | 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests. |
2n-skx Tests | Open | 1901-Test | 2n-skx: drive test parity with 3n-skx, focusing on non-tunnel encapsulations. |
3n-skx Tests | Open | 1901-Test | If new NICs added, add baseline tests for these. |
VPP_Device Xeon | Open | 1901-Test | 1n-skx: add device-driver focused tests (migration away from VIRL). |
VPP_Device Arm | Open | 1901-Test | 1n-arm: add device-driver focused tests (migration away from VIRL). |
VPP_Device Denverton | Open | 1901-Test | 1n-dnv: add device-driver focused tests (migration away from VIRL). |
NFV Density Tests | Open | 1901-Test | Experimental: Density performance tests with VNF Service Chains, CNF Service Chains and CNF Service Pipelines. |
Jira Task Tracking
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
Multi-Release Work Areas
Work Area | Description |
---|---|
Xeon Skx testbeds | Make Skylake performance test coverage complete:
i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest QEMU; iii) Push vpp-dev to Ubuntu 18.04. |
Arm testbeds | Introduce Arm performance tests. |
Atom testbeds | Introduce Denverton and Rangeley performance tests. |
Better vhost, memif coverage | Produce more complete test data for NFV service density:
i) Scaled-out Vhost-user/VM and Memif/Container tests; ii) Test the same packet paths and NF topologies: service chains, service pipelines; iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with. v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth). |
VPP per patch performance tests | Productise per VPP patch performance tests with change detection, prepare for voting:
i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. |
Trending Improved Detection | Make trending job use new Burst MRR trending tests for better anomaly detection:
i) Currently postponed, as the algorithm detects performance changes not related to VPP code. ii) We need heavy workarounds or way more predictable SUT behavior. |
More VPP telemetry reported and analysed | API based consumption of VPP telemetry including existing general counters, and future extended per node counters. |
Evolve throughput search | Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:
i) Compare MLRsearch with PLRsearch soak test results. |
General enhancements | General CSIT and VPP performance test and infrastructure enhancements:
i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data. |
External Dependencies
- No known external dependencies.