CSIT/csit1908 plan
Contents
Introduction
This page tracks release information for FD.io CSIT-1908. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.
Release Milestones
| Milestone | Date | Deliverables |
|---|---|---|
| F0 | 2019-07-31 | Test case keywords code complete. Only low-risk changes accepted. |
| RC1 | 2019-08-07 (F0+7) | Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. Start dry-runs to identify CSIT gaps on less frequently run tests. |
| RC2 | 2019-08-14 (RC1+7) | Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2. |
| CSIT Release | 2019-08-21 (RC2+7) | CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release. |
| Report Publish | 2019-09-04 (Rls+14) | CSIT report published for VPP release. |
Release Deliverables
| Name | Status | Jira Category | Description |
|---|---|---|---|
| VAT to PAPI migration | WIP | Framework | Moving CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords |
| IPsec tests | Done | Performance | Updated IPsec base tests with new crypto backends and ciphers, scale tests >>1k tunnel interfaces. |
| VXLAN VNF Chains | Done | Performance | Extension of NFV density tests for VM with VXLAN encap to SUT/VPP. |
| VPP API Gate-keeping | WIP | Framework | VPP API checks |
| TCP tests | WIP | Performance | Refactoring of TCP tests. |
| Arm perf framework | Done | Framework | Aiming to have Arm daily performance jobs running for csit-vpp-perf-mrr-daily. |
| Arm perf tests | Done | Performance | Aiming to have a number of Arm performance tests running. |
| Arm vpp_device framework | Done* | Framework | Aiming to have Arm functional vpp_device jobs running for csit-vpp-verify and vpp-csit-verify.
*NOTE: 1n-tx2 testbed is incomplete, missing 1 server. |
| Arm vpp_device tests | Done | Performance | Aiming to have a number of Arm functional vpp_device tests running. |
| Denverton perf framework | Done | Framework | Atom Denverton daily performance jobs running for csit-vpp-perf-mrr-daily. |
| Denverton perf tests | Done | Performance | Atom Denverton performance tests running, full set. |
| Name | Status | Jira Category | Description |
| Name | Status | Jira Category | Description |
Jira Task Tracking
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
Framework CI process Performance Device Methodology Telemetry Tools Presentation Honeycomb Aarch64
Multi-Release Work Areas
| Work Area | Description |
|---|---|
| Xeon Skx testbeds | Make Skylake performance test coverage complete:
i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest QEMU; iii) Push vpp-dev to Ubuntu 18.04. |
| Arm testbeds | Introduce Arm performance tests. |
| Atom testbeds | Introduce Denverton and Rangeley performance tests. |
| Better vhost, memif coverage | Produce more complete test data for NFV service density:
i) Scaled-out Vhost-user/VM and Memif/Container tests; ii) Test the same packet paths and NF topologies: service chains, service pipelines; iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry. iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with. v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth). |
| VPP per patch performance tests | Productise per VPP patch performance tests with change detection, prepare for voting:
i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection. |
| Trending Improved Detection | Make trending job use new Burst MRR trending tests for better anomaly detection:
i) Currently postponed, as the algorithm detects performance changes not related to VPP code. ii) We need heavy workarounds or way more predictable SUT behavior. |
| More VPP telemetry reported and analysed | API based consumption of VPP telemetry including existing general counters, and future extended per node counters. |
| Evolve throughput search | Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:
i) Compare MLRsearch with PLRsearch soak test results. |
| General enhancements | General CSIT and VPP performance test and infrastructure enhancements:
i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data. |
External Dependencies
- No known external dependencies.
