Difference between revisions of "CSIT/csit1904 plan"

From fd.io
Jump to: navigation, search
(Add Suite Autogeneration, update Soak Tests and VPP Patch Performance Tests.)
 
(4 intermediate revisions by 2 users not shown)
Line 44: Line 44:
 
|-
 
|-
 
| Soak Tests
 
| Soak Tests
| WIP
+
| Done
| 1904-Framework
+
| 1904-Optimizations
| Fix integrator bugs, apply convergence workarounds. Fine tune soak throughput parameters. Add comparison vs. NDR/PDR rates.
+
| Fix integrator bugs, apply convergence workarounds. Fine tune soak throughput parameters. Add comparison vs. NDR/PDR rates. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| Suite Autogeneration
 
| Suite Autogeneration
| WIP
+
| Done
| 1904-Framework
+
| 1904-Optimizations
| Generate all NIC and search type combination at runtime, from suites in repo.
+
| Generate all NIC and search type combination at runtime, from suites in repo. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| VPP Patch Performance Tests
 
| VPP Patch Performance Tests
| WIP
+
| Done
| 1904-Framework
+
| 1904-Optimizations
| Fine tune VPP patch vs. parent performance tests, degradation detection, gerrit voting. Add auto-trigger subject to physical testbeds' capacity. Improve documentation.
+
| Fine tune VPP patch vs. parent performance tests, degradation detection, gerrit voting. Add auto-trigger subject to physical testbeds' capacity. Improve documentation. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| CSIT PAPI Support
 
| CSIT PAPI Support
 
| WIP
 
| WIP
| 1904-Framework
+
| 1904-Optimizations
| Move CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords.
+
| Move CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| NFV Density
 
| NFV Density
| WIP
+
| Done
| 1904-Framework
+
| 1904-Optimizations
| Implement "thin-VM" with selectable NF workloads. Increase density matrix combination coverage.
+
| Implement "thin-VM" with selectable NF workloads. Increase density matrix combination coverage. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| 1n-arm VPP_Device
 
| 1n-arm VPP_Device
 
| WIP
 
| WIP
| 1904-Framework
+
| 1904-Optimizations
| Bring up VPP_Device test environment on 1n-arm testbed(s).
+
| Bring up VPP_Device test environment on 1n-arm testbed(s). https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| 1n-dnv VPP_Device
 
| 1n-dnv VPP_Device
| Open
+
| Next rls
| 1904-Framework
+
| 1904-Optimizations
| Bring up VPP_Device test environment on 1n-dnv testbed(s).
+
| Bring up VPP_Device test environment on 1n-dnv testbed(s). https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| VPP_Path
 
| VPP_Path
 
| Open
 
| Open
| 1904-Framework
+
| 1904-Optimizations
 
| Complete moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:
 
| Complete moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:
 
i) Required for functional acceptance of VPP feature path(s) driven by use case(s);
 
i) Required for functional acceptance of VPP feature path(s) driven by use case(s);
  
ii) See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress].
+
ii) See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress]. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| Per VPP node Perfmon
 
| Per VPP node Perfmon
| Open
+
| Next rls
| 1904-Framework
+
| 1904-Optimizations
| Use new VPP perfmon plugin for per node compute efficiency telemetry.
+
| Use new VPP perfmon plugin for per node compute efficiency telemetry. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| 3n-arm Environment
 
| 3n-arm Environment
 
| WIP
 
| WIP
| 1904-Framework
+
| 1904-Optimizations
| Bring up 2n-arm testbeds in LFN FD.io lab.
+
| Bring up 2n-arm testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| 2n-dnv Environment
 
| 2n-dnv Environment
 
| WIP
 
| WIP
| 1904-Framework
+
| 1904-Optimizations
| Bring up 2n-dnv testbeds in LFN FD.io lab.
+
| Bring up 2n-dnv testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446
 
|-
 
|-
 
| 3n-arm Tests
 
| 3n-arm Tests
 
| WIP
 
| WIP
 
| 1904-Test
 
| 1904-Test
| Port existing 2n-skx tests to 2n-arm environment.
+
| Port existing 2n-skx tests to 2n-arm environment. https://jira.fd.io/browse/CSIT-1445
 
|-
 
|-
 
| 2n-dnv Tests
 
| 2n-dnv Tests
 
| WIP
 
| WIP
 
| 1904-Test
 
| 1904-Test
| Port existing 2n-skx tests to 2n-dnv environment.
+
| Port existing 2n-skx tests to 2n-dnv environment. https://jira.fd.io/browse/CSIT-1445
 
|-
 
|-
 
| AVF Driver Tests
 
| AVF Driver Tests
| Open
+
| WIP
 
| 1904-Test
 
| 1904-Test
| 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests.
+
| 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests. https://jira.fd.io/browse/CSIT-1445
 
|-
 
|-
 
| VPP_Device Arm
 
| VPP_Device Arm
 
| WIP
 
| WIP
 
| 1904-Test
 
| 1904-Test
| 1n-arm: add device-driver focused tests (migration away from  VIRL).
+
| 1n-arm: add device-driver focused tests (migration away from  VIRL). https://jira.fd.io/browse/CSIT-1445
 
|-
 
|-
 
| VPP_Device Denverton
 
| VPP_Device Denverton
 
| Open
 
| Open
 
| 1904-Test
 
| 1904-Test
| 1n-dnv: add device-driver focused tests (migration away from  VIRL).
+
| 1n-dnv: add device-driver focused tests (migration away from  VIRL). https://jira.fd.io/browse/CSIT-1445
 
|-
 
|-
 
| HoneyComb Tests
 
| HoneyComb Tests
| WIP
+
| Done
 
| 1904-HoneyComb
 
| 1904-HoneyComb
| HoneyComb tests maintenance. (More detail to be added.)
+
| HoneyComb tests maintenance. (More detail to be added.) https://jira.fd.io/browse/CSIT-1517
 
|}
 
|}
  

Latest revision as of 07:26, 5 June 2019


Introduction

This page tracks release information for FD.io CSIT-1904. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.

Release Milestones

Milestone Date Deliverables
F0 2019-04-03 Test case keywords code complete. Only low-risk changes accepted.
RC1 2019-04-10 (F0+7) Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1.
RC2 2019-04-17 (RC1+7) Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2.
CSIT Release 2019-04-24 (RC2+7) CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release.
Report Publish 2019-05-08 (Rls+14) CSIT report published for VPP release.

Release Deliverables

Name Status Jira Category Description
Soak Tests Done 1904-Optimizations Fix integrator bugs, apply convergence workarounds. Fine tune soak throughput parameters. Add comparison vs. NDR/PDR rates. https://jira.fd.io/browse/CSIT-1446
Suite Autogeneration Done 1904-Optimizations Generate all NIC and search type combination at runtime, from suites in repo. https://jira.fd.io/browse/CSIT-1446
VPP Patch Performance Tests Done 1904-Optimizations Fine tune VPP patch vs. parent performance tests, degradation detection, gerrit voting. Add auto-trigger subject to physical testbeds' capacity. Improve documentation. https://jira.fd.io/browse/CSIT-1446
CSIT PAPI Support WIP 1904-Optimizations Move CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords. https://jira.fd.io/browse/CSIT-1446
NFV Density Done 1904-Optimizations Implement "thin-VM" with selectable NF workloads. Increase density matrix combination coverage. https://jira.fd.io/browse/CSIT-1446
1n-arm VPP_Device WIP 1904-Optimizations Bring up VPP_Device test environment on 1n-arm testbed(s). https://jira.fd.io/browse/CSIT-1446
1n-dnv VPP_Device Next rls 1904-Optimizations Bring up VPP_Device test environment on 1n-dnv testbed(s). https://jira.fd.io/browse/CSIT-1446
VPP_Path Open 1904-Optimizations Complete moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:

i) Required for functional acceptance of VPP feature path(s) driven by use case(s);

ii) See P1 and P2 markup in CSIT_VIRL migration progress. https://jira.fd.io/browse/CSIT-1446

Per VPP node Perfmon Next rls 1904-Optimizations Use new VPP perfmon plugin for per node compute efficiency telemetry. https://jira.fd.io/browse/CSIT-1446
3n-arm Environment WIP 1904-Optimizations Bring up 2n-arm testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446
2n-dnv Environment WIP 1904-Optimizations Bring up 2n-dnv testbeds in LFN FD.io lab. https://jira.fd.io/browse/CSIT-1446
3n-arm Tests WIP 1904-Test Port existing 2n-skx tests to 2n-arm environment. https://jira.fd.io/browse/CSIT-1445
2n-dnv Tests WIP 1904-Test Port existing 2n-skx tests to 2n-dnv environment. https://jira.fd.io/browse/CSIT-1445
AVF Driver Tests WIP 1904-Test 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests. https://jira.fd.io/browse/CSIT-1445
VPP_Device Arm WIP 1904-Test 1n-arm: add device-driver focused tests (migration away from VIRL). https://jira.fd.io/browse/CSIT-1445
VPP_Device Denverton Open 1904-Test 1n-dnv: add device-driver focused tests (migration away from VIRL). https://jira.fd.io/browse/CSIT-1445
HoneyComb Tests Done 1904-HoneyComb HoneyComb tests maintenance. (More detail to be added.) https://jira.fd.io/browse/CSIT-1517

Jira Task Tracking

All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:

  1. to be added

Multi-Release Work Areas

Work Area Description
Xeon Skx testbeds Make Skylake performance test coverage complete:

i) Boost tests in 2-Node setups, complete 3-Node setups;

ii) Complete Memif/Container and Vhost-user/VM with latest QEMU;

iii) Push vpp-dev to Ubuntu 18.04.

Arm testbeds Introduce Arm performance tests.
Atom testbeds Introduce Denverton and Rangeley performance tests.
Better vhost, memif coverage Produce more complete test data for NFV service density:

i) Scaled-out Vhost-user/VM and Memif/Container tests; ii) Test the same packet paths and NF topologies: service chains, service pipelines;

iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.

iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with.

v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth).

VPP per patch performance tests Productise per VPP patch performance tests with change detection, prepare for voting:

i) Improve detection accuracy and precision;

ii) Nail down current results variance;

iii) Apply improvements to continuous trending and (future) git auto-bisection.

Trending Improved Detection Make trending job use new Burst MRR trending tests for better anomaly detection:

i) Currently postponed, as the algorithm detects performance changes not related to VPP code.

ii) We need heavy workarounds or way more predictable SUT behavior.

More VPP telemetry reported and analysed API based consumption of VPP telemetry including existing general counters, and future extended per node counters.
Evolve throughput search Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:

i) Compare MLRsearch with PLRsearch soak test results.

General enhancements General CSIT and VPP performance test and infrastructure enhancements:

i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm;

ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram;

iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI;

iv) Start migration from VAT to VPP Python API;

v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data.

External Dependencies

  1. No known external dependencies.