Difference between revisions of "CSIT/csit1901 plan"

From fd.io
Jump to: navigation, search
Line 42: Line 42:
 
! Jira Category
 
! Jira Category
 
! Description
 
! Description
−
|-
 
−
| 3n-skx Tests
 
−
| Open
 
−
| 1901-Test
 
−
| If new NICs present (e.g. 100GE), add tests for these.
 
−
|-
 
−
| 2n-skx Tests
 
−
| Open
 
−
| 1901-Test
 
−
| Drive parity of test coverage with 3n-skx, focusing on non-tunnel encapsulations.
 
−
|-
 
−
| AVF Driver Tests
 
−
| Open
 
−
| 1901-Test
 
−
| Drive parity of test coverage with DPDK for all baseline and scale tests.
 
 
|-
 
|-
 
| Soak Tests
 
| Soak Tests
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Productize soak performamce tests for extended test duration and throughput discovery at given PLR.
+
| Productize soak throughput tests for given PLR and extended test duration.
 
|-
 
|-
−
| VPP_Device Xeon
+
| VPP Patch Performance Tests
 
| Open
 
| Open
−
| 1901-Test
+
| 1901-Framework
−
| Execute on migrating VIRL tests to VPP_Device Container based functional VPP device tests running on 1n-skx.
+
| VPP patch vs. parent performance tests, degradation detection, gerrit voting. Auto-trigger subject to physical testbeds' capacity.
 
|-
 
|-
−
| VPP_Device Arm
+
| CSIT PAPI Support
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Container based functional VPP device tests framework for 1n-arm testbeds.
+
| Move CSIT code from VAT to PAPI, rewrite of Level-1 Keywords.
 
|-
 
|-
−
| VPP_Device Arm
+
| NFV Density
 
| Open
 
| Open
−
| 1901-test
+
| 1901-Framework
−
| Execute on migrating VIRL tests to VPP_Device Container based functional VPP device tests running on 1n-arm.
+
| Experimental: infrastructure code for VNF and CNF density performance tests.
 
|-
 
|-
−
| VPP_Device Atom
+
| 1n-arm VPP_Device
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Container based functional VPP device tests framework for 1n-atom testbeds.
+
| Bring up VPP_Device test environment on 1n-arm testbed(s).
 
|-
 
|-
−
| VPP_Device Atom
+
| 1n-dnv VPP_Device
 
| Open
 
| Open
−
| 1901-test
+
| 1901-Framework
−
| Execute on migrating VIRL tests to VPP_Device Container based functional VPP device tests running on 1n-atom.
+
| Bring up VPP_Device test environment on 1n-dnv testbed(s).
 
|-
 
|-
 
| VPP_Path
 
| VPP_Path
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress].
+
| Continue moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:
 +
i) Required for functional acceptance of VPP feature path(s) driven by use case(s);
 +
 
 +
ii) See P1 and P2 markup in [https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing CSIT_VIRL migration progress].
 
|-
 
|-
−
| Per VPP Patch Performance Checks
+
| Clock cycles per VPP node into CSIT-PAL
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Per VPP patch vs. parent performance tests, anomaly detection, voting (-1/0/+1). Automated trigger subject to available physical testbed processing capacity.
+
| Use the new VPP stats infra to retrieve runtime counters, requires PAPI support in FD.io CSIT.
 
|-
 
|-
−
| CSIT PAPI Support
+
| 2n-arm Environment
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Implementation of PAPI L1 KWs in CSIT. Required for migraing away from VAT.
+
| Bring up 2n-arm testbeds in LFN FD.io lab.
−
|-
+
−
| Clock cycles per VPP node into CSIT-PAL
+
−
| Open
+
−
| 1901-PAL
+
−
| Use the new VPP stats infra available via PAPI to retrieve runtime counters instead of using "show run". Blocked by no PAPI support in FD.io CSIT.
+
 
|-
 
|-
 
| 2n-dnv Environment
 
| 2n-dnv Environment
 
| Open
 
| Open
 
| 1901-Framework
 
| 1901-Framework
−
| Onboard 2n-dnv (3n-dnv?) testbeds in LFN FD.io hosted lab.
+
| Bring up 2n-dnv testbeds in LFN FD.io lab.
 +
|-
 +
| 2n-arm Tests
 +
| Open
 +
| 1901-Test
 +
| Port existing 2n-skx tests to 2n-arm environment.
 
|-
 
|-
 
| 2n-dnv Tests
 
| 2n-dnv Tests
Line 118: Line 106:
 
| Port existing 2n-skx tests to 2n-dnv environment.
 
| Port existing 2n-skx tests to 2n-dnv environment.
 
|-
 
|-
−
| 2n-arm Environment
+
| AVF Driver Tests
 
| Open
 
| Open
−
| 1901-Framework
+
| 1901-Test
−
| Onboard 2n-arm (3n-arm?) testbeds in LFN FD.io hosted lab.
+
| 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests.
 
|-
 
|-
−
| 2n-arm Tests
+
| 2n-skx Tests
 
| Open
 
| Open
 
| 1901-Test
 
| 1901-Test
−
| Port existing 2n-skx tests to 2n-arm environment.
+
| 2n-skx: drive test parity with 3n-skx, focusing on non-tunnel encapsulations.
 +
|-
 +
| 3n-skx Tests
 +
| Open
 +
| 1901-Test
 +
| If new NICs added, add baseline tests for these.
 +
|-
 +
| VPP_Device Xeon
 +
| Open
 +
| 1901-Test
 +
| 1n-skx: add device-driver focused tests (migration away from  VIRL).
 +
|-
 +
| VPP_Device Arm
 +
| Open
 +
| 1901-Test
 +
| 1n-arm: add device-driver focused tests (migration away from  VIRL).
 +
|-
 +
| VPP_Device Denverton
 +
| Open
 +
| 1901-Test
 +
| 1n-dnv: add device-driver focused tests (migration away from  VIRL).
 +
|-
 +
| NFV Density Tests
 +
| Open
 +
| 1901-Test
 +
| Experimental: Density performance tests with VNF Service Chains, CNF Service Chains and CNF Service Pipelines.
 
|}
 
|}
  
Line 133: Line 146:
 
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
 
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
  
−
# [https://jira.fd.io/browse/CSIT-1240 CSIT Framework]
+
# [https://jira.fd.io/browse/CSIT-1240 Framework]
 
# [https://jira.fd.io/browse/CSIT-1238 Operations]
 
# [https://jira.fd.io/browse/CSIT-1238 Operations]
 
# [https://jira.fd.io/browse/CSIT-1236 Test]
 
# [https://jira.fd.io/browse/CSIT-1236 Test]
Line 148: Line 161:
 
|-
 
|-
 
| Xeon Skx testbeds
 
| Xeon Skx testbeds
−
| Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
+
| Make Skylake performance test coverage complete:
 +
i) Boost tests in 2-Node setups, complete 3-Node setups;
 +
 
 +
ii) Complete Memif/Container and Vhost-user/VM with latest QEMU;
 +
 
 +
iii) Push vpp-dev to Ubuntu 18.04.
 
|-
 
|-
 
| Arm testbeds
 
| Arm testbeds
Line 154: Line 172:
 
|-
 
|-
 
| Atom testbeds
 
| Atom testbeds
−
| Introduce Atom performance tests.
+
| Introduce Denverton and Rangeley performance tests.
 
|-
 
|-
 
| Better vhost, memif coverage
 
| Better vhost, memif coverage
−
| Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
+
| Produce more complete test data for NFV service density:
 +
i) Scaled-out Vhost-user/VM and Memif/Container tests;
 +
ii) Test the same packet paths and NF topologies: service chains, service pipelines;
 +
 
 +
iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
 +
 
 +
iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with.
 +
 
 +
v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth).
 
|-
 
|-
 
| VPP per patch performance tests
 
| VPP per patch performance tests
−
| Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
+
| Productise per VPP patch performance tests with change detection, prepare for voting:
 +
i) Improve detection accuracy and precision;
 +
 
 +
ii) Nail down current results variance;
 +
 
 +
iii) Apply improvements to continuous trending and (future) git auto-bisection.
 
|-
 
|-
 
| Trending Improved Detection
 
| Trending Improved Detection
−
| Make trending job use new Burst MRR trending tests for better anomaly detection. Currently postponed, as the algorithm detects performance changes not related to VPP code. We need heavy workarounds or way more predictable SUT behavior.
+
| Make trending job use new Burst MRR trending tests for better anomaly detection:
 +
i) Currently postponed, as the algorithm detects performance changes not related to VPP code.
 +
 
 +
ii) We need heavy workarounds or way more predictable SUT behavior.
 
|-
 
|-
 
| More VPP telemetry reported and analysed
 
| More VPP telemetry reported and analysed
Line 169: Line 203:
 
|-
 
|-
 
| Evolve throughput search
 
| Evolve throughput search
−
| Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
+
| Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:
 +
i) Compare MLRsearch with PLRsearch soak test results.
 
|-
 
|-
 
| General enhancements
 
| General enhancements
−
| General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.
+
| General CSIT and VPP performance test and infrastructure enhancements:
 +
i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm;
 +
 
 +
ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram;
 +
 
 +
iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI;
 +
 
 +
iv) Start migration from VAT to VPP Python API;
 +
 
 +
v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data.
 
|}
 
|}
  

Revision as of 14:13, 19 December 2018


Introduction

This page tracks release information for FD.io CSIT 1901. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.

Release Milestones

Milestone Date Deliverables
F0 2019-01-09 Test case keywords code complete. Only low-risk changes accepted.
RC1 2019-01-16 (F0+7) Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1.
RC2 2019-01-23 (RC1+7) Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2.
CSIT Release 2019-01-30 (RC2+7) CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release.
Report Publish 2019-02-13 (Rls+14) CSIT report published for VPP release.

Release Deliverables

Name Status Jira Category Description
Soak Tests Open 1901-Framework Productize soak throughput tests for given PLR and extended test duration.
VPP Patch Performance Tests Open 1901-Framework VPP patch vs. parent performance tests, degradation detection, gerrit voting. Auto-trigger subject to physical testbeds' capacity.
CSIT PAPI Support Open 1901-Framework Move CSIT code from VAT to PAPI, rewrite of Level-1 Keywords.
NFV Density Open 1901-Framework Experimental: infrastructure code for VNF and CNF density performance tests.
1n-arm VPP_Device Open 1901-Framework Bring up VPP_Device test environment on 1n-arm testbed(s).
1n-dnv VPP_Device Open 1901-Framework Bring up VPP_Device test environment on 1n-dnv testbed(s).
VPP_Path Open 1901-Framework Continue moving non-device-driver CSIT VIRL tests to VPP_Path (VPP make test) integration tests:

i) Required for functional acceptance of VPP feature path(s) driven by use case(s);

ii) See P1 and P2 markup in CSIT_VIRL migration progress.

Clock cycles per VPP node into CSIT-PAL Open 1901-Framework Use the new VPP stats infra to retrieve runtime counters, requires PAPI support in FD.io CSIT.
2n-arm Environment Open 1901-Framework Bring up 2n-arm testbeds in LFN FD.io lab.
2n-dnv Environment Open 1901-Framework Bring up 2n-dnv testbeds in LFN FD.io lab.
2n-arm Tests Open 1901-Test Port existing 2n-skx tests to 2n-arm environment.
2n-dnv Tests Open 1901-Test Port existing 2n-skx tests to 2n-dnv environment.
AVF Driver Tests Open 1901-Test 2n-skx: drive parity of test coverage with DPDK for all baseline and scale tests.
2n-skx Tests Open 1901-Test 2n-skx: drive test parity with 3n-skx, focusing on non-tunnel encapsulations.
3n-skx Tests Open 1901-Test If new NICs added, add baseline tests for these.
VPP_Device Xeon Open 1901-Test 1n-skx: add device-driver focused tests (migration away from VIRL).
VPP_Device Arm Open 1901-Test 1n-arm: add device-driver focused tests (migration away from VIRL).
VPP_Device Denverton Open 1901-Test 1n-dnv: add device-driver focused tests (migration away from VIRL).
NFV Density Tests Open 1901-Test Experimental: Density performance tests with VNF Service Chains, CNF Service Chains and CNF Service Pipelines.

Jira Task Tracking

All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:

  1. Framework
  2. Operations
  3. Test
  4. PAL
  5. VIRL
  6. HoneyComb
  7. [DMM]

Multi-Release Work Areas

Work Area Description
Xeon Skx testbeds Make Skylake performance test coverage complete:

i) Boost tests in 2-Node setups, complete 3-Node setups;

ii) Complete Memif/Container and Vhost-user/VM with latest QEMU;

iii) Push vpp-dev to Ubuntu 18.04.

Arm testbeds Introduce Arm performance tests.
Atom testbeds Introduce Denverton and Rangeley performance tests.
Better vhost, memif coverage Produce more complete test data for NFV service density:

i) Scaled-out Vhost-user/VM and Memif/Container tests; ii) Test the same packet paths and NF topologies: service chains, service pipelines;

iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.

iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with.

v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth).

VPP per patch performance tests Productise per VPP patch performance tests with change detection, prepare for voting:

i) Improve detection accuracy and precision;

ii) Nail down current results variance;

iii) Apply improvements to continuous trending and (future) git auto-bisection.

Trending Improved Detection Make trending job use new Burst MRR trending tests for better anomaly detection:

i) Currently postponed, as the algorithm detects performance changes not related to VPP code.

ii) We need heavy workarounds or way more predictable SUT behavior.

More VPP telemetry reported and analysed API based consumption of VPP telemetry including existing general counters, and future extended per node counters.
Evolve throughput search Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:

i) Compare MLRsearch with PLRsearch soak test results.

General enhancements General CSIT and VPP performance test and infrastructure enhancements:

i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm;

ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram;

iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI;

iv) Start migration from VAT to VPP Python API;

v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data.

External Dependencies

  1. No known external dependencies.
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/csit1901_plan&oldid=8807"