Difference between revisions of "CSIT/csit1908 plan"

From fd.io
Jump to: navigation, search
(Marked as done: gating, tcp.)
 
(8 intermediate revisions by 4 users not shown)
Line 42: Line 42:
 
! Jira Category
 
! Jira Category
 
! Description
 
! Description
 +
|-
 +
| VAT to PAPI migration
 +
| WIP
 +
| [https://jira.fd.io/browse/CSIT-1333 Framework ]
 +
| Moving CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords
 +
|-
 +
| IPsec tests
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1526 Performance]
 +
| Updated IPsec base tests with new crypto backends and ciphers, scale tests >>1k tunnel interfaces.
 +
|-
 +
| VXLAN VNF Chains
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1526 Performance]
 +
| Extension of NFV density tests for VM with VXLAN encap to SUT/VPP.
 +
|-
 +
| VPP API Gate-keeping
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1524 Framework]
 +
| VPP API checks
 +
|-
 +
| TCP tests
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1526 Performance]
 +
| Refactoring of TCP tests.
 +
|-
 +
| Arm perf framework
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1524 Framework]
 +
| Aiming to have Arm daily performance jobs running for csit-vpp-perf-mrr-daily.
 +
|-
 +
| Arm perf tests
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1526 Performance]
 +
| Aiming to have a number of Arm performance tests running.
 +
|-
 +
| Arm vpp_device framework
 +
| Done*
 +
| [https://jira.fd.io/browse/CSIT-1524 Framework]
 +
| Aiming to have Arm functional vpp_device jobs running for csit-vpp-verify and vpp-csit-verify.
 +
<nowiki/>*NOTE: 1n-tx2 testbed is incomplete, missing 1 server.
 +
|-
 +
| Arm vpp_device tests
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1526 Performance]
 +
| Aiming to have a number of Arm functional vpp_device tests running.
 +
|-
 +
| Denverton perf framework
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1524 Framework]
 +
| Atom Denverton daily performance jobs running for csit-vpp-perf-mrr-daily.
 +
|-
 +
| Denverton perf tests
 +
| Done
 +
| [https://jira.fd.io/browse/CSIT-1526 Performance]
 +
| Atom Denverton performance tests running, full set.
 
|-
 
|-
 
| Name
 
| Name
 
| Status
 
| Status
| Category
+
| Jira Category
 
| Description
 
| Description
 
|-
 
|-
 
| Name
 
| Name
 
| Status
 
| Status
| Category
+
| Jira Category
 
| Description
 
| Description
 
|}
 
|}
Line 58: Line 114:
 
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
 
All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:
  
  Framework
+
  [https://jira.fd.io/browse/CSIT-1524 Framework]
  CI process
+
  [https://jira.fd.io/browse/CSIT-1522 CI process]
  Performance
+
  [https://jira.fd.io/browse/CSIT-1526 Performance]
  Device
+
  [https://jira.fd.io/browse/CSIT-1523 Device]
  Methodology
+
  [https://jira.fd.io/browse/CSIT-1525 Methodology]
  Telemetry
+
  [https://jira.fd.io/browse/CSIT-1528 Telemetry]
  Tools
+
  [https://jira.fd.io/browse/CSIT-1529 Tools]
  Presentation
+
  [https://jira.fd.io/browse/CSIT-1527 Presentation]
  [[Honeycomb]]
+
  [https://jira.fd.io/browse/CSIT-1517 Honeycomb]
[htts://jira.fd.io/browse/CSIT-1050 Aarch64]
+
[https://jira.fd.io/browse/CSIT-1050 Aarch64]
  
 
== Multi-Release Work Areas ==
 
== Multi-Release Work Areas ==

Latest revision as of 13:26, 2 September 2019


Introduction

This page tracks release information for FD.io CSIT-1908. It is updated regularly by hand. Real-time information is available in FD.io CSIT code repository and auto-generated docs.

Release Milestones

Milestone Date Deliverables
F0 2019-07-31 Test case keywords code complete. Only low-risk changes accepted.
RC1 2019-08-07 (F0+7) Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1. Start dry-runs to identify CSIT gaps on less frequently run tests.
RC2 2019-08-14 (RC1+7) Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2.
CSIT Release 2019-08-21 (RC2+7) CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release.
Report Publish 2019-09-04 (Rls+14) CSIT report published for VPP release.

Release Deliverables

Name Status Jira Category Description
VAT to PAPI migration WIP Framework Moving CSIT code from VAT to PAPI APIs, re-write of Level-1 Keywords
IPsec tests Done Performance Updated IPsec base tests with new crypto backends and ciphers, scale tests >>1k tunnel interfaces.
VXLAN VNF Chains Done Performance Extension of NFV density tests for VM with VXLAN encap to SUT/VPP.
VPP API Gate-keeping Done Framework VPP API checks
TCP tests Done Performance Refactoring of TCP tests.
Arm perf framework Done Framework Aiming to have Arm daily performance jobs running for csit-vpp-perf-mrr-daily.
Arm perf tests Done Performance Aiming to have a number of Arm performance tests running.
Arm vpp_device framework Done* Framework Aiming to have Arm functional vpp_device jobs running for csit-vpp-verify and vpp-csit-verify.

*NOTE: 1n-tx2 testbed is incomplete, missing 1 server.

Arm vpp_device tests Done Performance Aiming to have a number of Arm functional vpp_device tests running.
Denverton perf framework Done Framework Atom Denverton daily performance jobs running for csit-vpp-perf-mrr-daily.
Denverton perf tests Done Performance Atom Denverton performance tests running, full set.
Name Status Jira Category Description
Name Status Jira Category Description

Jira Task Tracking

All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:

Framework
CI process
Performance
Device
Methodology
Telemetry
Tools
Presentation
Honeycomb
Aarch64

Multi-Release Work Areas

Work Area Description
Xeon Skx testbeds Make Skylake performance test coverage complete:

i) Boost tests in 2-Node setups, complete 3-Node setups;

ii) Complete Memif/Container and Vhost-user/VM with latest QEMU;

iii) Push vpp-dev to Ubuntu 18.04.

Arm testbeds Introduce Arm performance tests.
Atom testbeds Introduce Denverton and Rangeley performance tests.
Better vhost, memif coverage Produce more complete test data for NFV service density:

i) Scaled-out Vhost-user/VM and Memif/Container tests; ii) Test the same packet paths and NF topologies: service chains, service pipelines;

iii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.

iv) Test with VM and Containers running on a single Processor (single socket) with no core oversubscription and with.

v) Extend the test over two Processors to quantify impact of UPI latency (and bandwidth).

VPP per patch performance tests Productise per VPP patch performance tests with change detection, prepare for voting:

i) Improve detection accuracy and precision;

ii) Nail down current results variance;

iii) Apply improvements to continuous trending and (future) git auto-bisection.

Trending Improved Detection Make trending job use new Burst MRR trending tests for better anomaly detection:

i) Currently postponed, as the algorithm detects performance changes not related to VPP code.

ii) We need heavy workarounds or way more predictable SUT behavior.

More VPP telemetry reported and analysed API based consumption of VPP telemetry including existing general counters, and future extended per node counters.
Evolve throughput search Build upon MLRsearch and PLRsearch experience vs. ordinary binary search:

i) Compare MLRsearch with PLRsearch soak test results.

General enhancements General CSIT and VPP performance test and infrastructure enhancements:

i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm;

ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram;

iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI;

iv) Start migration from VAT to VPP Python API;

v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.10 results data.

External Dependencies

  1. No known external dependencies.