CSIT/csit1810 plan

Please be aware that this page is work-in-progress and more info is being added continuously.

Deliverables Tracking

All CSIT release deliverables should be tracked in FDio CSIT Jira using one of the following Jira Epic categories:

1. CSIT Framework
2. Operations
3. Test
4. PAL
5. VIRL
6. HoneyComb
7. [DMM]

Focus Areas

Work Area	Description
Xeon Skylake tests	Make Skylake performance test coverage complete: i) Boost tests in 2-Node setups, complete 3-Node setups; ii) Complete Memif/Container and Vhost-user/VM with latest greatest QEMU etc; iii) Push vpp-dev to Ubuntu 18.04.
Arm tests	Introduce Arm performance tests.
Atom tests	Introduce Atom performance tests.
Better vhost, memif coverage	Make CSIT produce more complete test data for scaled-out Vhost-user/VM and Memif/Container: i) Complete same packet paths and topologies for a low number of VMs and Containers, then scale-up VM and Container numbers; ii) See if we can isolate the actual cost of Vhostuser-virtio and Memif-Memif virtual interfaces based on the test and system telemetry.
VPP per patch performance tests	Productise per VPP patch performance tests with change detection, prepare for voting: i) Improve detection accuracy and precision; ii) Nail down current results variance; iii) Apply improvements to continuous trending and (future) git auto-bisection.
Evolve throughput search	Build upon MLRsearch experience vs. ordinary binary search: i) New POC for extended soak test for validating NDR (zero packet-loss-ratio PLR) and(?) PDR (non-zero PLR).
General enhancements	General CSIT and VPP performance test and infrastructure enhancements: i) Productize VPP_Device container-based functional tests in 1-Node Skylake testbeds, assist with the same for Arm; ii) Add proper packet latency measurements with T-Rex HDRhistogram, push T-Rex to productize HDRh'gram; iii) Start using the new VPP stats infra for per test counters and "gauges" collection incl. "show runtime", instead of VPP show CLI; iv) Start migration from VAT to VPP Python API; v) Nail down "broken"/not-performing VPP data plane feature arcs (incl. multi-threading) indicated by CSIT-18.07 results data.

Release Deliverables

Name	Status	Jira Category	Description
2n-Skx tests	WIP	1810-Test	Adding baseline and scale tests to new 2n-Skx (2-Node) testbeds.
VPP_Device	WIP	1810-Framework	Initial CI/CD implementation of Container based VPP integration tests for functional acceptance of VPP device driver interactions, running on 1-Node testbeds: 1n-skx, 1n-arm.
VPP_Path	Open	1810-Framework	Continuing migration of CSIT VIRL tests to VPP-make_test VPP integration tests for functional acceptance of VPP feature path(s) driven by use case(s). See P1 and P2 markup in VPP_Path: CSIT_VIRL migration progress.
Performance trending PAL evolution	WIP	1810-PAL	Make PAL use new Burst MRR trending tests once productized for better anomaly detection.
Performance change detection	WIP	1810-Framework	Introduction of Burst MRR tests to trending.
Per VPP patch performance tests	WIP	1810-Framework	Per VPP patch vs. parent performance tests and voting (-1/0/+1).
Clock cycles per VPP node into CSIT-PAL	Open	1810-PAL	Use the new VPP stats infra to retrieve runtime counters instead of using "show run".

Release Milestones

Milestone	Date	Deliverables
F0	2018-10-03	Test case keywords code complete. Only low-risk changes accepted.
RC1	2018-10-10 (F0+7)	Code complete. Pull first release branch. Only bug fixes accepted in release branch. Date aligned with VPP RC1.
RC2	2018-10-17 (RC1+7)	Dry-run testing begins of VPP RC2, performance and functional. Date aligned with VPP RC2.
CSIT Release	2018-10-24 (RC2+7)	CSIT release complete. VPP release testing starts. Date aligned with VPP Formal Release.
Report Publish	2018-11-07 (Rls+14)	CSIT report published for VPP release.