Difference between revisions of "CSIT/PerfTestPlan"

From fd.io
Jump to: navigation, search
 
(48 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
'''WORK IN PROGRESS - PLEASE TREAT IT AS SUCH'''
 +
 
__TOC__
 
__TOC__
−
==Performance testing - Jobs==
 
  
−
[https://wiki.fd.io/view/CSIT/Jobs CSIT Jenkins Jobs for Performance Tests]
+
=CSIT development plan - background=
 +
 
 +
CSIT test development priorities are aligned with VPP releases timeline, with relative dev work prioritization for each VPP release driven by perceived importance of test verification for VPP functionality and performance. Specific CSIT dev work priorities are driven by rough consensus within the project and contributors and committers availability and willingness to do work.
 +
 
 +
Following priority tagging scheme is used in this CSIT wiki and FD.io jira for CSIT:
 +
 
 +
* [$VppRlsNo-$RelPrio]
 +
** VppRlsNo - aligned VPP release number - 16.09, 16.12, 17.03.
 +
** RelPrio - CSIT release priority - P0 for Must Have, P1 for Should Have, P2 for Could Have
 +
* CSIT dev work tagging examples:
 +
** [1609-P0] - CSIT Must Have for vpp-v1609
 +
** [1609-P1] - CSIT Should Have for vpp-v1609
 +
** [1609-P2] - CSIT Could Have for vpp-v1609
 +
 
 +
CSIT dev work plan execution tracked on [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=1&projectKey=CSIT CSIT jira], with priority tags encoded as Jira labels for easy of tracking and reporting.
 +
 
 +
=CSIT performance tests - dev plan=
 +
 
 +
CSIT performance test cases devevelopment work is grouped and tracked across the following VPP functional areas:
 +
 
 +
# NIC devices and drivers
 +
# IPv4 data plane
 +
# IPv4 control plane
 +
# IPv4 encapsulations
 +
# IPv4 telemetry
 +
# IPv6 data plane
 +
# IPv6 control plane
 +
# IPv6 encapsulations
 +
# IPv6 telemetry
 +
# Ethernet L2 data plane
 +
# Ethernet L2 control plane
 +
# Ethernet L2 encapsulations
 +
# Ethernet L2 telemetry
 +
# MPLS data plane
 +
# NSH data plane
 +
 
 +
CSIT team is tracking completed CSIT dev work per above area (completed = test cases running in FD.io), as well as top-level test coverage metric [%] of VPP functions. VPP code development dependencies are marked with [Blocked-by VPP-XXX]. Performance tests require the functional tests to be completed - see sub-sections "Functional tests preconditions".
 +
 
 +
CSIT performance tests are split into the two main groups:
 +
* NDR(PDR) throughput discovery - tests searching for NDR non-drop rate throughput or PDR partial-drop rate throughput for specified configuration. Also referred to as "long" tests.
 +
* refNDR throughput verification - tests verifying code patches against the reference NDR throughput. Also referred to as "short" tests.
 +
 
 +
==Infrastructure - NIC devices and drivers==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = infra AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = infra AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = infra AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1606-DONE] NIC Intel X520 2p10GE - DPDK niantic driver, 2 ports within the same NIC.
 +
# [1609-P0] NIC Intel XL710 2p40GE - DPDK i40e driver, 2 ports within the same NIC.
 +
# [1609-DONE] VPP vhost-user - VPP vhost-user driver, virtio in VM.
 +
# [1609-P1] NIC Intel X710 2p40GE - DPDK i40e driver, 2 ports within the same NIC.
 +
# [1609-DONE] DPDK vhost-user - DPDK vhost-user driver, virtio in VM.
 +
# [1612-P0] vhost-user - XXX vhost-user driver, virtio in VM, box-full test with NxVMs.
 +
# [1612-P2] continue to expand VPP driver coverage - more NIC-DPDK drivers, other TBD.
 +
 
 +
==IPv4 data plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4dp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4dp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4dp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
'''Baseline performance'''
 +
 
 +
# [1609-P0] IPv4 routing RFC791 - baseline - NIC-to-NIC(XL710).
 +
# [1609-DONE] IPv4 routing RFC791 - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
 +
# [1609-P1] VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(X520).
 +
# [1609-P1] VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(XL710).
 +
# [1609-P1] extranet VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(X520).
 +
# [1609-P1] extranet VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(XL710).
 +
# [1606-DONE] IPv4 routing RFC791 - baseline - NIC-to-NIC(X520).
 +
 
 +
'''FIB performance'''
 +
 
 +
Large-scale FIB tests
 +
# [1609-DONE] same prefix length.
 +
# [1609-P1] varying prefix lengths.
 +
 
 +
'''Security performance'''
 +
 
 +
# [1606-DONE] security - iacl with n-tuple classification, ip4.
 +
# [1606-DONE] security - cop white-/black-list ip4 src addr filtering.
 +
 
 +
'''QoS performance'''
 +
 
 +
# [1609-REVIEW] QoS - ingress policing - rate metering, marking and limiting - 2r3c-(cb,ca) policers, RFC2698.
 +
# [1609-P1] QoS - ingress policing - rate metering, marking and limiting - 1r2c-(cb,ca) policers, RFC2697.
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P2] security - source IPv4 Reverse Path Forwarding (RPF).
 +
    # [1609-P1] routing multipath (ECMP, UECMP) - baseline.
 +
    # [1609-P1] [Blocked-by VPP-xxx] carrier grade nat 44, CGN44, RFC6598, RFC6888.
 +
    # [1609-P2] [Blocked-by VPP-xxx] security - stateful firewall.
 +
 
 +
==IPv4 control plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4cp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4cp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4cp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] Proxy ARP.
 +
    # [1609-P0] IKEv2.
 +
    # [1609-P1] DHCPv4 proxy.
 +
    # [1606-DONE] ARP.
 +
    # [1606-DONE] ICMPv4.
 +
    # [1606-DONE] DHCPv4 client.
 +
 
 +
==IPv4 encapsulations==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4enc AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4enc AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4enc AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
'''Baseline performance'''
 +
 
 +
# [1609-P1] ip4-gre-ip4 - baseline main fib.
 +
# [1609-DONE] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] ip4-ipsec-ip4 - baseline main fib.
 +
    # [1609-P0] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), virt2lisp.
 +
    # [1609-P0] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
 +
    # [1609-P0] ip4-ipsec-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
 +
    # [1609-P0] ip4-ipsec-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
 +
    # [1606-DONE] ip4-gre-ip4 - baseline main fib.
 +
 
 +
==IPv4 telemetry==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4tel AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4tel AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip4tel AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] IPv4 IPFIX - stats collection and export - baseline.
 +
    # [1609-P0] IPv4 IPFIX - stats collection and export - functional scale.
 +
    # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
 +
    # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.
 +
 
 +
==IPv6 data plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6dp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6dp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6dp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
'''Baseline performance'''
 +
 
 +
# [1609-P0] IPv6 routing RFC2460 - baseline - NIC-to-NIC(XL710).
 +
# [1609-P0] IPv6 routing RFC2460 - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
 +
# [1609-P1] VPNv6 routed forwarding (VRF) - baseline - NIC-to-NIC(X520).
 +
# [1609-P1] VPNv6 routed forwarding (VRF) - baseline - NIC-to-NIC(XL710).
 +
# [1609-P2] extranet VPN routed forwarding (VRF) - baseline.
 +
# [1606-DONE] IPv6 routing RFC2460 - baseline - NIC-to-NIC(X520).
 +
 
 +
'''FIB performance'''
 +
 
 +
Large-scale FIB tests
 +
# [1609-DONE] same prefix length.
 +
# [1609-P1] varying prefix lengths.
 +
 
 +
'''Security performance'''
 +
 
 +
# [1609-DONE] security - iacl with n-tuple classification, ip6.
 +
# [1606-DONE] security - cop white-/black-list ip6 src addr filtering.
 +
 
 +
'''QoS performance'''
 +
 
 +
# [1609-P1] QoS - ingress policing - rate metering, marking and rate-limiting -1r2c-(cb,ca), 2r3c-(cb,ca) policers, RFC2475, RFC2697, RFC2698.
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] security - source IPv6 Reverse Path Forwarding (RPF).
 +
    # [1609-P1] routing multipath (ECMP, UECMP) - baseline.
 +
    # [1609-P2] [Blocked-by VPP-xxx] security - stateful firewall.
 +
 
 +
==IPv6 control plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6cp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6cp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6cp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] router advertisement.
 +
    # [1609-P1] DHCPv6 proxy.
 +
    # [1609-P2] [Blocked-by VPP-xxx] ip6-ioam, in-band OAM.
 +
    # [1606-DONE] neighbor discovery, RFC4861.
 +
    # [1606-DONE] ICMPv6.
 +
 
 +
==IPv6 encapsulations==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6enc AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6enc AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6enc AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-DONE] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.
 +
# [1609-DONE] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] ip4-o-ip6 softwire.
 +
    # [1609-P0] ip6-ipsec-ip6.
 +
    # [1609-P0] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), virt2lisp.
 +
    # [1609-P0] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
 +
    # [1609-P1] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), virt2lisp.
 +
    # [1609-P1] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
 +
    # [1609-P0] ip6-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
 +
    # [1609-P0] ip6-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
 +
    # [1609-P1] ip4-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
 +
    # [1609-P1] ip4-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
 +
    # [1609-P1] [Blocked-by VPP-xxx] ip6-segment-routing.
 +
 
 +
==IPv6 telemetry==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6tel AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6tel AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = ip6tel AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] IPv6 IPFIX - stats collection and export - baseline.
 +
    # [1609-P0] IPv6 IPFIX - stats collection and export - functional scale.
 +
    # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
 +
    # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.
 +
 
 +
==Ethernet L2 data plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2dp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2dp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2dp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
'''Baseline performance'''
 +
 
 +
# [1609-P0] L2 Cross-connect - baseline - NIC-to-NIC(XL710).
 +
# [1609-P0] Bridge domain - baseline - NIC-to-NIC(XL710).
 +
# [1609-DONE] L2 Cross-connect - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
 +
# [1609-DONE] Bridge domain - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
 +
# [1606-DONE] L2 Cross-connect - baseline - NIC-to-NIC(X520).
 +
# [1606-DONE] Bridge domain - baseline - NIC-to-NIC(X520).
 +
 
 +
'''FIB performance'''
 +
 
 +
Large-scale FIB tests
 +
 
 +
# [1609-P1] [Blocked-by VPP-201] MAC addresses.
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] Bridge domain - with VLAN tag acrobatics 1-to-1, 1-to-2, 2-to-1, 2-to-2.
 +
    # [1609-P1] Integrated Routing-Bridging (IRB) - baseline.
 +
    # [1606-DONE] L2 Cross-connect - with VLAN tag acrobatics 1-to-1, 1-to-2, 2-to-1, 2-to-2.
 +
 
 +
==Ethernet L2 control plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2cp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2cp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2cp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
# [1606-DONE] MAC learning.
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P2] [Blocked-by VPP-xxx] CDP/LLDP.
 +
    # [1606-DONE] MAC split-horizon.
 +
 
 +
==Ethernet L2 encapsulations==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2enc AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2enc AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2enc AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-P0] l2-eth-lispgpe-ip6 - L2BD.
 +
# [1609-P0] l2-eth-lispgpe-ip4 - L2BD.
 +
# [1609-P1] l2-vxlan-ipv4 - with bridge-domain.
 +
# [1609-DONE] l2-vxlan-ipv4 - with L2 cross-connect.
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P0] l2-eth-lispgpe-ip6 - L2BD, L2XC.
 +
    # [1609-P0] l2-eth-lispgpe-ip4 - L2BD, L2XC.
 +
    # [1609-P0] l2eth-gre-ip4-ipsec-ip4-eth - L2BD, L2XC.
 +
    # [1609-P2] l2-eth-l2tpv3-ip4 - L2BD, L2XC.
 +
    # [1609-P2] l2-eth-l2tpv3-ip6 - L2BD, L2XC.
 +
    # [1609-P2] l2-eth-gre-ip4 - L2BD, L2XC.
 +
    # [1609-P2] l2-eth-gre-ip6 - L2BD, L2XC.
 +
 
 +
==L2 telemetry==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2tel AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2tel AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = l2tel AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
 +
 
 +
    # [1609-P1] L2 FIB - stats collection and export - baseline.
 +
    # [1609-P2] L2 FIB - stats collection and export - functional scale.
 +
    # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
 +
    # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.
 +
 
 +
==MPLS data plane==
 +
 
 +
'''1609-P0'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = mplsdp AND labels = 1609-p0'></jira>
 +
 
 +
'''1609-P1'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = mplsdp AND labels = 1609-p1'></jira>
 +
 
 +
'''1609-P2'''
 +
 
 +
<jira jql='project = CSIT AND labels = perf AND labels = mplsdp AND labels = 1609-p2'></jira>
 +
 
 +
Indicative coverage = <to_be_added>
 +
 
 +
# [1609-Px] <to_be_added>
 +
 
 +
'''Functional tests preconditions'''
 +
 
 +
Below copied from CSIT functional tests plan - provided for reference only:
  
−
==Performance testing - Testbeds==
+
    # [1609-P2] vpnv4-mpls-gre-ip4.
 +
    # [1609-P2] mpls-o-eth.
  
−
[https://wiki.fd.io/view/CSIT/CSIT_LF_testbed CSIT LF testbed for Performance Tests]: all 3 LF physical testbeds are all live
+
==NSH data plane==
  
−
==Performance testing - Test Cases==
+
'''1609-P0'''
  
−
===CSIT test suite list for VPP v16.06===
+
<jira jql='project = CSIT AND labels = perf AND labels = nshdp AND labels = 1609-p0'></jira>
  
−
[https://wiki.fd.io/view/CSIT/VPP-16.06_Test_Report#Long_Performance_Tests_-_NDR_and_PDR_Search Test Suites]
+
'''1609-P1'''
  
−
===CSIT test suite list for VPP v16.09===
+
<jira jql='project = CSIT AND labels = perf AND labels = nshdp AND labels = 1609-p1'></jira>
  
−
[TO-DO]
+
'''1609-P2'''
  
−
==Performance Test Cases - Plan and Status==
+
<jira jql='project = CSIT AND labels = perf AND labels = nshdp AND labels = 1609-p2'></jira>
  
−
For CSIT project to work and be useful, it is proposed that CSIT code and test cases development should be done in coordination with VPP Release Candidate content and release schedule.
+
Indicative coverage = <to_be_added>
−
CSIT development work needs to be prioritized based on FD.io community needs and feedback, and csit-dev work distributed. To facilitate this, we compiled an initial list of proposed performance test cases that must, should, could be tested. Each csit-dev work item is prefixed with status and/or proposed priority relative to FD.io VPP release:
+
−
* [Pn-Rm] – Proposed coverage of VPP functionality by FD.io CSIT functional and performance test cases.
+
−
** '''P'''n – CSIT development and integration '''priority'''; n=0,1,2, 0-for-MUST, 1-for-SHOULD, 2-for-COULD.
+
−
** '''R'''m – VPP major '''release''', R1 stands for the first FD.io VPP release, R2 for the 2nd FD.io VPP release.
+
−
* [DONE-Rm] - Done and working test cases, executed for VPP Rm code.
+
−
* [TBD] - Development work scheduling priority to-be-defined.
+
  
−
Plan execution tracked on [https://jira.fd.io/secure/RapidBoard.jspa?rapidView=1&projectKey=CSIT CSIT jira]
+
# [1609-Px] <to_be_added>
  
−
===CSIT-DEV plan for VPP v16.06===
+
'''Functional tests preconditions'''
  
−
# Long throughput benchmarks - libraries
+
Below copied from CSIT functional tests plan - provided for reference only:
−
#* L2/IPv4/IPv6 NonDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/515/ Gerrit #515]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/564/ Gerrit #564]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/603/ Gerrit #603]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/671/ Gerrit #671]
+
−
#* L2/IPv4/IPv6 PartialDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
# Long throughput benchmarks - test cases
+
−
#* IPv4 baseline NonDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#* L2 baseline NonDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#* IPv6 baseline NonDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-2 Jira CSIT-2]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-2 Jira CSIT-2]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-2 Jira CSIT-2]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-2 Jira CSIT-2]
+
−
#* IPv4 baseline PartialDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#* L2 baseline PartialDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#* IPv6 baseline PartialDropRate search
+
−
#** Linear search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Binary search (rfc2544) - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** BESTofN/WORSTofN search - [https://gerrit.fd.io/r/#/c/1092/ Gerrit #1092] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
#** Combined binary+linear search - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-67 Jira CSIT-67]
+
−
# Short throughput benchmarks - libraries
+
−
#* Short IPv4 throughput test against reference NDR - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#* Short L2 throughput test against reference NDR - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#* Short IPv6 throughput test against reference NDR - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-2 Jira CSIT-2]
+
−
# Short throughput benchmarks - test cases
+
−
#* Short IPv4 throughput - routed-forwarding - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#* Short L2 throughput - bridge-domain, l2-xconnect - [https://gerrit.fd.io/r/#/c/574/ Gerrit #574]
+
−
#* Short IPv6 throughput - routed-forwarding - [https://gerrit.fd.io/r/#/c/824/ Gerrit #824] [https://jira.fd.io/browse/CSIT-2 Jira CSIT-2]
+
−
# 3-node physical testbed - libraries
+
−
#* Create host topology files for tb2 - [https://gerrit.fd.io/r/#/c/954/ Gerrit #954] [https://jira.fd.io/browse/CSIT-8 Jira CSIT-8]
+
−
#* Extend host topology with NIC type filtering - [https://gerrit.fd.io/r/#/c/808/ Gerrit #808] [https://jira.fd.io/browse/CSIT-1 Jira CSIT-1]
+
−
#* Create host topology files for tb1, tb3 - [https://gerrit.fd.io/r/#/c/1202/ Gerrit #1202] [https://gerrit.fd.io/r/#/c/1042/ Gerrit #1042] [https://jira.fd.io/browse/CSIT-8 Jira CSIT-8]
+
−
# Multi-core multi-thread tests - libraries
+
−
#* Multicore tests - [https://gerrit.fd.io/r/#/c/710/ Gerrit #710]
+
−
# Security feature
+
−
#* Classify IPv4 iACL - [https://gerrit.fd.io/r/#/c/1272/ Gerrit #1272] [https://jira.fd.io/browse/CSIT-119 Jira CSIT-119]
+
−
#* IPv6 COP - [https://gerrit.fd.io/r/#/c/1262/ Gerrit #1262] [https://jira.fd.io/browse/CSIT-109 Jira CSIT-109]
+
−
#* IPv4 COP - [https://gerrit.fd.io/r/#/c/1254/ Gerrit #1254] [https://jira.fd.io/browse/CSIT-105 Jira CSIT-105]
+
−
# L2 encapsulations
+
−
#* l2-vlan-dot1q - [https://gerrit.fd.io/r/#/c/973/ Gerrit #973] [https://jira.fd.io/browse/CSIT-59 Jira CSIT-59]
+
−
#* l2-vlan-dot1ad - [https://gerrit.fd.io/r/#/c/1250/ Gerrit #1250] [https://jira.fd.io/browse/CSIT-60 Jira CSIT-60]
+
−
# VPP vNet topology baselines - libraries
+
−
#* Host stack VPP baseline - Phy1-VPP-Phy2
+
−
# vNet VPP topologies - libraries
+
−
#* Phy1-VPP-Phy2
+
−
# Traffic generator drivers
+
−
#* T-Rex
+
−
#** T-Rex driver
+
−
#** TRex support for packet delay and packet delay variation [https://gerrit.fd.io/r/#/c/1209/ Gerrit #1209] [https://jira.fd.io/browse/CSIT-102 Jira CSIT-102]
+
−
# Other work items
+
−
#* Collect VPP counter statistics - [https://gerrit.fd.io/r/#/c/1195/ Gerrit #1195] [https://jira.fd.io/browse/CSIT-71 Jira CSIT-71]
+
−
#* Tune VPP performance by modifying startup config - [https://gerrit.fd.io/r/#/c/1071/ Gerrit #1071] [https://gerrit.fd.io/r/#/c/1433/ Gerrit #1433] [https://jira.fd.io/browse/CSIT-73 Jira CSIT-73]
+
−
#* Replace linear search with binary search - [https://gerrit.fd.io/r/#/c/1189/ Gerrit #1189] [https://jira.fd.io/browse/CSIT-97 Jira CSIT-97]
+
−
#* Troubleshooting VPP - [https://jira.fd.io/browse/CSIT-118 CSIT-118]
+
−
#* Unified performance trend dashboard [https://gerrit.fd.io/r/#/c/1493/ Gerrit #1493] [https://gerrit.fd.io/r/#/c/1787/ Gerrit #1787] [https://gerrit.fd.io/r/#/c/1413/ Gerrit #1413] [https://jira.fd.io/browse/CSIT-135 Jira CSIT-135] [https://jira.fd.io/browse/CIMANAGE-9 Jira CIMANAGE-9] [https://jira.fd.io/browse/CIMANAGE-10 Jira CIMANAGE-10]
+
  
−
===CSIT-DEV plan for VPP v16.09===
+
    # [1609-P1] ip4-nsh-gre-ip4.
 +
    # [1609-P1] ip4-nsh-vxlan-gpe-ip4.
  
−
# Large-scale FIB tests - libraries
+
END OF NOTE
−
#* [P0-R2] IPv4 same prefix length - [https://gerrit.fd.io/r/#/c/1613/ Gerrit #1613] [https://jira.fd.io/browse/CSIT-169 Jira CSIT-169]
+
−
#* [P1-R2] IPv4 varying prefix lengths
+
−
#* [P0-R2] IPv6 same prefix length - [https://gerrit.fd.io/r/#/c/1664/ Gerrit #1664] [https://jira.fd.io/browse/CSIT-179 Jira CSIT-179]
+
−
#* [P1-R2] IPv6 varying prefix lengths
+
−
#* [P0-R2] MAC addresses - [https://gerrit.fd.io/r/#/c/1862/ Gerrit #1862]
+
−
# Packet encapsulations - test cases
+
−
#* IP4 encapsulations
+
−
#** [P1-R2] ip4-gre-ip4
+
−
#** [P1-R2] ip4-ipsec-ip4
+
−
#* IP6 encapsulations
+
−
#** [P1-R2] ip4-o-ip6 softwire
+
−
#*** [P1-R2] lightweight46 [rfc7596]
+
−
#** [P2-R2] map-e [rfc7597]
+
−
#** [P2-R2] map-t [rfc7599]
+
−
#* L2 encapsulations
+
−
#** [P1-R2] l2-vxlan-ipv4
+
−
#** [TBD] l2-eth-l2tpv3-ip4
+
−
#** [TBD] l2-eth-l2tpv3-ip6
+
−
#** [TBD] l2-eth-gre-ip4
+
−
#** [TBD] l2-eth-gre-ip6
+
−
#* [TBD] ip6-segment-routing
+
−
#* [P1-R2] ip6-ipsec-ip6
+
−
# Security feature
+
−
#* [TBD] IPv6 iACL
+
−
# VPP vNet topology baselines - libraries
+
−
#* Host stack baseline - Phy1-App-Phy2
+
−
#** [P0-R2] Reference host user-mode app, L3FWD-xconnect, DPDK driver per NIC type - [https://gerrit.fd.io/r/#/c/808/ Gerrit #808] [https://gerrit.fd.io/r/#/c/1267/ Gerrit #1267] [https://jira.fd.io/browse/CSIT-1 Jira CSIT-1] [https://jira.fd.io/browse/CSIT-106 Jira CSIT-106]
+
−
#* Host stack VPP baseline - Phy1-VPP-Phy2
+
−
#** [TBD] Host VPP IPv4 baseline config, DPDK driver per NIC type
+
−
#* Guest stack baseline - Phy1-pcipt-AppVM-pcipt-Phy2
+
−
#** [TBD] Reference guest user-mode app, L3FWD-xconnect, DPDK driver per NIC type
+
−
#* Guest stack VPP baseline - Phy1-pcipt-VPPVM-pcipt-Phy2
+
−
#** [TBD] VPP IPv4 baseline config, DPDK driver per NIC type
+
−
#* [P0-R2] Host and guest stack baseline - Phy1-VPP-AppVM-VPP-Phy2  - [https://gerrit.fd.io/r/#/c/1319/ Gerrit #1319] [https://jira.fd.io/browse/CSIT-63 Jira CSIT-63]
+
−
#** [TBD] VPP IPv4 baseline config, vhost-user to AppVM
+
−
# vNet VPP topologies - libraries
+
−
#* [P0-R2] Phy1-VPP-VM-VPP-Phy2 - [https://gerrit.fd.io/r/#/c/1319/ Gerrit #1319] [https://jira.fd.io/browse/CSIT-63 Jira CSIT-63]
+
−
# Traffic generator drivers
+
−
#* IXIA IxNetwork driver
+
−
#** [P1-R2] driver design - [https://gerrit.fd.io/r/#/c/1561/ Gerrit #1561] [https://jira.fd.io/browse/CSIT-66 Jira CSIT-66]
+
−
#** [P1-R2] driver implementation - [https://gerrit.fd.io/r/#/c/1561/ Gerrit #1561] [https://jira.fd.io/browse/CSIT-66 Jira CSIT-66]
+
−
#*** [TBD] test case setup, teardown
+
−
#*** [TBD] traffic stream configuration
+
−
#*** [TBD] statistics retrieval
+
−
#** latency measurements
+
−
#*** [TBD] feasibility
+
−
#*** [TBD] implementation
+
−
#** [TBD] integration with CSIT tests
+
−
#* Moongen driver
+
−
#** [TBD] verify requirements and available HW against latest MoonGen codebase
+
−
#** [TBD] CSIT integration
+
−
#* LWPG driver
+
−
#** [TBD] driver design
+
−
#** [TBD] driver implementation
+
−
#*** [TBD] test case setup, teardown
+
−
#*** [TBD] traffic stream configuration
+
−
#*** [TBD] statistics retrieval
+
−
#** [TBD] latency measurements
+
−
#*** [TBD] feasibility
+
−
#*** [TBD] implementation
+
−
#** [TBD] integration with CSIT tests
+
−
# Other work items
+
−
#* [TBD] integrate [https://pypi.python.org/pypi/sysinfosuite Python sysinfosuite] to CSIT
+

Latest revision as of 11:45, 20 September 2016

WORK IN PROGRESS - PLEASE TREAT IT AS SUCH

CSIT development plan - background

CSIT test development priorities are aligned with VPP releases timeline, with relative dev work prioritization for each VPP release driven by perceived importance of test verification for VPP functionality and performance. Specific CSIT dev work priorities are driven by rough consensus within the project and contributors and committers availability and willingness to do work.

Following priority tagging scheme is used in this CSIT wiki and FD.io jira for CSIT:

  • [$VppRlsNo-$RelPrio]
    • VppRlsNo - aligned VPP release number - 16.09, 16.12, 17.03.
    • RelPrio - CSIT release priority - P0 for Must Have, P1 for Should Have, P2 for Could Have
  • CSIT dev work tagging examples:
    • [1609-P0] - CSIT Must Have for vpp-v1609
    • [1609-P1] - CSIT Should Have for vpp-v1609
    • [1609-P2] - CSIT Could Have for vpp-v1609

CSIT dev work plan execution tracked on CSIT jira, with priority tags encoded as Jira labels for easy of tracking and reporting.

CSIT performance tests - dev plan

CSIT performance test cases devevelopment work is grouped and tracked across the following VPP functional areas:

  1. NIC devices and drivers
  2. IPv4 data plane
  3. IPv4 control plane
  4. IPv4 encapsulations
  5. IPv4 telemetry
  6. IPv6 data plane
  7. IPv6 control plane
  8. IPv6 encapsulations
  9. IPv6 telemetry
  10. Ethernet L2 data plane
  11. Ethernet L2 control plane
  12. Ethernet L2 encapsulations
  13. Ethernet L2 telemetry
  14. MPLS data plane
  15. NSH data plane

CSIT team is tracking completed CSIT dev work per above area (completed = test cases running in FD.io), as well as top-level test coverage metric [%] of VPP functions. VPP code development dependencies are marked with [Blocked-by VPP-XXX]. Performance tests require the functional tests to be completed - see sub-sections "Functional tests preconditions".

CSIT performance tests are split into the two main groups:

  • NDR(PDR) throughput discovery - tests searching for NDR non-drop rate throughput or PDR partial-drop rate throughput for specified configuration. Also referred to as "long" tests.
  • refNDR throughput verification - tests verifying code patches against the reference NDR throughput. Also referred to as "short" tests.

Infrastructure - NIC devices and drivers

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1606-DONE] NIC Intel X520 2p10GE - DPDK niantic driver, 2 ports within the same NIC.
  2. [1609-P0] NIC Intel XL710 2p40GE - DPDK i40e driver, 2 ports within the same NIC.
  3. [1609-DONE] VPP vhost-user - VPP vhost-user driver, virtio in VM.
  4. [1609-P1] NIC Intel X710 2p40GE - DPDK i40e driver, 2 ports within the same NIC.
  5. [1609-DONE] DPDK vhost-user - DPDK vhost-user driver, virtio in VM.
  6. [1612-P0] vhost-user - XXX vhost-user driver, virtio in VM, box-full test with NxVMs.
  7. [1612-P2] continue to expand VPP driver coverage - more NIC-DPDK drivers, other TBD.

IPv4 data plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P0] IPv4 routing RFC791 - baseline - NIC-to-NIC(XL710).
  2. [1609-DONE] IPv4 routing RFC791 - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
  3. [1609-P1] VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(X520).
  4. [1609-P1] VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(XL710).
  5. [1609-P1] extranet VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(X520).
  6. [1609-P1] extranet VPNv4 routed forwarding (VRF) - baseline - NIC-to-NIC(XL710).
  7. [1606-DONE] IPv4 routing RFC791 - baseline - NIC-to-NIC(X520).

FIB performance

Large-scale FIB tests

  1. [1609-DONE] same prefix length.
  2. [1609-P1] varying prefix lengths.

Security performance

  1. [1606-DONE] security - iacl with n-tuple classification, ip4.
  2. [1606-DONE] security - cop white-/black-list ip4 src addr filtering.

QoS performance

  1. [1609-REVIEW] QoS - ingress policing - rate metering, marking and limiting - 2r3c-(cb,ca) policers, RFC2698.
  2. [1609-P1] QoS - ingress policing - rate metering, marking and limiting - 1r2c-(cb,ca) policers, RFC2697.

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P2] security - source IPv4 Reverse Path Forwarding (RPF). 
   # [1609-P1] routing multipath (ECMP, UECMP) - baseline. 
   # [1609-P1] [Blocked-by VPP-xxx] carrier grade nat 44, CGN44, RFC6598, RFC6888. 
   # [1609-P2] [Blocked-by VPP-xxx] security - stateful firewall.

IPv4 control plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] Proxy ARP. 
   # [1609-P0] IKEv2. 
   # [1609-P1] DHCPv4 proxy. 
   # [1606-DONE] ARP. 
   # [1606-DONE] ICMPv4. 
   # [1606-DONE] DHCPv4 client.

IPv4 encapsulations

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P1] ip4-gre-ip4 - baseline main fib.
  2. [1609-DONE] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] ip4-ipsec-ip4 - baseline main fib.
   # [1609-P0] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), virt2lisp.
   # [1609-P0] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
   # [1609-P0] ip4-ipsec-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P0] ip4-ipsec-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
   # [1606-DONE] ip4-gre-ip4 - baseline main fib.

IPv4 telemetry

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] IPv4 IPFIX - stats collection and export - baseline. 
   # [1609-P0] IPv4 IPFIX - stats collection and export - functional scale. 
   # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline. 
   # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.

IPv6 data plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P0] IPv6 routing RFC2460 - baseline - NIC-to-NIC(XL710).
  2. [1609-P0] IPv6 routing RFC2460 - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
  3. [1609-P1] VPNv6 routed forwarding (VRF) - baseline - NIC-to-NIC(X520).
  4. [1609-P1] VPNv6 routed forwarding (VRF) - baseline - NIC-to-NIC(XL710).
  5. [1609-P2] extranet VPN routed forwarding (VRF) - baseline.
  6. [1606-DONE] IPv6 routing RFC2460 - baseline - NIC-to-NIC(X520).

FIB performance

Large-scale FIB tests

  1. [1609-DONE] same prefix length.
  2. [1609-P1] varying prefix lengths.

Security performance

  1. [1609-DONE] security - iacl with n-tuple classification, ip6.
  2. [1606-DONE] security - cop white-/black-list ip6 src addr filtering.

QoS performance

  1. [1609-P1] QoS - ingress policing - rate metering, marking and rate-limiting -1r2c-(cb,ca), 2r3c-(cb,ca) policers, RFC2475, RFC2697, RFC2698.

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] security - source IPv6 Reverse Path Forwarding (RPF). 
   # [1609-P1] routing multipath (ECMP, UECMP) - baseline.
   # [1609-P2] [Blocked-by VPP-xxx] security - stateful firewall.

IPv6 control plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] router advertisement.
   # [1609-P1] DHCPv6 proxy.
   # [1609-P2] [Blocked-by VPP-xxx] ip6-ioam, in-band OAM.
   # [1606-DONE] neighbor discovery, RFC4861.
   # [1606-DONE] ICMPv6.

IPv6 encapsulations

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-DONE] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.
  2. [1609-DONE] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] ip4-o-ip6 softwire.
   # [1609-P0] ip6-ipsec-ip6.
   # [1609-P0] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), virt2lisp.
   # [1609-P0] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P1] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), virt2lisp.
   # [1609-P1] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P0] ip6-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P0] ip6-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
   # [1609-P1] ip4-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P1] ip4-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp. ?
   # [1609-P1] [Blocked-by VPP-xxx] ip6-segment-routing.

IPv6 telemetry

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] IPv6 IPFIX - stats collection and export - baseline.
   # [1609-P0] IPv6 IPFIX - stats collection and export - functional scale.
   # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
   # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.

Ethernet L2 data plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P0] L2 Cross-connect - baseline - NIC-to-NIC(XL710).
  2. [1609-P0] Bridge domain - baseline - NIC-to-NIC(XL710).
  3. [1609-DONE] L2 Cross-connect - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
  4. [1609-DONE] Bridge domain - baseline vhost-user - NIC(X520)-to-VM(l3fwd,virtio).
  5. [1606-DONE] L2 Cross-connect - baseline - NIC-to-NIC(X520).
  6. [1606-DONE] Bridge domain - baseline - NIC-to-NIC(X520).

FIB performance

Large-scale FIB tests

  1. [1609-P1] [Blocked-by VPP-201] MAC addresses.

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] Bridge domain - with VLAN tag acrobatics 1-to-1, 1-to-2, 2-to-1, 2-to-2.
   # [1609-P1] Integrated Routing-Bridging (IRB) - baseline.
   # [1606-DONE] L2 Cross-connect - with VLAN tag acrobatics 1-to-1, 1-to-2, 2-to-1, 2-to-2.

Ethernet L2 control plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>
  2. [1606-DONE] MAC learning.

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P2] [Blocked-by VPP-xxx] CDP/LLDP.
   # [1606-DONE] MAC split-horizon.

Ethernet L2 encapsulations

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-P0] l2-eth-lispgpe-ip6 - L2BD.
  2. [1609-P0] l2-eth-lispgpe-ip4 - L2BD.
  3. [1609-P1] l2-vxlan-ipv4 - with bridge-domain.
  4. [1609-DONE] l2-vxlan-ipv4 - with L2 cross-connect.

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P0] l2-eth-lispgpe-ip6 - L2BD, L2XC.
   # [1609-P0] l2-eth-lispgpe-ip4 - L2BD, L2XC.
   # [1609-P0] l2eth-gre-ip4-ipsec-ip4-eth - L2BD, L2XC.
   # [1609-P2] l2-eth-l2tpv3-ip4 - L2BD, L2XC.
   # [1609-P2] l2-eth-l2tpv3-ip6 - L2BD, L2XC.
   # [1609-P2] l2-eth-gre-ip4 - L2BD, L2XC.
   # [1609-P2] l2-eth-gre-ip6 - L2BD, L2XC.

L2 telemetry

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P1] L2 FIB - stats collection and export - baseline.
   # [1609-P2] L2 FIB - stats collection and export - functional scale.
   # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
   # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.

MPLS data plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P2] vpnv4-mpls-gre-ip4.
   # [1609-P2] mpls-o-eth.

NSH data plane

1609-P0

Error while calling JIRA...

1609-P1

Error while calling JIRA...

1609-P2

Error while calling JIRA...

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan - provided for reference only: 

   # [1609-P1] ip4-nsh-gre-ip4.
   # [1609-P1] ip4-nsh-vxlan-gpe-ip4.

END OF NOTE

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/PerfTestPlan&oldid=3097"