CSIT/PerfTestPlan

From fd.io
< CSIT
Revision as of 09:12, 27 July 2016 by Mackonstan (Talk | contribs)

Jump to: navigation, search

WORK IN PROGRESS - PLEASE TREAT IT AS SUCH

CSIT development plan - background

CSIT test development priorities are aligned with VPP releases timeline, with relative dev work prioritization for each VPP release driven by perceived importance of test verification for VPP functionality and performance. Specific CSIT dev work priorities are driven by rough consensus within the project and contributors and committers availability and willingness to do work.

Following priority tagging scheme is used in this CSIT wiki and FD.io jira for CSIT:

  • [$VppRlsNo-$RelPrio]
    • VppRlsNo - aligned VPP release number - 16.09, 16.12, 17.03.
    • RelPrio - CSIT release priority - P0 for Must Have, P1 for Should Have, P2 for Could Have
  • CSIT dev work tagging examples:
    • [1609-P0] - CSIT Must Have for vpp-v1609
    • [1609-P1] - CSIT Should Have for vpp-v1609
    • [1609-P2] - CSIT Could Have for vpp-v1609

CSIT dev work plan execution tracked on CSIT jira, with priority tags encoded as Jira labels for easy of tracking and reporting.

CSIT performance tests - dev plan

CSIT performance test cases devevelopment work is grouped and tracked across the following VPP functional areas:

  1. IPv4 data plane
  2. IPv4 control plane
  3. IPv4 encapsulations
  4. IPv4 telemetry
  5. IPv6 data plane
  6. IPv6 control plane
  7. IPv6 encapsulations
  8. IPv6 telemetry
  9. Ethernet L2 data plane
  10. Ethernet L2 control plane
  11. Ethernet L2 encapsulations
  12. Ethernet L2 telemetry
  13. MPLS data plane
  14. NSH data plane

CSIT team is tracking completed CSIT dev work per above area (completed = test cases running in FD.io), as well as top-level test coverage metric [%] of VPP functions. VPP code development dependencies are marked with [VPP-DEP]. Performance tests require the functional tests to be completed - see sub-sections "Functional tests preconditions".

CSIT performance tests are split into the two main groups:

  • NDR(PDR) throughput discovery - tests searching for NDR non-drop rate throughput or PDR partial-drop rate throughput for specified configuration. Also referred to as "long" tests.
  • refNDR throughput verification - tests verifying code patches against the reference NDR throughput. Also referred to as "short" tests.

IPv4 data plane

ip4dp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip4dp].

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P1] continue to expand VPP driver coverage - DPDK, other TBD (Netmap?, ) CSIT-272
    • (single_physical_interface_type,NIC_type)
    • (all_physical_interface_type,NIC_type)
    • (single_virtual_interface_type,VF_type)
    • (all_virtual_interface_type,VF_type) - box-full tests
  2. [1606-DONE] IPv4 routing RFC791 - baseline.

FIB performance

Large-scale FIB tests

  1. [1609-P0] IPv4 same prefix length - Gerrit #1613 Jira CSIT-169
  2. [1609-P1] IPv4 varying prefix lengths CSIT-273

Security performance

  1. [1606-DONE] security - iacl with n-tuple classification, ip4. NDR, refNDR.
  2. [1606-DONE] security - cop white-/black-list ip4 src addr filtering. NDR, refNDR.

QoS performance

  1. [1609-P1] QoS - ingress policing - rate metering, marking and rate-limiting. CSIT-274
  2. [1609-P0] QoS - ingress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers, RFC2475, RFC2697, RFC2698. CSIT-275

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] VPN routed forwarding (VRF) - baseline. CSIT-276
   # [1609-P0] extranet VPN routed forwarding (VRF) - baseline. CSIT-277
   # [1609-P0] security - source IPv4 Reverse Path Forwarding (RPF). CSIT-278
   # [1609-P1] routing multipath (ECMP, UECMP) - baseline. CSIT-279
   # [1609-P1] [TODO VPP-xxx] carrier grade nat 44, CGN44, RFC6598, RFC6888. CSIT-280
   # [1609-P2] [TODO VPP-xxx] security - stateful firewall. CSIT-281

IPv4 control plane

ip4cp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip4cp].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] Proxy ARP. CSIT-282
   # [1609-P0] IKEv2. CSIT-283
   # [1609-P1] DHCPv4 proxy. CSIT-284
   # [1606-DONE] ARP. CSIT-285
   # [1606-DONE] ICMPv4. CSIT-286
   # [1606-DONE] DHCPv4 client. CSIT-287

IPv4 encapsulations

ip4enc label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip4enc].

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P1] ip4-gre-ip4 - baseline main fib.
  2. [1609-P1] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] ip4-ipsec-ip4 - baseline main fib.
   # [1609-P0] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P0] ip4-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P0] ip4-ipsec-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P0] ip4-ipsec-lispgpe-ip4 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1606-DONE] ip4-gre-ip4 - baseline main fib.

IPv4 telemetry

ip4tel label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip4tel].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] IPv4 IPFIX - stats collection and export - baseline.
   # [1609-P0] IPv4 IPFIX - stats collection and export - functional scale.
   # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
   # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.

IPv6 data plane

ip6dp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip6dp].

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P1] continue to expand VPP driver coverage - DPDK, other TBD (Netmap?, )
    • (single_physical_interface_type,NIC_type)
    • (all_physical_interface_type,NIC_type)
    • (single_virtual_interface_type,VF_type)
    • (all_virtual_interface_type,VF_type) - box-full tests
  2. [1609-P1] VPN routed forwarding (VRF) - baseline.

FIB performance

Large-scale FIB tests

  1. [1609-REVIEW] IPv6 same prefix length - Gerrit #1664 Jira CSIT-179
  2. [1609-P1] IPv6 varying prefix lengths

Security performance

  1. [1609-DONE] security - iacl with n-tuple classification, ip6. NDR, refNDR.
  2. [1606-DONE] security - cop white-/black-list ip6 src addr filtering. NDR, refNDR.

QoS performance

  1. [1609-P1] QoS - ingress policing - rate metering, marking and rate-limiting.

Functional tests preconditions

  1. [1606-DONE] routing [rfc2460] - baseline.

Below copied from CSIT functional tests plan. 

   # [1609-P0] VPN routed forwarding (VRF) - baseline.
   # [1609-P0] extranet VPN routed forwarding (VRF) - baseline.
   # [1609-P0] security - source IPv6 Reverse Path Forwarding (RPF).
   # [1609-P0] qos - ingress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers, RFC2475, RFC2697, RFC2698.
   # [1609-P1] routing multipath (ECMP, UECMP) - baseline.
   # [1609-P2] [TODO VPP-xxx] security - stateful firewall.

IPv6 control plane

ip6cp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip6cp].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] router advertisement.
   # [1609-P1] DHCPv6 proxy.
   # [1609-P2] [TODO VPP-xxx] ip6-ioam, in-band OAM.
   # [1606-DONE] neighbor discovery, RFC4861.
   # [1606-DONE] ICMPv6.

IPv6 encapsulations

ip6enc label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip6enc].

Indicative coverage = <to_be_added>

  1. [1609-P1] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp.

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] ip4-o-ip6 softwire.
   # [1609-P0] ip6-ipsec-ip6.
   # [1609-P0] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P0] ip6-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P1] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P1] ip4-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P0] ip6-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P0] ip6-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P1] ip4-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), phy2lisp, virt2lisp.
   # [1609-P1] ip4-ipsec-lispgpe-ip6 - main fib, vrf (gpe_vni-to-vrf), lisp2lisp.
   # [1609-P1] [TODO VPP-xxx] ip6-segment-routing.

IPv6 telemetry

ip6tel label used for tracking work status on CSIT JIRA - [url JIRA filtered view for ip6tel].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P0] IPv6 IPFIX - stats collection and export - baseline.
   # [1609-P0] IPv6 IPFIX - stats collection and export - functional scale.
   # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
   # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.

Ethernet L2 data plane

l2dp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for l2dp].

Indicative coverage = <to_be_added>

Baseline performance

  1. [1609-P1] continue to expand VPP driver coverage - DPDK, other TBD (Netmap?, )
    • (single_physical_interface_type,NIC_type)
    • (all_physical_interface_type,NIC_type)
    • (single_virtual_interface_type,VF_type)
    • (all_virtual_interface_type,VF_type) - box-full tests

FIB performance

Large-scale FIB tests

  1. [1609-P0] [TODO VPP-201] MAC addresses - Gerrit #1862

Functional tests preconditions

  1. [1606-DONE] L2 Cross-connect - baseline.
  2. [1606-DONE] Bridge domain - baseline.


Below copied from CSIT functional tests plan. 

   # [1609-P0] Bridge domain - with VLAN tag acrobatics 1-to-1, 1-to-2, 2-to-1, 2-to-2.
   # [1609-P1] Integrated Routing-Bridging (IRB) - baseline.
   # [1606-DONE] L2 Cross-connect - with VLAN tag acrobatics 1-to-1, 1-to-2, 2-to-1, 2-to-2.

Ethernet L2 control plane

l2cp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for l2cp].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

  1. [1606-DONE] MAC learning.

Below copied from CSIT functional tests plan. 

   # [1609-P2] [TODO VPP-xxx] CDP/LLDP.
   # [1606-DONE] MAC split-horizon.

Ethernet L2 encapsulations

l2enc label used for tracking work status on CSIT JIRA - [url JIRA filtered view for l2enc].

Indicative coverage = <to_be_added>

  1. [1609-P0] l2-eth-lispgpe-ip6 - L2BD, L2XC.
  2. [1609-P0] l2-eth-lispgpe-ip4 - L2BD, L2XC.
  3. [1606-DONE] l2-vxlan-ipv4 - with bridge-domain.
  4. [1606-DONE] l2-vxlan-ipv4 - with L2 cross-connect.

Inputs from functional test plan: 

   # [1609-P0] l2-eth-lispgpe-ip6 - L2BD, L2XC.
   # [1609-P0] l2-eth-lispgpe-ip4 - L2BD, L2XC.
   # [1609-P0] l2eth-gre-ip4-ipsec-ip4-eth - L2BD, L2XC.
   # [1609-P2] l2-eth-l2tpv3-ip4 - L2BD, L2XC.
   # [1609-P2] l2-eth-l2tpv3-ip6 - L2BD, L2XC.
   # [1609-P2] l2-eth-gre-ip4 - L2BD, L2XC.
   # [1609-P2] l2-eth-gre-ip6 - L2BD, L2XC.
   # [1606-DONE] l2-vxlan-ipv4 - with bridge-domain.
   # [1606-DONE] l2-vxlan-ipv4 - with L2 cross-connect.

L2 telemetry

l2tel label used for tracking work status on CSIT JIRA - [url JIRA filtered view for l2tel].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P1] L2 FIB - stats collection and export - baseline.
   # [1609-P2] L2 FIB - stats collection and export - functional scale.
   # [1609-P0] Switched port analyzer mirroring SPAN - (flow,port) mirroring to local port - baseline.
   # [1609-P1] Lawful intercept  - (flow,port) mirroring to remote locatio via LI UDP tunnel - baseline.

MPLS data plane

mplsdp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for mplsdp].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P2] vpnv4-mpls-gre-ip4.
   # [1609-P2] mpls-o-eth.

NSH data plane

nshdp label used for tracking work status on CSIT JIRA - [url JIRA filtered view for nshdp].

Indicative coverage = <to_be_added>

  1. [1609-Px] <to_be_added>

Functional tests preconditions

Below copied from CSIT functional tests plan. 

   # [1609-P1] ip4-nsh-gre-ip4.
   # [1609-P1] ip4-nsh-vxlan-gpe-ip4.

END OF NOTE

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/PerfTestPlan&oldid=2260"