Difference between revisions of "CSIT/FuncTestPlan"

From fd.io
Jump to: navigation, search
Line 1: Line 1:
 +
160416-fdio-csit-dev-func-test-plan.wiki
 +
 
==FD.io IT systems integration==
 
==FD.io IT systems integration==
  
−
One Jenkins job for CSIT functional test cases:
+
Three Jenkins jobs involving CSIT functional test cases:
  
−
# Job name: '''vpp-csit-verify-virl''' [https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/]
+
# Job name: [https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/ '''vpp-csit-verify-virl'''].
−
#* Description: Job to test VPP build with verified CSIT test cases.
+
#* Description: Job to test VPP build against verified CSIT functional test cases (a.k.a. "golden CSIT version"), running in VIRL vNet testbeds hosted by LF FD.io.
 
#* Triggers:
 
#* Triggers:
−
#** VPP patch-based automatic trigger
+
#** VPP patch-based automatic trigger.
−
#** manual trigger: Comment "recheck" in proposed patch in VPP project
+
#** Manual trigger: Comment "recheck" in proposed patch in VPP project.
−
#* Status: WORKING
+
#* Status: WORKING.
−
 
+
# Job name: [https://jenkins.fd.io/view/csit/job/csit-vpp-functional-virl/ '''csit-vpp-functional-virl'''].
−
One Jenkins job to get verified CSIT test cases for VPP patch tests:
+
#* Description: Job to test CSIT patch against verified VPP version (a.k.a. "golden VPP version").
−
 
+
−
# Job name: '''csit-vpp-master-verify-weekly''' [https://jenkins.fd.io/view/csit/job/csit-vpp-master-verify-weekly/]
+
−
#* Description: Weekly job that validates the latest VPP build against the latest CSIT code. The whole test suite is run three times in sequence to check stability and reliability of the results. Two goals achieved in case of successful run:
+
−
#** verified CSIT test cases ("golden branch") version with tag "csit-verified" for VPP verify job (vpp-csit-verify-virl)
+
−
#** verified VPP version ("golden version") for CSIT verify job (csit-vpp-functional-virl)
+
 
#* Triggers:
 
#* Triggers:
−
#** scheduled to run periodicaly every Sunday at 1PM UTC
+
#** CSIT patch-based automatic trigger.
−
#** no manual trigger
+
#** Mnual trigger: Comment "recheck" in proposed patch in CSIT project.
−
#* Status: tagging not working - should be solved till Sunday 17th
+
#* Status: WORKING.
−
 
+
# Job name: [https://jenkins.fd.io/view/csit/job/csit-vpp-master-verify-weekly/ '''csit-vpp-master-verify-weekly'''].
−
One Jenkins job for CSIT functional test cases development:
+
#* Description: Weekly job that validates the latest VPP build against the latest CSIT code. The whole CSIT test suite is run three times in sequence to check stability and consistency of results. Two goals achieved in case of successful run:
−
 
+
#** Verified CSIT test cases ("golden CSIT version") version with tag "csit-verified" used in VPP verify jobs (job name: vpp-csit-verify-virl).
−
# Job name: '''csit-vpp-functional-virl''' [https://jenkins.fd.io/view/csit/job/csit-vpp-functional-virl/]
+
#** Verified VPP version ("golden VPP version") used in CSIT verify jobs (csit-vpp-functional-virl).
−
#* Description: Job to test CSIT patch with verified VPP version.
+
 
#* Triggers:
 
#* Triggers:
−
#** CSIT patch-based automatic trigger
+
#** Scheduled to run periodicaly every Sunday at 13:00 UTC.
−
#** manual trigger: Comment "recheck" in proposed patch in CSIT project
+
#** No manual trigger.
−
#* Status: WORKING
+
#* Status: tagging not working - should be solved by Sunday 17-April 2016.
  
 
==CSIT functional test cases - working==
 
==CSIT functional test cases - working==
Line 35: Line 31:
 
# Last manual run results reports:
 
# Last manual run results reports:
 
#* https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/
 
#* https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/
−
# List of working functionald test cases:
+
# List of working functional test cases:
 
## ~/csit$ grep "^| [a-zA-Z]" -nr tests | grep -v Documentation | grep -vi " | "  | grep -v performance
 
## ~/csit$ grep "^| [a-zA-Z]" -nr tests | grep -v Documentation | grep -vi " | "  | grep -v performance
 
## tests/suites/bridge_domain/test.robot:25:| VPP reports interfaces
 
## tests/suites/bridge_domain/test.robot:25:| VPP reports interfaces
Line 75: Line 71:
 
'''VPP network functions:'''
 
'''VPP network functions:'''
  
−
# ip4 data plane
+
# IP4 data plane
−
## [DONE-R1] baseline routing
+
## [DONE-R1] baseline routing [rfc 791]
 
##* [DONE-R1] TG packets routed to DUT ingress interface
 
##* [DONE-R1] TG packets routed to DUT ingress interface
 
##* [DONE-R1] TG packets routed to DUT egress interface
 
##* [DONE-R1] TG packets routed to DUT egress interface
Line 88: Line 84:
 
##* [P2-R1] TG packets routed to DUT2 egress interface through DUT1, VPP configured with two VRFs
 
##* [P2-R1] TG packets routed to DUT2 egress interface through DUT1, VPP configured with two VRFs
 
##* [P2-R1] TG packets routed to TG through DUT1 and DUT2, VPP configured with two VRFs
 
##* [P2-R1] TG packets routed to TG through DUT1 and DUT2, VPP configured with two VRFs
 +
## [P1-R2] vpn extranet baseline routed forwarding (VRF)
 +
##* [P1-R2] TG packets routed to DUT ingress interface, VPP configured with two extranet VRFs
 +
##* [P1-R2] TG packets routed to DUT egress interface, VPP configured with two extranet  VRFs
 +
##* [P1-R2] TG packets routed to DUT2 through DUT1, VPP configured with two extranet VRFs
 +
##* [P1-R2] TG packets routed to DUT2 egress interface through DUT1, VPP configured with two extranet VRFs
 +
##* [P1-R2] TG packets routed to TG through DUT1 and DUT2, VPP configured with two extranet VRFs
 
## [TBD] routing multipath
 
## [TBD] routing multipath
 
##* [TBD] equal cost multi-path (ECMP)
 
##* [TBD] equal cost multi-path (ECMP)
Line 118: Line 120:
 
##* [P2-R1] VPP 2r3c-ca policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
 
##* [P2-R1] VPP 2r3c-ca policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
 
## [TBD] qos - egress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers
 
## [TBD] qos - egress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers
−
## [P1-R1] carrier grade nat 44 (CGNAT44)
+
## [P1-R1] carrier grade nat 44, CGN44 [rfc 6598], [rfc 6888]
−
##* [P1-R1] CGNAT address-only
+
##* [P1-R1] CGN address-only
−
##* [P1-R1] CGNAT address-and-port
+
##* [P1-R1] CGN address-and-port
−
##* [TBD] CGNAT address-only and application layer gateway (ALG)
+
##* [TBD] CGN address-only and application layer gateway (ALG)
−
##* [TBD] CGNAT address-and-port and ALG
+
##* [TBD] CGN address-and-port and ALG
−
# ip4 control plane
+
# IP4 control plane
−
## [P1-R1] arp
+
## [P1-R1] ARP
−
##* [DONE-R1] VPP responds to ARP request (rfc:826)
+
##* [DONE-R1] VPP responds to ARP request [rfc 826]
 
##* [P1-R1] VPP sends ARP requests for unknown destinations
 
##* [P1-R1] VPP sends ARP requests for unknown destinations
 +
## [P1-R1] Proxy ARP
 +
##* [P1-R1] VPP responds to ARP requests on behalf of host present on another VPP interface.
 
## [DONE-R1] ICMPv4
 
## [DONE-R1] ICMPv4
 
##* [DONE-R1] VPP can process ICMP echo request from min to 1500B packet size with 1B increment
 
##* [DONE-R1] VPP can process ICMP echo request from min to 1500B packet size with 1B increment
 
##* [DONE-R1] VPP can process ICMP echo request from 1500B to max packet size with 10B increment
 
##* [DONE-R1] VPP can process ICMP echo request from 1500B to max packet size with 10B increment
−
##* [DONE-R1] VPP replies to ICMPv4 echo request (rfc:792)
+
##* [DONE-R1] VPP replies to ICMPv4 echo request [rfc 792]
 
## [P2-R1] DHCPv4 proxy
 
## [P2-R1] DHCPv4 proxy
 
##* [P2-R1] VPP proxies valid DHCPv4 request to DHCPv4 server
 
##* [P2-R1] VPP proxies valid DHCPv4 request to DHCPv4 server
Line 145: Line 149:
 
##* [P1-R2] rsa key auth
 
##* [P1-R2] rsa key auth
 
##* [P1-R2] ID type IPv4 address
 
##* [P1-R2] ID type IPv4 address
−
##* [P1-R2] ID type rfc822 (email)
+
##* [P1-R2] ID type (email) [rfc 822]
 
##* [P1-R2] ID type key-id
 
##* [P1-R2] ID type key-id
 
##* [P1-R2] test various supported encryption/prf/integrity algs, DH groups, extended sequence number
 
##* [P1-R2] test various supported encryption/prf/integrity algs, DH groups, extended sequence number
−
# ip4 encapsulations
+
# IP4 encapsulations
−
## [DONE-R1]ip4-vxlan-ipv4
+
−
##* [DONE-R1] VPP can encapsulate L2 in VXLAN over V4 using bridge-domain
+
−
##* [DONE-R1] VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
+
 
## [P1-R1] ip4-gre-ip4
 
## [P1-R1] ip4-gre-ip4
 
##* [P1-R1] VPP can encapsulate IPv4 traffic in GRE
 
##* [P1-R1] VPP can encapsulate IPv4 traffic in GRE
 
##* [P1-R1] VPP can route IPv4 traffic from GRE tunnel
 
##* [P1-R1] VPP can route IPv4 traffic from GRE tunnel
−
## [TBD] ip4-nsh-gre-ip4
+
## [P1-R2] ip4-ipsec-ip4
−
## [TBD] ip4-nsh-vxlan-gpe-ip4
+
−
## [TBD] ip4-ipsec-ip4
+
 
##* [P1-R2] basic connectivity test - create ipsec SA and policy on VPP, test different encryption and integrity algorithms
 
##* [P1-R2] basic connectivity test - create ipsec SA and policy on VPP, test different encryption and integrity algorithms
 
##* [P1-R2] add/del SA and policy tests
 
##* [P1-R2] add/del SA and policy tests
Line 163: Line 162:
 
##* [P1-R2] some negative tests (each side has different enryption/integrity alg)
 
##* [P1-R2] some negative tests (each side has different enryption/integrity alg)
 
# ip6 data plane
 
# ip6 data plane
−
## [DONE-R1] baseline routing
+
## [DONE-R1] baseline routing [rfc 2460]
 
##* [DONE-R1] TG can route to first DUT egress interface
 
##* [DONE-R1] TG can route to first DUT egress interface
 
##* [DONE-R1] TG can route to second DUT through first DUT
 
##* [DONE-R1] TG can route to second DUT through first DUT
Line 186: Line 185:
 
##* [P2-R1] VPP can drop packets based on IPv6 UDP src+dst port
 
##* [P2-R1] VPP can drop packets based on IPv6 UDP src+dst port
 
##* [P2-R1] VPP can drop packets based on src MAC + IPv6 UDP src+dst port
 
##* [P2-R1] VPP can drop packets based on src MAC + IPv6 UDP src+dst port
−
## [P0-R1] Vsecurity - cop white-/black-lists
+
## [P0-R1] security - cop white-/black-lists
 
##* [P0-R1] VPP permits packets based on IPv6 src-addr
 
##* [P0-R1] VPP permits packets based on IPv6 src-addr
 
##* [P0-R1] VPP drops packets based on IPv6 src-addr
 
##* [P0-R1] VPP drops packets based on IPv6 src-addr
 
## [P2-R1] security - source IPv6 Reverse Path Forwarding (RPF)
 
## [P2-R1] security - source IPv6 Reverse Path Forwarding (RPF)
 
##* [P2-R1] - source RPF check on IPv6 src-addr
 
##* [P2-R1] - source RPF check on IPv6 src-addr
−
## qos - ingress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers (cb=color-blind, ca=color-aware)
+
## [P1-R1] qos traffic conditioning - ingress rate metering, marking and rate-limiting - 1r2c-(cb,ca), 2r3c-(cb,ca) policers (cb=color-blind, ca=color-aware) [rfc 2475], [rfc 2697], [rfc 2698]
 
##* [P1-R1] VPP 1r2c-cb policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
 
##* [P1-R1] VPP 1r2c-cb policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
 
##* [P2-R1] VPP 1r2c-ca policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
 
##* [P2-R1] VPP 1r2c-ca policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
Line 205: Line 204:
 
##* [P0-R1] ??? ^ retransmits ???
 
##* [P0-R1] ??? ^ retransmits ???
 
##* [P0-R1] VPP ??? handles RA ???
 
##* [P0-R1] VPP ??? handles RA ???
−
## [P0-R1] neighbor discovery (rfc4861)
+
## [P0-R1] neighbor discovery [rfc 4861]
 
##* [P0-R1] VPP validates and replies to IPv6 NS/NA messages
 
##* [P0-R1] VPP validates and replies to IPv6 NS/NA messages
 
## [DONE-R1] ICMPv6
 
## [DONE-R1] ICMPv6
Line 217: Line 216:
 
# ip6 encapsulations
 
# ip6 encapsulations
 
## [P1-R1] ip4-o-ip6 softwire
 
## [P1-R1] ip4-o-ip6 softwire
−
##* [P1-R1] lightweight46
+
##* [P1-R1] lightweight46 [rfc 7596]
−
##* [P2-R1] map-e
+
##* [P2-R1] map-e [rfc 7597]
 +
##* [P2-R1] map-t [rfc 7599]
 
## [TBD] ip6-segment-routing
 
## [TBD] ip6-segment-routing
 
## [P1-R2] ip6-ipsec-ip6
 
## [P1-R2] ip6-ipsec-ip6
Line 228: Line 228:
 
## [DONE-R1] L2 xconnect
 
## [DONE-R1] L2 xconnect
 
##* [DONE-R1] VPP forwards packets via L2 xconnect in 3-node topology
 
##* [DONE-R1] VPP forwards packets via L2 xconnect in 3-node topology
−
##* [TBD] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 1-to-1, dot1q-to-dot1q
+
##* [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 1-to-1, dot1q-to-dot1q
−
##* [TBD] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 1-to-2, dot1q-to-dot1ad
+
##* [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 1-to-2, dot1q-to-dot1ad
−
##* [TBD] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 2-to-1, dot1ad-to-dot1q
+
##* [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 2-to-1, dot1ad-to-dot1q
−
##* [TBD] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 2-to-2, dot1ad-to-dot1ad7
+
##* [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 2-to-2, dot1ad-to-dot1ad7
 
## [DONE-R1] Bridge domain
 
## [DONE-R1] Bridge domain
 
##* [DONE-R1] VPP forwards packets via L2 bridge domain 2 ports
 
##* [DONE-R1] VPP forwards packets via L2 bridge domain 2 ports
 
##* [DONE-R1] VPP forwards packets via L2 bridge domain in 3-node topology
 
##* [DONE-R1] VPP forwards packets via L2 bridge domain in 3-node topology
 
##* [DONE-R1] VPP forwards packets via L2 bridge domain in 3-node topology with static L2FIB entries
 
##* [DONE-R1] VPP forwards packets via L2 bridge domain in 3-node topology with static L2FIB entries
−
##* [TBD] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 1-to-1, dot1q-to-dot1q
+
##* [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 1-to-1, dot1q-to-dot1q
−
##* [TBD] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 1-to-2, dot1q-to-dot1ad
+
##* [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 1-to-2, dot1q-to-dot1ad
−
##* [TBD] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 2-to-1, dot1ad-to-dot1q
+
##* [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 2-to-1, dot1ad-to-dot1q
−
##* [TBD] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 2-to-2, dot1ad-to-dot1ad
+
##* [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 2-to-2, dot1ad-to-dot1ad
 
## [TBD] irb
 
## [TBD] irb
 
##* [TBD] TG can route to DUT ingress irb interface
 
##* [TBD] TG can route to DUT ingress irb interface
Line 251: Line 251:
 
## [TBD] cdp/lldp
 
## [TBD] cdp/lldp
 
# l2 encapsulations
 
# l2 encapsulations
 +
## [DONE-R1] l2-vxlan-ipv4
 +
##* [DONE-R1] VPP can encapsulate L2 in VXLAN over V4 using bridge-domain
 +
##* [DONE-R1] VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
 
## [TBD] l2-eth-l2tpv3-ip4
 
## [TBD] l2-eth-l2tpv3-ip4
 
## [TBD] l2-eth-l2tpv3-ip6
 
## [TBD] l2-eth-l2tpv3-ip6
Line 260: Line 263:
 
## [TBD] vpnv4-mpls-gre-ip4
 
## [TBD] vpnv4-mpls-gre-ip4
 
## [TBD] mpls-o-eth
 
## [TBD] mpls-o-eth
 +
# nsh
 +
## [TBD] ip4-nsh-gre-ip4
 +
## [TBD] ip4-nsh-vxlan-gpe-ip4
  
 
'''VPP deployment topologies:'''
 
'''VPP deployment topologies:'''
Line 297: Line 303:
 
* future per code change functional and vpp node graph dependencies
 
* future per code change functional and vpp node graph dependencies
  
−
Gathering of stats/logs/core dumps from DUTs/TGs after test exec
+
Gathering of stats/logs/core dumps from DUTs/TGs after test execution:
 
* naming schemes, storage, packaging
 
* naming schemes, storage, packaging
 
* collection of TELEMETRY
 
* collection of TELEMETRY
Line 305: Line 311:
 
** for human driven part-automated debugging
 
** for human driven part-automated debugging
 
** for machine driven fully-automated analysis
 
** for machine driven fully-automated analysis
−
* use PaNDA in future?
+
* use Network Data Analytics engine in the future
  
 
Honeycomb tests framework
 
Honeycomb tests framework

Revision as of 11:42, 16 April 2016

160416-fdio-csit-dev-func-test-plan.wiki

FD.io IT systems integration

Three Jenkins jobs involving CSIT functional test cases:

  1. Job name: vpp-csit-verify-virl.
    • Description: Job to test VPP build against verified CSIT functional test cases (a.k.a. "golden CSIT version"), running in VIRL vNet testbeds hosted by LF FD.io.
    • Triggers:
      • VPP patch-based automatic trigger.
      • Manual trigger: Comment "recheck" in proposed patch in VPP project.
    • Status: WORKING.
  2. Job name: csit-vpp-functional-virl.
    • Description: Job to test CSIT patch against verified VPP version (a.k.a. "golden VPP version").
    • Triggers:
      • CSIT patch-based automatic trigger.
      • Mnual trigger: Comment "recheck" in proposed patch in CSIT project.
    • Status: WORKING.
  3. Job name: csit-vpp-master-verify-weekly.
    • Description: Weekly job that validates the latest VPP build against the latest CSIT code. The whole CSIT test suite is run three times in sequence to check stability and consistency of results. Two goals achieved in case of successful run:
      • Verified CSIT test cases ("golden CSIT version") version with tag "csit-verified" used in VPP verify jobs (job name: vpp-csit-verify-virl).
      • Verified VPP version ("golden VPP version") used in CSIT verify jobs (csit-vpp-functional-virl).
    • Triggers:
      • Scheduled to run periodicaly every Sunday at 13:00 UTC.
      • No manual trigger.
    • Status: tagging not working - should be solved by Sunday 17-April 2016.

CSIT functional test cases - working

  1. Working functional test cases - 25 working in FD.io today.
  2. Last manual run results reports:
  3. List of working functional test cases:
    1. ~/csit$ grep "^| [a-zA-Z]" -nr tests | grep -v Documentation | grep -vi " | " | grep -v performance
    2. tests/suites/bridge_domain/test.robot:25:| VPP reports interfaces
    3. tests/suites/bridge_domain/test.robot:29:| Vpp forwards packets via L2 bridge domain 2 ports
    4. tests/suites/bridge_domain/test.robot:40:| Vpp forwards packets via L2 bridge domain in circular topology
    5. tests/suites/bridge_domain/test.robot:55:| Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
    6. tests/suites/honeycomb/sanity.robot:20:| Honeycomb reports running configuration
    7. tests/suites/ipv4/ipv4_untagged.robot:30:| VPP replies to ICMPv4 echo request
    8. tests/suites/ipv4/ipv4_untagged.robot:39:| TG can route to DUT egress interface
    9. tests/suites/ipv4/ipv4_untagged.robot:48:| TG can route to DUT2 through DUT1
    10. tests/suites/ipv4/ipv4_untagged.robot:57:| TG can route to DUT2 egress interface through DUT1
    11. tests/suites/ipv4/ipv4_untagged.robot:66:| TG can route to TG through DUT1 and DUT2
    12. tests/suites/ipv4/ipv4_untagged.robot:87:| VPP can process ICMP echo request from min to 1500B packet size with 1B increment
    13. tests/suites/ipv4/ipv4_untagged.robot:91:| VPP can process ICMP echo request from 1500B to max packet size with 10B increment
    14. tests/suites/ipv4/ipv4_untagged.robot:109:| VPP responds to ARP request
    15. tests/suites/ipv6/ipv6_untagged.robot:33:| VPP replies to ICMPv6 echo request
    16. tests/suites/ipv6/ipv6_untagged.robot:37:| VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
    17. tests/suites/ipv6/ipv6_untagged.robot:41:| VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
    18. tests/suites/ipv6/ipv6_untagged.robot:59:| TG can route to first DUT egress interface
    19. tests/suites/ipv6/ipv6_untagged.robot:64:| TG can route to second DUT through first DUT
    20. tests/suites/ipv6/ipv6_untagged.robot:69:| TG can route to second DUT egress interface through first DUT
    21. tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
    22. tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
    23. tests/suites/l2_xconnect/l2_xconnect_untagged.robot:26:| Vpp forwards packets via L2 xconnect in circular topology
    24. tests/suites/vxlan/vxlan_bd_dot1q.robot:31:| VPP can encapsulate L2 in VXLAN over IPv4 over Dot1Q
    25. tests/suites/vxlan/vxlan_bd_untagged.robot:30:| VPP can pass IPv4 bidirectionally through VXLAN
    26. tests/suites/vxlan/vxlan_xconnect_untagged.robot:29:| VPP can pass IPv4 bidirectionally through VXLAN

CSIT functional test cases - work in progress

  • 802.1ad/QinQ.
  • vhost-user.
  • basic HoneyComb handling in tests.

CSIT functional test cases - plan and status

Below laundry list of functions that could/should be tested - need to prioritize based on FD.io community needs and feedback, and scheduled for CSIT development. To work and be useful, CSIT development should be done in coordination with VPP Release Candidate content and release schedule.

VPP network functions:

  1. IP4 data plane
    1. [DONE-R1] baseline routing [rfc 791]
      • [DONE-R1] TG packets routed to DUT ingress interface
      • [DONE-R1] TG packets routed to DUT egress interface
      • [DONE-R1] TG packets routed to DUT2 through DUT1
      • [DONE-R1] TG packets routed to DUT2 egress interface through DUT1
      • [DONE-R1] TG packets routed to TG through DUT1 and DUT2
    2. [P2-R1] vpn baseline routed forwarding (VRF)
      • [P2-R1] TG packets routed to DUT ingress interface, VPP configured with two VRFs
      • [P2-R1] TG packets routed to DUT egress interface, VPP configured with two VRFs
      • [P2-R1] TG packets routed to DUT2 through DUT1, VPP configured with two VRFs
      • [P2-R1] TG packets routed to DUT2 egress interface through DUT1, VPP configured with two VRFs
      • [P2-R1] TG packets routed to TG through DUT1 and DUT2, VPP configured with two VRFs
    3. [P1-R2] vpn extranet baseline routed forwarding (VRF)
      • [P1-R2] TG packets routed to DUT ingress interface, VPP configured with two extranet VRFs
      • [P1-R2] TG packets routed to DUT egress interface, VPP configured with two extranet VRFs
      • [P1-R2] TG packets routed to DUT2 through DUT1, VPP configured with two extranet VRFs
      • [P1-R2] TG packets routed to DUT2 egress interface through DUT1, VPP configured with two extranet VRFs
      • [P1-R2] TG packets routed to TG through DUT1 and DUT2, VPP configured with two extranet VRFs
    4. [TBD] routing multipath
      • [TBD] equal cost multi-path (ECMP)
      • [TBD] unequal cost multi-path (UECMP)
    5. [P0-R1] security - iacl with n-tuple classification
      • [P0-R1] VPP drops packets based on IPv4 source addresses (src-addr)
      • [P0-R1] VPP drops packets based on IPv4 destination addresses (dst-addr)
      • [P0-R1] VPP drops packets based on IPv4 src-addr and dst-addr
      • [P2-R1] VPP drops packets based on IPv4 protocol (TCP/UDP)
      • [P2-R1] VPP drops packets based on IPv4 TCP src ports
      • [P2-R1] VPP drops packets based on IPv4 TCP dst ports
      • [P2-R1] VPP drops packets based on IPv4 TCP src+dst ports
      • [P2-R1] VPP drops packets based on IPv4 UDP src ports
      • [P2-R1] VPP drops packets based on IPv4 UDP dst ports
      • [P2-R1] VPP drops packets based on IPv4 UDP src+dst ports
      • [P2-R1] VPP drops packets based on MAC src addr
    6. [P0-R1] security - cop white-/black-lists
      • [P0-R1] VPP permits packets based on IPv4 src addr
      • [P0-R1] VPP drops packets based on IPv4 src addr
    7. [P2-R1]security - source IPv4 Reverse Path Forwarding (RPF)
      • [P2-R1] - source RPF check on IPv4 src-addr
    8. [P1-R1]qos - ingress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers (cb=color-blind, ca=color-aware)
      • [P1-R1] VPP 1r2c-cb policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 1r2c-ca policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P1-R1] VPP 2r3c-cb policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 2r3c-ca policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 1r2c-cb policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 1r2c-ca policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 2r3c-cb policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 2r3c-ca policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
    9. [TBD] qos - egress policing - 1r2c-(cb,ca), 2r3c-(cb,ca) policers
    10. [P1-R1] carrier grade nat 44, CGN44 [rfc 6598], [rfc 6888]
      • [P1-R1] CGN address-only
      • [P1-R1] CGN address-and-port
      • [TBD] CGN address-only and application layer gateway (ALG)
      • [TBD] CGN address-and-port and ALG
  2. IP4 control plane
    1. [P1-R1] ARP
      • [DONE-R1] VPP responds to ARP request [rfc 826]
      • [P1-R1] VPP sends ARP requests for unknown destinations
    2. [P1-R1] Proxy ARP
      • [P1-R1] VPP responds to ARP requests on behalf of host present on another VPP interface.
    3. [DONE-R1] ICMPv4
      • [DONE-R1] VPP can process ICMP echo request from min to 1500B packet size with 1B increment
      • [DONE-R1] VPP can process ICMP echo request from 1500B to max packet size with 10B increment
      • [DONE-R1] VPP replies to ICMPv4 echo request [rfc 792]
    4. [P2-R1] DHCPv4 proxy
      • [P2-R1] VPP proxies valid DHCPv4 request to DHCPv4 server
      • [P2-R1] VPP proxy ignores invalid DHCPv4 request
    5. [P2-R1] DHCPv4 client
      • [P2-R1] VPP sends DHCPv4 Discover
      • [P2-R1] VPP sends DHCPv4 Request after Offer
      • [P2-R1] VPP honors DHCPv4 lease time
      • [P2-R1] VPP releases allocated address
      • [P2-R1] VPP retries DHCPv4 Discover ???times until ???total second timeout
      • [P2-R1] VPP retries DHCPv4 Request ???times until ???total second timeout
    6. [P1-R2] IKEv2
      • [P1-R2] psk auth
      • [P1-R2] rsa key auth
      • [P1-R2] ID type IPv4 address
      • [P1-R2] ID type (email) [rfc 822]
      • [P1-R2] ID type key-id
      • [P1-R2] test various supported encryption/prf/integrity algs, DH groups, extended sequence number
  3. IP4 encapsulations
    1. [P1-R1] ip4-gre-ip4
      • [P1-R1] VPP can encapsulate IPv4 traffic in GRE
      • [P1-R1] VPP can route IPv4 traffic from GRE tunnel
    2. [P1-R2] ip4-ipsec-ip4
      • [P1-R2] basic connectivity test - create ipsec SA and policy on VPP, test different encryption and integrity algorithms
      • [P1-R2] add/del SA and policy tests
      • [P1-R2] encryprion/integrity key update test
      • [P1-R2] some negative tests (each side has different enryption/integrity alg)
  4. ip6 data plane
    1. [DONE-R1] baseline routing [rfc 2460]
      • [DONE-R1] TG can route to first DUT egress interface
      • [DONE-R1] TG can route to second DUT through first DUT
      • [DONE-R1] TG can route to second DUT egress interface through first DUT
      • [DONE-R1] TG can route to TG through first and second DUT
    2. [P2-R1] vpn baseline routed forwarding (VRF)
      • [P2-R1] TG can ping to DUT ingress interface, VPP configured with two VRFs
      • [P2-R1] TG can ping to DUT egress interface, VPP configured with two VRFs
      • [P2-R1] TG can ping to DUT2 through DUT1, VPP configured with two VRFs
      • [P2-R1] TG can ping to DUT2 egress interface through DUT1, VPP configured with two VRFs
      • [P2-R1] TG can ping to TG through DUT1 and DUT2, VPP configured with two VRFs
    3. [P0-R1] security - iacl with n-tuple classification
      • [P0-R1] VPP can drop packets based on IPv6 src-addr
      • [P0-R1] VPP can drop packets based on IPv6 dst-addr
      • [P0-R1] VPP can drop packets based on IPv6 src-addr and dst-addr
      • [P2-R1] VPP can drop packets based on IPv6 protocol (TCP/UDP)
      • [P2-R1] VPP can drop packets based on IPv6 TCP src port
      • [P2-R1] VPP can drop packets based on IPv6 TCP dst port
      • [P2-R1] VPP can drop packets based on IPv6 TCP src+dst port
      • [P2-R1] VPP can drop packets based on IPv6 UDP src port
      • [P2-R1] VPP can drop packets based on IPv6 UDP dst port
      • [P2-R1] VPP can drop packets based on IPv6 UDP src+dst port
      • [P2-R1] VPP can drop packets based on src MAC + IPv6 UDP src+dst port
    4. [P0-R1] security - cop white-/black-lists
      • [P0-R1] VPP permits packets based on IPv6 src-addr
      • [P0-R1] VPP drops packets based on IPv6 src-addr
    5. [P2-R1] security - source IPv6 Reverse Path Forwarding (RPF)
      • [P2-R1] - source RPF check on IPv6 src-addr
    6. [P1-R1] qos traffic conditioning - ingress rate metering, marking and rate-limiting - 1r2c-(cb,ca), 2r3c-(cb,ca) policers (cb=color-blind, ca=color-aware) [rfc 2475], [rfc 2697], [rfc 2698]
      • [P1-R1] VPP 1r2c-cb policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 1r2c-ca policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P1-R1] VPP 2r3c-cb policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 2r3c-ca policing packets based on DiffServ marking - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 1r2c-cb policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 1r2c-ca policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 2r3c-cb policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
      • [P2-R1] VPP 2r3c-ca policing packets based on packet classify - [TBD-VPP] depends on VPP code ready
  5. ip6 control plane
    1. [P0-R1] router advertisement
      • [P0-R1] VPP transmits RA from IPv6 enabled interface
      • [P0-R1] ??? ^ retransmits ???
      • [P0-R1] VPP ??? handles RA ???
    2. [P0-R1] neighbor discovery [rfc 4861]
      • [P0-R1] VPP validates and replies to IPv6 NS/NA messages
    3. [DONE-R1] ICMPv6
      • [DONE-R1] VPP replies to ICMPv6 echo request
      • [DONE-R1] VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
      • [DONE-R1] VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
    4. [TBD] ip6-ioam (in-band OAM)
    5. [P1-R2] DHCPv6 proxy
      • [P1-R2] VPP proxies valid DHCPv6 request to DHCPv6 server
      • [P1-R2] VPP proxy ignores invalid DHCPv6 request
  6. ip6 encapsulations
    1. [P1-R1] ip4-o-ip6 softwire
      • [P1-R1] lightweight46 [rfc 7596]
      • [P2-R1] map-e [rfc 7597]
      • [P2-R1] map-t [rfc 7599]
    2. [TBD] ip6-segment-routing
    3. [P1-R2] ip6-ipsec-ip6
      • [P1-R2] basic connectivity test - create ipsec SA and policy on VPP, test different encryption and integrity algorithms
      • [P1-R2] add/del SA and policy tests
      • [P1-R2] encryprion/integrity key update test
      • [P1-R2] some negative tests (each side has different enryption/integrity alg)
  7. l2 data plane
    1. [DONE-R1] L2 xconnect
      • [DONE-R1] VPP forwards packets via L2 xconnect in 3-node topology
      • [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 1-to-1, dot1q-to-dot1q
      • [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 1-to-2, dot1q-to-dot1ad
      • [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 2-to-1, dot1ad-to-dot1q
      • [P2-R1] VPP forwards packets via L2 xconnect with VLAN tag acrobatics a'la MEF EVC - 2-to-2, dot1ad-to-dot1ad7
    2. [DONE-R1] Bridge domain
      • [DONE-R1] VPP forwards packets via L2 bridge domain 2 ports
      • [DONE-R1] VPP forwards packets via L2 bridge domain in 3-node topology
      • [DONE-R1] VPP forwards packets via L2 bridge domain in 3-node topology with static L2FIB entries
      • [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 1-to-1, dot1q-to-dot1q
      • [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 1-to-2, dot1q-to-dot1ad
      • [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 2-to-1, dot1ad-to-dot1q
      • [P2-R1] VPP forwards packets via L2 bridge domain with VLAN tag acrobatics a'la MEF EVC - 2-to-2, dot1ad-to-dot1ad
    3. [TBD] irb
      • [TBD] TG can route to DUT ingress irb interface
      • [TBD] TG can route to DUT egress irb interface
      • [TBD] TG can route to DUT2 through DUT1, both DUTs using irb interfaces
      • [TBD] TG can route to DUT2 egress irb interface through DUT1
      • [TBD] TG can route to TG through DUT1 and DUT2, both DUTs using irb interfaces
  8. l2 control plane
    1. [DONE-R1] MAC learning
    2. [P0-R1] MAC split-horizon
    3. [TBD] cdp/lldp
  9. l2 encapsulations
    1. [DONE-R1] l2-vxlan-ipv4
      • [DONE-R1] VPP can encapsulate L2 in VXLAN over V4 using bridge-domain
      • [DONE-R1] VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
    2. [TBD] l2-eth-l2tpv3-ip4
    3. [TBD] l2-eth-l2tpv3-ip6
    4. [TBD] l2-eth-gre-ip4
    5. [TBD] l2-eth-gre-ip6
  10. l2 management plane
    1. [DONE-R1] VPP reports interfaces
  11. mpls
    1. [TBD] vpnv4-mpls-gre-ip4
    2. [TBD] mpls-o-eth
  12. nsh
    1. [TBD] ip4-nsh-gre-ip4
    2. [TBD] ip4-nsh-vxlan-gpe-ip4

VPP deployment topologies:

  1. [DONE-R1] VPP in host user-mode, NIC1-VPP-NIC2
  2. [P0-R1] VPP in host user-mode, NIC1-VPP-VM-VPP-NIC2
  3. [TBD] VPP in host user-mode, VM-VPP-VM
  4. [TBD] VPP in guest user-mode, pci-passthrough to NIC
  5. [TBD] VPP in guest user-mode, sriov to NIC
  6. [P1-R1] VPP in container
  7. [P1-R1] VPP as vSwtich for containers

VPP negative testing:

  1. [TBD] physical interfaces down/up
  2. [TBD] virtual interfaces down/up
  3. [TBD] VPP crash recovery

Other tests:

  1. Memory utilization tests
  2. [TBD] Memory leak tests
  3. [TBD] Consistency of memory footprint for selected configs

CSIT functional tests infra work areas

CSIT Golden Branch

  • Name of the CSIT branch is "csit-verified".
  • It is created by CSIT team in CSIT gerrit project.
  • It represents the version of CSIT code that is tested and verified to work perfectly.
  • At the moment, csit-verified is a git branch, stored in gerrit.fd.io, that points to concrete version of master branch of CSIT project.
  • Link to branch: https://gerrit.fd.io/r/gitweb?p=csit.git;a=shortlog;h=refs%2Fheads%2Fcsit-verified
  • In future, we plan to switch from using branch to tag, because is suites gerrit better than branching (since we use the branch only to point to a version in master).

Run testcases based on vpp code change (no clear ideas yet)

  • for now execute all baseline tests - l2, ip4, ip6
  • future per code change functional and vpp node graph dependencies

Gathering of stats/logs/core dumps from DUTs/TGs after test execution:

  • naming schemes, storage, packaging
  • collection of TELEMETRY
  • plain text - structured
    • one-time snapshot of the system
    • for human driven hand-crafted debugging
    • for human driven part-automated debugging
    • for machine driven fully-automated analysis
  • use Network Data Analytics engine in the future

Honeycomb tests framework

  • NETCONF FrameWork
    • library that understands netconf and ssh
  • Existing CSIT RF (Robot-Framework) and Python libraries / tools extension to use HC/VAT/CLI "driver" to execute things
    • the same logical code - execute across different driver APIs
      • HC, VAT, CLI

VPP project

  • plannig and coordinating with code development
  • specifying and identifying dependencies to/from
    • deployment use cases
    • CSIT

Honeycomb project

  • plannig and coordinating with code development
  • specifying and identifying dependencies to/from
    • deployment use cases
    • CSIT
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/FuncTestPlan&oldid=874"