Difference between revisions of "CSIT/FuncTestPlan"

From fd.io
Jump to: navigation, search
(Created page with "==FD.io IT systems integration== Gerrit-jenkins trigger->job flows - working: * jenkins job: vpp-csit-verify-virl ** patch-based automatic trigger + manual trigger *** specif...")
 
(→‎CSIT functional test cases - working)
Line 57: Line 57:
 
Working functional test cases - 23 total, 20 working in FD.io today.
 
Working functional test cases - 23 total, 20 working in FD.io today.
 
Manual run results report - https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/
 
Manual run results report - https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/
−
 
+
<pre>
 
~/vpp/fd.io/csit$ grep "^| [a-zA-Z]" -nr tests | grep -v Documentation | grep -vi " | "  | grep -v performance
 
~/vpp/fd.io/csit$ grep "^| [a-zA-Z]" -nr tests | grep -v Documentation | grep -vi " | "  | grep -v performance
 
tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
 
tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
Line 82: Line 82:
 
tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
 
tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
 
tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
 
tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation
 +
</pre>
  
 
==CSIT functional test cases - work in progress==
 
==CSIT functional test cases - work in progress==

Revision as of 14:25, 6 April 2016

FD.io IT systems integration

Gerrit-jenkins trigger->job flows - working:

  • jenkins job: vpp-csit-verify-virl
    • patch-based automatic trigger + manual trigger
      • specified text string in vpp gerrit patch comment ("recheck").

Gerrit-jenkins trigger->job flows - work in progress:

  • weekly job that validates latest VPP + latest CSIT code
    • if successful, csit golden branch version is updated (for VPP verify job), and golden VPP version is updated (for CSIT verify job).

CSIT functional tests infra work areas

CSIT Golden Branch

  • Name of the CSIT branch is "csit-verified".
  • It is created by CSIT team in CSIT gerrit project.
  • It represents the version of CSIT code that is tested and verified to work perfectly.
  • At the moment, csit-verified is a git branch, stored in gerrit.fd.io, that points to concrete version of master branch of CSIT project.
  • Link to branch: https://gerrit.fd.io/r/gitweb?p=csit.git;a=shortlog;h=refs%2Fheads%2Fcsit-verified
  • In future, we plan to switch from using branch to tag, because is suites gerrit better than branching (since we use the branch only to point to a version in master).

Run testcases based on vpp code change (no clear ideas yet)

  • for now execute all baseline tests - l2, ip4, ip6
  • future per code change functional and vpp node graph dependencies

Gathering of stats/logs/core dumps from DUTs/TGs after test exec

  • naming schemes, storage, packaging
  • collection of TELEMETRY
  • plain text - structured
    • one-time snapshot of the system
    • for human driven hand-crafted debugging
    • for human driven part-automated debugging
    • for machine driven fully-automated analysis
  • use PaNDA in future?

Honeycomb tests framework

  • NETCONF FrameWork
    • library that understands netconf and ssh
  • Existing CSIT RF (Robot-Framework) and Python libraries / tools extension to use HC/VAT/CLI "driver" to execute things
    • the same logical code - execute across different driver APIs
      • HC, VAT, CLI

VPP project

  • plannig and coordinating with code development
  • specifying and identifying dependencies to/from
    • deployment use cases
    • CSIT

Honeycomb project

  • plannig and coordinating with code development
  • specifying and identifying dependencies to/from
    • deployment use cases
    • CSIT

CSIT functional test cases - working

Working functional test cases - 23 total, 20 working in FD.io today. Manual run results report - https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl/ 

~/vpp/fd.io/csit$ grep "^| [a-zA-Z]" -nr tests | grep -v Documentation | grep -vi " | "  | grep -v performance
tests/suites/vxlan/vxlan_bd_untagged.robot:29:| VPP can encapsulate L2 in VXLAN over V4
tests/suites/vxlan/vxlan_xconnect_untagged.robot:29:| VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
tests/suites/ipv4/ipv4_untagged.robot:31:| VPP replies to ICMPv4 echo request (rfc:2544,5432,3452,4567)
tests/suites/ipv4/ipv4_untagged.robot:40:| TG can route to DUT egress interface
tests/suites/ipv4/ipv4_untagged.robot:49:| TG can route to DUT2 through DUT1
tests/suites/ipv4/ipv4_untagged.robot:58:| TG can route to DUT2 egress interface through DUT1
tests/suites/ipv4/ipv4_untagged.robot:67:| TG can route to TG through DUT1 and DUT2
tests/suites/ipv4/ipv4_untagged.robot:88:| VPP can process ICMP echo request from min to 1500B packet size with 1B increment
tests/suites/ipv4/ipv4_untagged.robot:92:| VPP can process ICMP echo request from 1500B to max packet size with 10B increment
tests/suites/ipv4/ipv4_untagged.robot:110:| VPP responds to ARP request
tests/suites/bridge_domain/test.robot:27:| VPP reports interfaces
tests/suites/bridge_domain/test.robot:30:| Vpp forwards packets via L2 bridge domain 2 ports // not for current virl 3-node topology
tests/suites/bridge_domain/test.robot:42:| Vpp forwards packets via L2 bridge domain in circular topology
tests/suites/bridge_domain/test.robot:58:| Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
tests/suites/l2_xconnect/l2_xconnect_untagged.robot:26:| Vpp forwards packets via L2 xconnect in circular topology
tests/suites/ipv6/ipv6_untagged.robot:33:| VPP replies to ICMPv6 echo request
tests/suites/ipv6/ipv6_untagged.robot:37:| VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
tests/suites/ipv6/ipv6_untagged.robot:41:| VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
tests/suites/ipv6/ipv6_untagged.robot:59:| TG can route to first DUT egress interface
tests/suites/ipv6/ipv6_untagged.robot:64:| TG can route to second DUT through first DUT
tests/suites/ipv6/ipv6_untagged.robot:69:| TG can route to second DUT egress interface through first DUT
tests/suites/ipv6/ipv6_untagged.robot:74:| TG can route to TG through first and second DUT
tests/suites/ipv6/ipv6_untagged.robot:79:| VPP replies to IPv6 Neighbor Solicitation

CSIT functional test cases - work in progress

  • 802.1ad/QinQ.
  • vhost-user.
  • basic HoneyComb handling in tests.

CSIT functional test cases - in planning

Below laundry list of functions that could/should be tested - need to prioritize based on FD.io community needs and feedback, and scheduled for CSIT development. To work and be useful, CSIT development should be done in coordination with VPP Release Candidate content and release schedule.

VPP network functions:

  • ip4 data plane
    • routing
      • [DONE] TG can route to DUT ingress interface
      • [DONE] TG can route to DUT egress interface
      • [DONE] TG can route to DUT2 through DUT1
      • [DONE] TG can route to DUT2 egress interface through DUT1
      • [DONE] TG can route to TG through DUT1 and DUT2
    • security - iacl with n-tuple classification
      • [TBD] VPP drops packets based on IPv4 source addresses (src addr)
      • [TBD] VPP drops packets based on IPv4 destination addresses (dst addr)
      • [TBD] VPP drops packets based on IPv4 dst addr
      • [TBD] VPP drops packets based on IPv4 protocol (TCP/UDP)
      • [TBD] VPP drops packets based on IPv4 TCP src ports
      • [TBD] VPP drops packets based on IPv4 TCP dst ports
      • [TBD] VPP drops packets based on IPv4 TCP src+dst ports
      • [TBD] VPP drops packets based on IPv4 UDP src ports
      • [TBD] VPP drops packets based on IPv4 UDP dst ports
      • [TBD] VPP drops packets based on IPv4 UDP src+dst ports
      • [TBD] VPP drops packets based on MAC src addr
    • security - cop white-/black-lists
      • [TBD] VPP permits packets based on IPv4 src addr
      • [TBD] VPP drops packets based on IPv4 src addr
    • policing - dependency on VPP code readiness
    • carrier-grade-nat - dependency on VPP code readiness
  • ip4 control plane
    • arp
      • [DONE] VPP responds to ARP request (rfc:826)
      • [TBD] VPP sends ARP requests for unknown destinations
    • ICMPv4
      • [DONE] VPP can process ICMP echo request from min to 1500B packet size with 1B increment
      • [DONE] VPP can process ICMP echo request from 1500B to max packet size with 10B increment
      • [DONE] VPP replies to ICMPv4 echo request (rfc:792)
    • DHCPv4 proxy
      • [TBD] VPP proxies valid DHCPv4 request to DHCPv4 server
      • [TBD] VPP proxy ignores invalid DHCPv4 request
    • DHCPv4 client
      • [TBD] VPP sends DHCPv4 Discover
      • [TBD] VPP sends DHCPv4 Request after Offer
      • [TBD] VPP honors DHCPv4 lease time
      • [TBD] VPP releases allocated address
      • [TBD] VPP retries DHCPv4 Discover ???times until ???total second timeout
      • [TBD] VPP retries DHCPv4 Request ???times until ???total second timeout
  • ip4 encapsulations
    • ip4-vxlan-ipv4
      • [DONE] VPP can encapsulate L2 in VXLAN over V4 using bridge-domain
      • [DONE] VPP can pass IPv4 bidirectionally through VXLAN tunnel using l2-xconnect
    • ip4-gre-ip4
      • [TBD] VPP can encapsulate IPv4 traffic in GRE
      • [TBD] VPP can route IPv4 traffic from GRE tunnel
    • ip4-nsh-gre-ip4
    • ip4-nsh-vxlan-gpe-ip4
    • ip4-ipsec-ip4
      • [TBD] basic connectivity test - create ipsec SA and policy on VPP, test different encryption and integrity algorithms
      • [TBD] add/del SA and policy tests
      • [TBD] encryprion/integrity key update test
      • [TBD] some negative tests (each side has different enryption/integrity alg)
  • ikev2
    • [TBD] psk auth
    • [TBD] rsa key auth
    • [TBD] ID type IPv4 address
    • [TBD] ID type rfc822 (email)
    • [TBD] ID type key-id
    • [TBD] test various supported encryption/prf/integrity algs, DH groups, extended sequence number
  • ip6 data plane
    • routing
      • [DONE] TG can route to first DUT egress interface
      • [DONE] TG can route to second DUT through first DUT
      • [DONE] TG can route to second DUT egress interface through first DUT
      • [DONE] TG can route to TG through first and second DUT
    • security - iacl with n-tuple classification
      • [TBD] VPP can drop packets based on source IPv6 address
      • [TBD] VPP can drop packets based on destrination IPv6 address
      • [TBD] VPP can drop packets based on destrination IPv6 address
      • [TBD] VPP can drop packets based on IPv6 protocol (TCP/UDP)
      • [TBD] VPP can drop packets based on IPv6 TCP src port
      • [TBD] VPP can drop packets based on IPv6 TCP dst port
      • [TBD] VPP can drop packets based on IPv6 TCP src+dst port
      • [TBD] VPP can drop packets based on IPv6 UDP src port
      • [TBD] VPP can drop packets based on IPv6 UDP dst port
      • [TBD] VPP can drop packets based on IPv6 UDP src+dst port
      • [TBD] VPP can drop packets based on src MAC + IPv6 UDP src+dst port
    • security - cop white-/black-lists
    • policing
  • ip6 control plane
    • router advertisement
      • [TBD] VPP transmits RA from IPv6 enabled interface
      • [TBD] ??? ^ retransmits ???
      • [TBD] VPP ??? handles RA ???
    • neighbor discovery (rfc4861)
      • [DONE] VPP validates and replies to IPv6 NS/NA messages
    • ICMPv6
      • [DONE] VPP replies to ICMPv6 echo request
      • [DONE] VPP can process ICMPv6 echo request from min to 1500B packet size with 1B increment
      • [DONE] VPP can process ICMPv6 echo request from 1500B to max packet size with 10B increment
    • ip6-ioam (in-band OAM)
    • DHCPv6 proxy
      • [TBD] VPP proxies valid DHCPv6 request to DHCPv6 server
      • [TBD] VPP proxy ignores invalid DHCPv6 request
  • ip6 encapsulations
    • ip4-o-ip6 softwire
    • ip6-segment-routing
    • ip6-ipsec-ip6
      • [TBD] basic connectivity test - create ipsec SA and policy on VPP, test different encryption and integrity algorithms
      • [TBD] add/del SA and policy tests
      • [TBD] encryprion/integrity key update test
      • [TBD] some negative tests (each side has different enryption/integrity alg)
  • l2 data plane
    • irb
    • L2 xconnect
      • [DONE] Vpp forwards packets via L2 xconnect in circular topology
    • Bridge domain
      • [DONE] Vpp forwards packets via L2 bridge domain 2 ports // not for current virl 3-node topology
      • [DONE] Vpp forwards packets via L2 bridge domain in circular topology
      • [DONE] Vpp forwards packets via L2 bridge domain in circular topology with static L2FIB entries
  • l2 control plane
    • cdp/lldp
  • l2 encapsulations
    • l2-eth-l2tpv3-ip4
    • l2-eth-l2tpv3-ip6
  • l2 management plane
      • [DONE] VPP reports interfaces
  • mpls
    • vpnv4-mpls-gre-ip4
    • mpls-o-eth

VPP deployment topologies:

  • VPP in host user-mode, NIC-VPP-NIC
  • VPP in host user-mode, NIC-VPP-VM
  • VPP in host user-mode, VM-VPP-VM
  • VPP in guest user-mode, pci-passthrough to NIC
  • VPP in guest user-mode, sriov to NIC

VPP negative testing:

  • physical interfaces down/up
  • virtual interfaces down/up
  • VPP crash recovery

Other tests:

  • Memory utilization tests
    • Memory leak tests
    • Consistency of memory footprint for selected configs
Retrieved from "https://wiki.fd.io/index.php?title=CSIT/FuncTestPlan&oldid=757"