CSIT/csit-func-tests-reorg-proposal

From fd.io
Jump to: navigation, search

DRAFT PROPOSAL: CSIT functional tests re-organisation

Background

CSIT functional job included too many test cases per VPP patch and took quite a long time to execute - vpp-csit-verify-virl-master job used to take 1 hour of execution. To address it, following actions have been taken:

  • identify test cases that can be skipped in case of vpp-csit-verify-virl-{branch} job execution:
    • tests covered by unittests in vpp (make test)
    • tests covering old/obsolete/unmaintained vpp code
    • tests covering features with limited use
    • scale tests / combined tests
  • apply special tag to these test cases
  • adapt corresponding bootstrap files to exclude these test from execution per VPP patch

All test cases skipped per VPP patch are executed by csit weekly and semi-weekly jobs.

tests covered by unittests

  1. dhcp
    • Eth2P-Ethip4-Ip4Base-Ip4Dhcpproxy-Func
    • Eth2P-Ethip6-Ip6Base-Ip6Dhcpproxy-Func
  2. gre
    • Eth2P-Ethip4Gre-Ip4Base-Func
  3. iacl
    • Eth2P-Eth-L2Xcbase-Iaclbase-Func
    • Eth2P-Ethip4-Ip4Base-Iaclbase-Func
  4. ip4
    • Eth2P-Ethip4-Ip4Base-Func
  5. ip6
    • Eth2P-Ethip6-Ip6Base-Func
  6. ipfix
    • Eth2P-Ethip4-Ip4Base-Ip4Ipfixscale-Func
  7. l2bd
    • Eth2P-Eth-L2Bdbasemaclrn-Func
    • Eth2P-Eth-L2Bdbasemacstc-Func
  8. l2xc
    • Eth2P-Eth-L2Xcbase-Func
  9. softwire
    • Eth2P-Ethip4--Ethip6Ip4-Ip4Base--Ip6Base-Swiremape-Func
  10. vrf
    • Eth2P-Ethip4-Ip4Basevrf-Func
    • Eth2P-Ethip6-Ip6Basevrf-Func

tests covering old/obsolete/unmaintained vpp code

  1. ipsec
    • Eth2P-Ethip4Ipsectnl-Ip4Base-Func
    • Eth2P-Ethip4Ipsectpt-Ip4Base-Func
    • Eth2P-Ethip6Ipsectnl-Ip6Base-Func
    • Eth2P-Ethip6Ipsectpt-Ip6Base-Func
  2. ipsec+lisp
    • Eth2P-Ethip4Ipsectptlispgpe-Ip4Base-Eth-2Vhost-1Vm-Func
    • Eth2P-Ethip4Ipsectptlispgpe-Ip6Base-Eth-2Vhost-1Vm-Func
    • Eth2P-Ethip4Ipsectptlispgpe-Ip6Basevrf-Func
    • Eth2P-Ethip6Ipsectptlispgpe-Ip4Base-Eth-2Vhost-1Vm-Func
    • Eth2P-Ethip6Ipsectptlispgpe-Ip6Base-Eth-2Vhost-1Vm-Func
    • Eth2P-Ethip4Ipsectptlispgpe-Ip4Base-Func
    • Eth2P-Ethip4Ipsectptlispgpe-Ip6Base-Func
    • Eth2P-Ethip6Ipsectptlispgpe-Ip4Base-Func
    • Eth2P-Ethip6Ipsectptlispgpe-Ip6Base-Func

tests covering features with limited use

  1. vlan
    • Eth2P-Dot1Ad--Dot1Q-L2Bdbasemaclrn-Vlantrans21-Func
    • Eth2P-Dot1Ad--Dot1Q-L2Xcbase-Vlantrans21-Func
    • Eth2P-Dot1Ad-L2Bdbasemaclrn-Vlantrans22-Func
    • Eth2P-Dot1Ad-L2Xcbase-Vlantrans22-Func
    • Eth2P-Dot1Q--Dot1Ad-L2Bdbasemaclrn-Vlantrans12-Func
    • Eth2P-Dot1Q--Dot1Ad-L2Xcbase-Vlantrans12-Func
    • Eth2P-Dot1Q-L2Bdbasemaclrn-Vlantrans11-Func
    • Eth2P-Dot1Q-L2Xcbase-Vlantrans11-Func

scale tests / combined tests

  1. ipfix
    • Eth2P-Ethip6-Ip6Base-Ip6Dhcpproxy-Func
    • Eth2P-Ethip4-Ip4Base-Ip4Ipfixbase-Func - TC05
    • Eth2P-Ethip6-Ip6Base-Ip6Ipfixbase-Func - TC04

test case tag

New tag SKIP_VPP_PATCH has been introduced to identify test cases to be excluded from execution in case of vpp-csit-verify job (i.e. per VPP patch) but not to be skipped in case of csit-vpp-verify job (i.e. per CSIT patch).

bootstrap files

Bootstrap files have been adapted to exclude from execution test cases tagged with SKIP_PATCH tag only (CSIT patch) or test cases tagged with SKIP_PATCH or SKIP_VPP_PATCH tags (VPP patch).

results

Overall execution time of vpp-csit-verify-virl-master job has been reduced to 40mins - it means 20 mins gain per VPP patch with current implementation.