CSIT/csit-func-tests-reorg-proposal
Contents
DRAFT PROPOSAL: CSIT functional tests re-organisation
Background
CSIT functional job included too many test cases per VPP patch and took quite a long time to execute - vpp-csit-verify-virl-master job used to take 1 hour of execution. To address it, following actions have been taken:
- identify test cases that can be skipped in case of vpp-csit-verify-virl-{branch} job execution:
- tests covered by unittests in vpp (make test)
- tests covering old/obsolete/unmaintained vpp code
- tests covering features with limited use
- scale tests / combined tests
- apply special tag to these test cases
- adapt corresponding bootstrap files to exclude these test from execution per VPP patch
All test cases skipped per VPP patch are executed by csit weekly and semi-weekly jobs.
tests covered by unittests
- dhcp
- Eth2P-Ethip4-Ip4Base-Ip4Dhcpproxy-Func
- Eth2P-Ethip6-Ip6Base-Ip6Dhcpproxy-Func
- gre
- Eth2P-Ethip4Gre-Ip4Base-Func
- iacl
- Eth2P-Eth-L2Xcbase-Iaclbase-Func
- Eth2P-Ethip4-Ip4Base-Iaclbase-Func
- ip4
- Eth2P-Ethip4-Ip4Base-Func
- ip6
- Eth2P-Ethip6-Ip6Base-Func
- ipfix
- Eth2P-Ethip4-Ip4Base-Ip4Ipfixscale-Func
- l2bd
- Eth2P-Eth-L2Bdbasemaclrn-Func
- Eth2P-Eth-L2Bdbasemacstc-Func
- l2xc
- Eth2P-Eth-L2Xcbase-Func
- softwire
- Eth2P-Ethip4--Ethip6Ip4-Ip4Base--Ip6Base-Swiremape-Func
- vrf
- Eth2P-Ethip4-Ip4Basevrf-Func
- Eth2P-Ethip6-Ip6Basevrf-Func
tests covering old/obsolete/unmaintained vpp code
- ipsec
- Eth2P-Ethip4Ipsectnl-Ip4Base-Func
- Eth2P-Ethip4Ipsectpt-Ip4Base-Func
- Eth2P-Ethip6Ipsectnl-Ip6Base-Func
- Eth2P-Ethip6Ipsectpt-Ip6Base-Func
- ipsec+lisp
- Eth2P-Ethip4Ipsectptlispgpe-Ip4Base-Eth-2Vhost-1Vm-Func
- Eth2P-Ethip4Ipsectptlispgpe-Ip6Base-Eth-2Vhost-1Vm-Func
- Eth2P-Ethip4Ipsectptlispgpe-Ip6Basevrf-Func
- Eth2P-Ethip6Ipsectptlispgpe-Ip4Base-Eth-2Vhost-1Vm-Func
- Eth2P-Ethip6Ipsectptlispgpe-Ip6Base-Eth-2Vhost-1Vm-Func
- Eth2P-Ethip4Ipsectptlispgpe-Ip4Base-Func
- Eth2P-Ethip4Ipsectptlispgpe-Ip6Base-Func
- Eth2P-Ethip6Ipsectptlispgpe-Ip4Base-Func
- Eth2P-Ethip6Ipsectptlispgpe-Ip6Base-Func
tests covering features with limited use
- vlan
- Eth2P-Dot1Ad--Dot1Q-L2Bdbasemaclrn-Vlantrans21-Func
- Eth2P-Dot1Ad--Dot1Q-L2Xcbase-Vlantrans21-Func
- Eth2P-Dot1Ad-L2Bdbasemaclrn-Vlantrans22-Func
- Eth2P-Dot1Ad-L2Xcbase-Vlantrans22-Func
- Eth2P-Dot1Q--Dot1Ad-L2Bdbasemaclrn-Vlantrans12-Func
- Eth2P-Dot1Q--Dot1Ad-L2Xcbase-Vlantrans12-Func
- Eth2P-Dot1Q-L2Bdbasemaclrn-Vlantrans11-Func
- Eth2P-Dot1Q-L2Xcbase-Vlantrans11-Func
scale tests / combined tests
- ipfix
- Eth2P-Ethip6-Ip6Base-Ip6Dhcpproxy-Func
- Eth2P-Ethip4-Ip4Base-Ip4Ipfixbase-Func - TC05
- Eth2P-Ethip6-Ip6Base-Ip6Ipfixbase-Func - TC04
test case tag
New tag SKIP_VPP_PATCH has been introduced to identify test cases to be excluded from execution in case of vpp-csit-verify job (i.e. per VPP patch) but not to be skipped in case of csit-vpp-verify job (i.e. per CSIT patch).
bootstrap files
Bootstrap files have been adapted to exclude from execution test cases tagged with SKIP_PATCH tag only (CSIT patch) or test cases tagged with SKIP_PATCH or SKIP_VPP_PATCH tags (VPP patch).
results
Overall execution time of vpp-csit-verify-virl-master job has been reduced to 40mins - it means 20 mins gain per VPP patch with current implementation.
