Difference between revisions of "CSIT/TestFailuresTracking"
From fd.io
< CSIT
(→(M) tests with 9000B payload frames not forwarded over memif interfaces =: Add VPP-2091 as the otehr ticket.) |
|||
| (104 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | = CSIT Test Failure Clasification = | |
All known CSIT failures grouped and listed in the following order: | All known CSIT failures grouped and listed in the following order: | ||
| Line 12: | Line 12: | ||
** Within each sub-group: most common use cases followed by less common. | ** Within each sub-group: most common use cases followed by less common. | ||
| − | + | = CSIT Test Fixing Priorities = | |
| − | + | Test fixing work priorities defined as follows: | |
| − | + | * (H)igh priority, most common use cases and most common test code. | |
| − | + | * (M)edium priority, specific HW and pervasive test code issue. | |
| − | + | * (L)ow priority, corner cases and external dependencies. | |
| − | == | + | = Current Failures = |
| + | |||
| + | == Deterministic Failures == | ||
=== In Trending === | === In Trending === | ||
| − | ==== ( | + | ==== (H) 3nb-spr hoststack: interface not up after first test ==== |
| − | * | + | * last update: 2024-02-07 |
| − | + | * work-to-fix: medium | |
| − | + | * rca: After first test, HundredGigabitEthernetab/0/0 never goes up within the run. Not sure which part f test setup is missing, the tests do work correctly on other testbeds. | |
| − | + | * test: All subsequent tests. | |
| − | + | * frequency: 100% | |
| − | + | * testbed: 3nb-apr | |
| − | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-3nb-spr/87/log.html.gz#s1-s1-s1-s1-s1-t1-k2-k3-k7-k7 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1942 CSIT-1942] | |
| − | + | ||
| − | ==== ( | + | ==== (H) Zero traffic reported in udpquic tests ==== |
| − | * | + | * last update: 2024-02-07 |
| − | + | * work-to-fix: medium | |
| − | + | * rca: There are errors when closing sessions. The current CSIT logic happens to report this as a passing test with zero traffic, which is wrong. | |
| − | + | * test: All tests udpquic tests | |
| − | + | * frequency: 100% | |
| − | + | * testbed: all running the tests | |
| − | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-3n-icx/217/log.html.gz#s1-s1-s1-s1-s4-t1-k2-k4-k17 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1935 CSIT-1935] | |
| − | + | ||
| − | ==== ( | + | ==== (H) 3na-spr, 2n-zn2: VPP fails to start in first test cases if dpdk/mlx5 ==== |
| − | * | + | * last update: 2024-02-07 |
| − | + | * work-to-fix: medium | |
| − | + | * rca: DUT1 fails to boot up in first test case. If DUT2 is present, it fails to start in second test case. Other test cases in the run are unaffected. This looks like an infra issue, ansible cleanup is doing something wrong, hard to tell what. | |
| − | + | * test: First test case, unless it uses AVF driver. | |
| − | + | * frequency: 100% | |
| − | + | * testbed: 3na-spr, 2n-zn2 | |
| − | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-3na-spr/139/log.html.gz#s1-s1-s1-s1-s1-t1-k2-k3-k2-k5-k1-k1-k1-k1 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1939 CSIT-1939] | |
| − | + | ||
| − | ==== (M) | + | ==== (M) DPDK 23.03 testpmd startup fails on some testbeds ==== |
| − | * | + | * last update: 2023-11-06 |
| − | + | * work-to-fix: medium | |
| − | + | * rca: The DUT-DUT link sometimes does not go up. The same consequences as CSIT-1848 but affects more testbed+NIC combinations. Can be fixed by a different startup procedure (better detection before for restart). | |
| − | + | * test: Testpmd (no vpp). Around half of tested+NIC combinations are affected. | |
| − | + | * frequency: Low, ~1%, as in most testbeds the link does go up. | |
| − | + | * testbed: multiple | |
| − | + | * example:https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2310-2n-zn2/8/log.html.gz#s1-s1-s1-s1-t6-k2-k4 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1904 CSIT-1904] | |
| − | + | ||
| − | === | + | ==== (M) 2n-spr: zero traffic on cx7 rdma ==== |
| − | + | * last update: 2023-06-22 | |
| + | * work-to-fix: medium | ||
| + | * rca: VPP reports "tx completion errors", more investigation ongoing. | ||
| + | * test: All tests on 2n-spr with 200Ge2P1Cx7Veat NIC and RDMA driver. | ||
| + | * frequency: always (since 2n-spr was set up) | ||
| + | * testbed: 2n-spr | ||
| + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-spr/58/log.html.gz#s1-s1-s1-s5-s19-t1-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1 | ||
| + | * ticket: [https://jira.fd.io/browse/CSIT-1906 CSIT-1906] | ||
| + | * note: Also would affect 3n-alt with mlxc6 and rdma. Will probably be made invisible by removing rdma (except mlxc5) from jobspecs. | ||
| − | + | ==== (M) Lossy trials in nat udp mlx5 tests ==== | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | * last update: 2024-02-07 | |
| + | * work-to-fix: | ||
| + | * rca: VPP counters suggest the packet is lost somewhere between TG on in-side [1] and VPP [2]. | ||
| + | * test: It is affecting only cx7 with mlx5 driver (not e810cq with avf driver), only udp tests (not tcp or other), and interestingly both ASTF (both cps and tput for nat44ed) and STL (det44) traffic profiles. It does not affect corresponding ASTF tests with ip4base routing. | ||
| + | * frequency: depends on scale, 100% on high scale tests. | ||
| + | * testbed: 2n-icx, 2n-spr. | ||
| + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-spr/45/log.html.gz#s1-s1-s1-s2-s49-t2-k2-k14-k14 | ||
| + | * ticket: [https://jira.fd.io/browse/CSIT-1929 CSIT-1929] | ||
| − | + | ==== (L) 3n-icx: negative ipackets on TB38 AVF 4c l2patch ==== | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | * | + | * last update: 2023-04-19 |
| − | + | * work-to-fix: | |
| − | + | * rca: | |
| − | + | * test: TB38 AVF 4c e810cq, earlier l2patch nowadays eth-l2xcbase | |
| − | + | * frequency: always | |
| − | + | * testbed: 3n-icx (only TB38, never TB37) | |
| − | * | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/214/log.html.gz#s1-s1-s1-s5-s3-t3-k2-k9-k8-k13-k1-k2 |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1901 CSIT-1901] | |
| − | + | ||
| − | + | ==== (L) 2n-tx2: af_xdp mrr failures ==== | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | * | + | * last update: 2023-11-08 |
| − | + | * work-to-fix: | |
| − | + | * rca: Some workers see no traffic, "error af_xdp_device_input_refill_db: rx poll() failed: Bad address" in show hardware. More examination needed. | |
| − | + | * test: ip4 and ip6, base and scale | |
| − | + | * frequency: more than 80% on a subset of cases, 100% on (multicore) 2n-tx2 | |
| − | + | * testbed: 2n-tx2; to a lesser extent also 2n-clx and 2n-icx, where just decreased performance (and failure in ndrpdr) is more likely outcome. | |
| − | * | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-spr/120/log.html.gz#s1-s1-s1-s2-s2-t2-k2-k10-k9-k14-k1-k1-k1-k1 |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1922 CSIT-1922] | |
| − | + | ||
| − | + | === Not In Trending === | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | ==== (M) | + | ==== (M) tests with 9000B payload frames not forwarded over memif interfaces ===== |
| − | * | + | * last update: 2023-02-09 |
| − | + | * work-to-fix: hard | |
| − | ** | + | * test: 9000B + memif |
| − | * | + | * testbed: 2n-skx, 3n-skx, 2n-clx |
| − | ** | + | * examples: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2202-2n-skx/33/log.html.gz#s1-s1-s1-s1-s1 2n-skx Memif] |
| − | ** | + | * ticket: [https://jira.fd.io/browse/CSIT-1808 CSIT-1808] VPP-2091 |
| − | + | ||
| − | + | ==== (M) IMIX 4c tests may fail PDR due to ~10% loss ===== | |
| − | ** | + | |
| + | * last update: 2024-02-07 | ||
| + | * work-to-fix: hard | ||
| + | * rca: Only seen in coverage tests so far, more data needed. | ||
| + | * test: various high-performance, mostly mlx5 | ||
| + | * frequency: <1% | ||
| + | * testbed: 2n-icx, 3n-icx, 3na-spr | ||
| + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-coverage-2310-2n-icx/1/log.html.gz#s1-s1-s1-s3-t9-k2-k5-k16-k9 | ||
| + | * ticket: [https://jira.fd.io/browse/CSIT-1943 CSIT-1943] | ||
| + | |||
| + | ==== (L) l3fwd error in 200Ge2P1Cx7Veat-Mlx5 test with 9000B ===== | ||
| + | |||
| + | * last update: 2023-06-28 | ||
| + | * work-to-fix: medium | ||
| + | * test: 9000B + Cx7 with DPDK DUT | ||
| + | * testbed: 2n-icx | ||
| + | * examples: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-coverage-2306-2n-icx/2/log.html.gz#s1-s1-s1-s4-t6-k2-k4] | ||
| + | * ticket: [https://jira.fd.io/browse/CSIT-1924 CSIT-1924] | ||
==== (L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail ==== | ==== (L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail ==== | ||
| − | * | + | * last update: 2023-07-12 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: Ramp-up trial takes more than 5 minutes so sessions are timing out. | |
| − | + | * test: 64B-avf-ethip4tcp-nat44ed-h262144-p63-s16515072-cps-ndrpdr 1c 2c 4c, 64B-avf-ethip4udp-nat44ed-h262144-p63-s16515072-cps-ndrpdr 1c 2c 4c | |
| − | + | * frequency: always | |
| − | + | * testbeds: 2n-skx, 2n-clx, 2n-icx | |
| − | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2306-2n-clx/8/log.html.gz#s1-s1-s1-s1-s14-t2-k2-k15-k1 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1799 CSIT-1799] | |
| − | + | ||
==== (L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions ==== | ==== (L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions ==== | ||
| − | * | + | * last update: 2023-07-12 |
| − | + | * work-to-fix: medium | |
| − | + | * rca: One possible cause is CSIT not counting ramp-up rate properly for IMIX (if multiple packets belong to the same session). | |
| − | + | * test: IMIX over 1M sessions bidir | |
| − | + | * frequency: always | |
| − | + | * testbed: 2n-skx, 2n-clx, 2n-icx | |
| − | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2306-2n-clx/7/log.html.gz#s1-s1-s1-s1-s2-t4-k2-k11-k1-k2 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1884 CSIT-1884] | |
| − | + | ||
| − | == | + | == Occasional Failures == |
| − | === | + | === In Trending === |
==== (H) 2n-icx: NFV density VPP does not start in container ==== | ==== (H) 2n-icx: NFV density VPP does not start in container ==== | ||
| − | * | + | * last update: before 2023-01-31 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: | |
| − | + | * test: all subsequent | |
| − | + | * frequency: medium | |
| − | + | * testbed: 2n-icx | |
| − | + | * example: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/57/log.html.gz 2n-icx mrr], [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/48/log.html.gz#s1-s1-s1-s5-s8-t1 2n-icx ndrpdr] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1881 CSIT-1881] | |
| − | + | * note: Once VPP breaks, all subsequent tests fail. Even all subsequent builds will be failing until Peter makes TB working again. Although it's failing with medium frequency when it happens it breaks all subsequent builds on the TB therefore [H] priority. | |
| − | ==== (M) | + | ==== (M) 3n-alt: high scale ipsec policy tests may crash VPP ==== |
| − | * | + | * last update: 2024-02-07 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: Vpp is crashing without creating core. | |
| − | + | * test: policy ipsec tests, large scale increases probability. | |
| − | + | * frequency: 15% and lower. | |
| − | + | * testbed: 3n-alt, also seen rarely on 3n-icx. | |
| − | + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/357/log.html.gz#s1-s1-s1-s1-s15-t4-k3-k5-k1-k1 | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1938 CSIT-1938] | |
| − | + | ||
| − | ==== (M) 3n-icx: wireguard 100 and 1000 tunnels mlrsearch tests failing with 2c and 4c ==== | + | ==== (M) 3n-icx, 3n-snr: wireguard 100 and 1000 tunnels mlrsearch tests failing with 2c and 4c ==== |
| − | * | + | * last update: before 2023-01-31 |
| − | + | * work-to-fix: easy | |
| − | + | * rca: | |
| − | + | * test: wireguard 100 tunnels and more | |
| − | + | * frequency: high | |
| − | + | * testbed: 3n-icx, 3n-snr | |
| − | + | * examples: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/56/log.html.gz#s1-s1-s1-s3-s8-t4 3n-icx] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1886 CSIT-1886] | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ==== (M) e810cq sometimes reporting link down ==== | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | * last update: 2024-02-07 | |
| + | * work-to-fix: hard | ||
| + | * rca: Mostly causing failure symptom of TRex complaining about link down. Probably also causes zero throughput in one direction in ASTF tests or even defice test failures. More frequent with more performant tests (L2) but seen affecting any test on occasion. | ||
| + | * test: Any that uses Intel-E810CQ NIC. | ||
| + | * frequency: <20% | ||
| + | * testbed: all with the NIC. | ||
| + | * examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-spr/45/log.html.gz#s1-s1-s1-s2-s24-t1-k2-k12-k14 | ||
| + | * ticket: [https://jira.fd.io/browse/CSIT-1936 CSIT-1936] | ||
| + | |||
| + | == Rare Failures == | ||
| + | |||
| + | === In Trending === | ||
==== (M) 3n-icx, 3n-snr: 1518B IPsec packets not passing ==== | ==== (M) 3n-icx, 3n-snr: 1518B IPsec packets not passing ==== | ||
| − | * | + | * last update: before 2023-01-31 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: | |
| − | + | * test: all AVF crypto | |
| − | + | * frequency: low | |
| − | + | * testbed: 3n-skx, 3n-icx, 3n-snr | |
| − | + | * example: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/197/log.html.gz#s1-s1-s1-s1-s4-t1 3n-icx daily], [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-snr/32/log.html.gz#s1-s1-s1-s1-s4-t1 3n-snr], [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-icx/57/log.html.gz#s1-s1-s1-s1-s4-t1 3n-icx weekly] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1827 CSIT-1827] | |
| − | + | ||
==== (M) all testbeds: mlrsearch fails to find NDR rate ==== | ==== (M) all testbeds: mlrsearch fails to find NDR rate ==== | ||
| − | * | + | * last update: before 2023-06-22 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: On 3n-tsh, the symptom is TRex reporting ierrors, only in one direction. Other testbeds may have a different symptom, but failures there are less frequent. | |
| − | + | * test: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected) | |
| − | + | * frequency: low | |
| − | + | * testbed: 3n-tsh, 3n-alt, 2n-clx | |
| − | + | * example: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3n-tsh/14/log.html.gz#s1-s1-s1-s5-s8-t2] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1804 CSIT-1804] | |
| − | + | ||
==== (M) all testbeds: AF_XDP mlrsearch fails to find NDR rate ==== | ==== (M) all testbeds: AF_XDP mlrsearch fails to find NDR rate ==== | ||
| − | * | + | * last update: before 2023-01-31 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: | |
| − | + | * test: af-xdp multicore tests | |
| − | + | * frequency: low | |
| − | + | * testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx | |
| − | + | * example: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-skx/202/log.html.gz#s1-s1-s1-s2-s4-t3 2n-skx], [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-clx/152/log.html.gz#s1-s1-s1-s5-s12-t3 2n-clx] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1802 CSIT-1802] | |
| − | + | * note: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100 | |
==== (L) all testbeds: vpp create avf interface failure in multi-core configs ==== | ==== (L) all testbeds: vpp create avf interface failure in multi-core configs ==== | ||
| − | * | + | * last update: 2023-02-06 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: issue in Intel FVL driver | |
| − | + | * test: multicore AVF | |
| − | + | * frequency: low | |
| − | + | * testbed: all testbeds | |
| − | + | * example: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1257/log.html.gz#s1-s1-s1-s5-s24-t2 2n-clx], [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/197/log.html.gz#s1-s1-s1-s5-s1-t3 3n-icx] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1782 CSIT-1782] | |
| − | + | * note: A long standing issue without a final permanent fix. | |
==== (L) all testbeds: nat44det 4M and 16M scale 1 session not established ==== | ==== (L) all testbeds: nat44det 4M and 16M scale 1 session not established ==== | ||
| − | * | + | * last update: 2023-02-14 |
| − | + | * work-to-fix: hard | |
| − | + | * rca: unknown | |
| − | + | * test: nat44det udp 4m and 16m (64k is ok, 1m can fail but rarely than bigger scales) | |
| − | + | * frequency: low | |
| − | + | * testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx | |
| − | + | * example: [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/672/log.html.gz#s1-s1-s1-s2-s22-t3 2n-zn2], [https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-clx/1271/log.html.gz#s1-s1-s1-s2-s60-t1-k2-k11-k1-k2 2n-clx] | |
| − | + | * ticket: [https://jira.fd.io/browse/CSIT-1795 CSIT-1795] | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ==== (L) TRex may wrongly detect link bandwidth ==== | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | * last update: 2024-02-07 | |
| + | * work-to-fix: hard | ||
| + | * rca: Quite rare failure affecting unpredictable tests. Perhaps a less severe symptom of CSIT-1936. | ||
| + | * test: No obvious pattern due to low frequency | ||
| + | * frequency: <0.4% | ||
| + | * testbed: recently seen on 3n-tsh and 3nb-spr | ||
| + | * example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2310-3nb-spr/28/log.html.gz#s1-s1-s1-s1-s4-t1-k2-k13-k14 | ||
| + | * ticket: [https://jira.fd.io/browse/CSIT-1941 CSIT-1941] | ||
| − | + | = Past Failures = | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
Latest revision as of 14:49, 7 February 2024
Contents
- 1 CSIT Test Failure Clasification
- 2 CSIT Test Fixing Priorities
- 3 Current Failures
- 3.1 Deterministic Failures
- 3.1.1 In Trending
- 3.1.1.1 (H) 3nb-spr hoststack: interface not up after first test
- 3.1.1.2 (H) Zero traffic reported in udpquic tests
- 3.1.1.3 (H) 3na-spr, 2n-zn2: VPP fails to start in first test cases if dpdk/mlx5
- 3.1.1.4 (M) DPDK 23.03 testpmd startup fails on some testbeds
- 3.1.1.5 (M) 2n-spr: zero traffic on cx7 rdma
- 3.1.1.6 (M) Lossy trials in nat udp mlx5 tests
- 3.1.1.7 (L) 3n-icx: negative ipackets on TB38 AVF 4c l2patch
- 3.1.1.8 (L) 2n-tx2: af_xdp mrr failures
- 3.1.2 Not In Trending
- 3.1.2.1 (M) tests with 9000B payload frames not forwarded over memif interfaces =
- 3.1.2.2 (M) IMIX 4c tests may fail PDR due to ~10% loss =
- 3.1.2.3 (L) l3fwd error in 200Ge2P1Cx7Veat-Mlx5 test with 9000B =
- 3.1.2.4 (L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail
- 3.1.2.5 (L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions
- 3.1.1 In Trending
- 3.2 Occasional Failures
- 3.3 Rare Failures
- 3.3.1 In Trending
- 3.3.1.1 (M) 3n-icx, 3n-snr: 1518B IPsec packets not passing
- 3.3.1.2 (M) all testbeds: mlrsearch fails to find NDR rate
- 3.3.1.3 (M) all testbeds: AF_XDP mlrsearch fails to find NDR rate
- 3.3.1.4 (L) all testbeds: vpp create avf interface failure in multi-core configs
- 3.3.1.5 (L) all testbeds: nat44det 4M and 16M scale 1 session not established
- 3.3.1.6 (L) TRex may wrongly detect link bandwidth
- 3.3.1 In Trending
- 3.1 Deterministic Failures
- 4 Past Failures
CSIT Test Failure Clasification
All known CSIT failures grouped and listed in the following order:
- Always failing followed by sometimes failing.
- Always failing tests:
- Most common use cases followed by less common.
- Sometimes failing tests:
- Most frequently failing followed by less frequently failing.
- High frequency 50%-100%
- medium frequency 10%-50%
- low frequency 0%-10%.
- Within each sub-group: most common use cases followed by less common.
- Most frequently failing followed by less frequently failing.
CSIT Test Fixing Priorities
Test fixing work priorities defined as follows:
- (H)igh priority, most common use cases and most common test code.
- (M)edium priority, specific HW and pervasive test code issue.
- (L)ow priority, corner cases and external dependencies.
Current Failures
Deterministic Failures
In Trending
(H) 3nb-spr hoststack: interface not up after first test
- last update: 2024-02-07
- work-to-fix: medium
- rca: After first test, HundredGigabitEthernetab/0/0 never goes up within the run. Not sure which part f test setup is missing, the tests do work correctly on other testbeds.
- test: All subsequent tests.
- frequency: 100%
- testbed: 3nb-apr
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-3nb-spr/87/log.html.gz#s1-s1-s1-s1-s1-t1-k2-k3-k7-k7
- ticket: CSIT-1942
(H) Zero traffic reported in udpquic tests
- last update: 2024-02-07
- work-to-fix: medium
- rca: There are errors when closing sessions. The current CSIT logic happens to report this as a passing test with zero traffic, which is wrong.
- test: All tests udpquic tests
- frequency: 100%
- testbed: all running the tests
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-3n-icx/217/log.html.gz#s1-s1-s1-s1-s4-t1-k2-k4-k17
- ticket: CSIT-1935
(H) 3na-spr, 2n-zn2: VPP fails to start in first test cases if dpdk/mlx5
- last update: 2024-02-07
- work-to-fix: medium
- rca: DUT1 fails to boot up in first test case. If DUT2 is present, it fails to start in second test case. Other test cases in the run are unaffected. This looks like an infra issue, ansible cleanup is doing something wrong, hard to tell what.
- test: First test case, unless it uses AVF driver.
- frequency: 100%
- testbed: 3na-spr, 2n-zn2
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-3na-spr/139/log.html.gz#s1-s1-s1-s1-s1-t1-k2-k3-k2-k5-k1-k1-k1-k1
- ticket: CSIT-1939
(M) DPDK 23.03 testpmd startup fails on some testbeds
- last update: 2023-11-06
- work-to-fix: medium
- rca: The DUT-DUT link sometimes does not go up. The same consequences as CSIT-1848 but affects more testbed+NIC combinations. Can be fixed by a different startup procedure (better detection before for restart).
- test: Testpmd (no vpp). Around half of tested+NIC combinations are affected.
- frequency: Low, ~1%, as in most testbeds the link does go up.
- testbed: multiple
- example:https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-iterative-2310-2n-zn2/8/log.html.gz#s1-s1-s1-s1-t6-k2-k4
- ticket: CSIT-1904
(M) 2n-spr: zero traffic on cx7 rdma
- last update: 2023-06-22
- work-to-fix: medium
- rca: VPP reports "tx completion errors", more investigation ongoing.
- test: All tests on 2n-spr with 200Ge2P1Cx7Veat NIC and RDMA driver.
- frequency: always (since 2n-spr was set up)
- testbed: 2n-spr
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-spr/58/log.html.gz#s1-s1-s1-s5-s19-t1-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1
- ticket: CSIT-1906
- note: Also would affect 3n-alt with mlxc6 and rdma. Will probably be made invisible by removing rdma (except mlxc5) from jobspecs.
(M) Lossy trials in nat udp mlx5 tests
- last update: 2024-02-07
- work-to-fix:
- rca: VPP counters suggest the packet is lost somewhere between TG on in-side [1] and VPP [2].
- test: It is affecting only cx7 with mlx5 driver (not e810cq with avf driver), only udp tests (not tcp or other), and interestingly both ASTF (both cps and tput for nat44ed) and STL (det44) traffic profiles. It does not affect corresponding ASTF tests with ip4base routing.
- frequency: depends on scale, 100% on high scale tests.
- testbed: 2n-icx, 2n-spr.
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-spr/45/log.html.gz#s1-s1-s1-s2-s49-t2-k2-k14-k14
- ticket: CSIT-1929
(L) 3n-icx: negative ipackets on TB38 AVF 4c l2patch
- last update: 2023-04-19
- work-to-fix:
- rca:
- test: TB38 AVF 4c e810cq, earlier l2patch nowadays eth-l2xcbase
- frequency: always
- testbed: 3n-icx (only TB38, never TB37)
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/214/log.html.gz#s1-s1-s1-s5-s3-t3-k2-k9-k8-k13-k1-k2
- ticket: CSIT-1901
(L) 2n-tx2: af_xdp mrr failures
- last update: 2023-11-08
- work-to-fix:
- rca: Some workers see no traffic, "error af_xdp_device_input_refill_db: rx poll() failed: Bad address" in show hardware. More examination needed.
- test: ip4 and ip6, base and scale
- frequency: more than 80% on a subset of cases, 100% on (multicore) 2n-tx2
- testbed: 2n-tx2; to a lesser extent also 2n-clx and 2n-icx, where just decreased performance (and failure in ndrpdr) is more likely outcome.
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-spr/120/log.html.gz#s1-s1-s1-s2-s2-t2-k2-k10-k9-k14-k1-k1-k1-k1
- ticket: CSIT-1922
Not In Trending
(M) tests with 9000B payload frames not forwarded over memif interfaces =
- last update: 2023-02-09
- work-to-fix: hard
- test: 9000B + memif
- testbed: 2n-skx, 3n-skx, 2n-clx
- examples: 2n-skx Memif
- ticket: CSIT-1808 VPP-2091
(M) IMIX 4c tests may fail PDR due to ~10% loss =
- last update: 2024-02-07
- work-to-fix: hard
- rca: Only seen in coverage tests so far, more data needed.
- test: various high-performance, mostly mlx5
- frequency: <1%
- testbed: 2n-icx, 3n-icx, 3na-spr
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-report-coverage-2310-2n-icx/1/log.html.gz#s1-s1-s1-s3-t9-k2-k5-k16-k9
- ticket: CSIT-1943
(L) l3fwd error in 200Ge2P1Cx7Veat-Mlx5 test with 9000B =
- last update: 2023-06-28
- work-to-fix: medium
- test: 9000B + Cx7 with DPDK DUT
- testbed: 2n-icx
- examples: [1]
- ticket: CSIT-1924
(L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail
- last update: 2023-07-12
- work-to-fix: hard
- rca: Ramp-up trial takes more than 5 minutes so sessions are timing out.
- test: 64B-avf-ethip4tcp-nat44ed-h262144-p63-s16515072-cps-ndrpdr 1c 2c 4c, 64B-avf-ethip4udp-nat44ed-h262144-p63-s16515072-cps-ndrpdr 1c 2c 4c
- frequency: always
- testbeds: 2n-skx, 2n-clx, 2n-icx
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2306-2n-clx/8/log.html.gz#s1-s1-s1-s1-s14-t2-k2-k15-k1
- ticket: CSIT-1799
(L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions
- last update: 2023-07-12
- work-to-fix: medium
- rca: One possible cause is CSIT not counting ramp-up rate properly for IMIX (if multiple packets belong to the same session).
- test: IMIX over 1M sessions bidir
- frequency: always
- testbed: 2n-skx, 2n-clx, 2n-icx
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2306-2n-clx/7/log.html.gz#s1-s1-s1-s1-s2-t4-k2-k11-k1-k2
- ticket: CSIT-1884
Occasional Failures
In Trending
(H) 2n-icx: NFV density VPP does not start in container
- last update: before 2023-01-31
- work-to-fix: hard
- rca:
- test: all subsequent
- frequency: medium
- testbed: 2n-icx
- example: 2n-icx mrr, 2n-icx ndrpdr
- ticket: CSIT-1881
- note: Once VPP breaks, all subsequent tests fail. Even all subsequent builds will be failing until Peter makes TB working again. Although it's failing with medium frequency when it happens it breaks all subsequent builds on the TB therefore [H] priority.
(M) 3n-alt: high scale ipsec policy tests may crash VPP
- last update: 2024-02-07
- work-to-fix: hard
- rca: Vpp is crashing without creating core.
- test: policy ipsec tests, large scale increases probability.
- frequency: 15% and lower.
- testbed: 3n-alt, also seen rarely on 3n-icx.
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-alt/357/log.html.gz#s1-s1-s1-s1-s15-t4-k3-k5-k1-k1
- ticket: CSIT-1938
(M) 3n-icx, 3n-snr: wireguard 100 and 1000 tunnels mlrsearch tests failing with 2c and 4c
- last update: before 2023-01-31
- work-to-fix: easy
- rca:
- test: wireguard 100 tunnels and more
- frequency: high
- testbed: 3n-icx, 3n-snr
- examples: 3n-icx
- ticket: CSIT-1886
(M) e810cq sometimes reporting link down
- last update: 2024-02-07
- work-to-fix: hard
- rca: Mostly causing failure symptom of TRex complaining about link down. Probably also causes zero throughput in one direction in ASTF tests or even defice test failures. More frequent with more performant tests (L2) but seen affecting any test on occasion.
- test: Any that uses Intel-E810CQ NIC.
- frequency: <20%
- testbed: all with the NIC.
- examples: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-spr/45/log.html.gz#s1-s1-s1-s2-s24-t1-k2-k12-k14
- ticket: CSIT-1936
Rare Failures
In Trending
(M) 3n-icx, 3n-snr: 1518B IPsec packets not passing
- last update: before 2023-01-31
- work-to-fix: hard
- rca:
- test: all AVF crypto
- frequency: low
- testbed: 3n-skx, 3n-icx, 3n-snr
- example: 3n-icx daily, 3n-snr, 3n-icx weekly
- ticket: CSIT-1827
(M) all testbeds: mlrsearch fails to find NDR rate
- last update: before 2023-06-22
- work-to-fix: hard
- rca: On 3n-tsh, the symptom is TRex reporting ierrors, only in one direction. Other testbeds may have a different symptom, but failures there are less frequent.
- test: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
- frequency: low
- testbed: 3n-tsh, 3n-alt, 2n-clx
- example: [2]
- ticket: CSIT-1804
(M) all testbeds: AF_XDP mlrsearch fails to find NDR rate
- last update: before 2023-01-31
- work-to-fix: hard
- rca:
- test: af-xdp multicore tests
- frequency: low
- testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx
- example: 2n-skx, 2n-clx
- ticket: CSIT-1802
- note: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100
(L) all testbeds: vpp create avf interface failure in multi-core configs
- last update: 2023-02-06
- work-to-fix: hard
- rca: issue in Intel FVL driver
- test: multicore AVF
- frequency: low
- testbed: all testbeds
- example: 2n-clx, 3n-icx
- ticket: CSIT-1782
- note: A long standing issue without a final permanent fix.
(L) all testbeds: nat44det 4M and 16M scale 1 session not established
- last update: 2023-02-14
- work-to-fix: hard
- rca: unknown
- test: nat44det udp 4m and 16m (64k is ok, 1m can fail but rarely than bigger scales)
- frequency: low
- testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx
- example: 2n-zn2, 2n-clx
- ticket: CSIT-1795
(L) TRex may wrongly detect link bandwidth
- last update: 2024-02-07
- work-to-fix: hard
- rca: Quite rare failure affecting unpredictable tests. Perhaps a less severe symptom of CSIT-1936.
- test: No obvious pattern due to low frequency
- frequency: <0.4%
- testbed: recently seen on 3n-tsh and 3nb-spr
- example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2310-3nb-spr/28/log.html.gz#s1-s1-s1-s1-s4-t1-k2-k13-k14
- ticket: CSIT-1941
