CSIT/TestFailuresTracking

From fd.io
< CSIT
Revision as of 13:26, 14 August 2024 by Vrpolak (Talk | contribs)

Jump to: navigation, search

Contents

CSIT Test Failure Clasification

All known CSIT failures grouped and listed in the following order:

  • Always failing followed by sometimes failing.
  • Always failing tests:
    • Most common use cases followed by less common.
  • Sometimes failing tests:
    • Most frequently failing followed by less frequently failing.
      • High frequency 50%-100%
      • medium frequency 10%-50%
      • low frequency 0%-10%.
    • Within each sub-group: most common use cases followed by less common.

CSIT Test Fixing Priorities

Test fixing work priorities defined as follows:

  • (H)igh priority, most common use cases and most common test code.
  • (M)edium priority, specific HW and pervasive test code issue.
  • (L)ow priority, corner cases and external dependencies.

Current Failures

Deterministic Failures

In Trending

(H) 3n spr: Unusable performance of ipsec tests with SHA_256_128

  • last update: 2024-08-14
  • work-to-fix: low (if you are Damjan)
  • rca: It seems compiler emits wrong instructions, VPP build system needs to be fixed.
  • test: Three new ipsec tests that use SHA_256_128 as integrity algorithm.
  • frequency: 50%
  • testbed: 3na-spr, 3nb-spr
  • example: [1]
  • ticket: VPP-2118
  • note: Other Xeon testbeds are also affected, but performance is not as bad to fail NDR. ARM is not affected at all.

(H) 3nb-spr hoststack: interface not up after first test

(H) Zero traffic reported in udpquic tests

(H) 3na-spr, 2n-zn2: VPP fails to start in first test cases if dpdk/mlx5

(M) DPDK 23.03 testpmd startup fails on some testbeds

(M) 2n-spr: zero traffic on cx7 rdma

(M) 3n-icx 3nb-spr: Failed to enable GTPU offload RX

  • last update: 2024-08-14
  • work-to-fix: low
  • rca: Retval is -1. More examination is needed to understand why.
  • test: any gtpuhw
  • frequency: 100%
  • testbed: all
  • example: [2]
  • ticket: CSIT-1950

(M) Lossy trials in nat udp mlx5 tests

(L) 3n-icx: negative ipackets on TB38 AVF 4c l2patch

(L) 2n-tx2: af_xdp mrr failures

Not In Trending

(M) Combination of AVF and vhost drops all 9000B packets

(M) 9000B tests with encap overhead and non-dpdk plugins see fragmented packets

  • last update: 2024-08-14
  • work-to-fix: medium
  • rca: Some internal MTU is at 9000 (not 9200). More examination needed to see if the issue is in VPP or CSIT.
  • test: 9000B testcases for loadbalance, geneva, vxlan or SRv6
  • frequency: 100%
  • testbed: all
  • example: [3]
  • ticket: CSIT-1950
  • note: Some tests drop packets (affects all tests), some tests fragment packets (does not fail MRR).

(M) tests with 9000B payload frames not forwarded over memif interfaces

  • last update: 2023-02-09
  • work-to-fix: hard
  • test: 9000B + memif
  • testbed: 2n-skx, 3n-skx, 2n-clx
  • examples: 2n-skx Memif
  • ticket: CSIT-1808 VPP-2091

(M) IMIX 4c tests may fail PDR due to ~10% loss

(L) 3n-snr: Increased heap size in ipsec policy tests prevents VPP from starting

  • last update: 2024-08-14
  • work-to-fix: low
  • rca: Probably not enough huge pages.
  • test: ipsec policy large scale
  • frequency: 100%
  • testbed: 3n-snr
  • example:
  • ticket: CSIT-1966
  • note: Not even in coverage. Found when investigating VPP-2121.

(L) Some tests have too long ramp-up trials

  • last update: 2024-08-14
  • work-to-fix: low
  • rca: Ramp-up trial should have high enough rate to fit into session timeout.
  • test: NAT ASTF TPUT at large scale
  • frequency: 100%
  • testbed: all
  • example: [4]
  • ticket: CSIT-1961
  • note: We removed largest scale from most jobs, only few stragglers remain on coverage.

(L) Memif crashes VPP in container with jumbo frames

  • last update: 2024-08-14
  • work-to-fix: hard (bug in VPP, no easy access to logs)
  • test: any memif with 9000B
  • testbed: all
  • examples: [5]
  • ticket: VPP-2091

(L) l3fwd error in 200Ge2P1Cx7Veat-Mlx5 test with 9000B

  • last update: 2023-06-28
  • work-to-fix: medium
  • test: 9000B + Cx7 with DPDK DUT
  • testbed: 2n-icx
  • examples: [6]
  • ticket: CSIT-1924

(L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail

(L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions

Occasional Failures

In Trending

(H) 3n-icxd: Various symptoms pointing to hardware (cable/nic/driver) issues

  • last update: 2024-08-14
  • work-to-fix: medium
  • rca: Unknown hardware instability. Not sure if on SUT or TG side.
  • test: any on 3n-icxd
  • frequency: medium
  • testbed: 3n-icxd
  • example: [7]
  • ticket: CSIT-1963
  • note: My be multiple issues, e.g. one hardware (loose cable) and one software (driver like in CSIT-1936).

(H) 2n-icx: NFV density VPP does not start in container

  • last update: before 2023-01-31
  • work-to-fix: hard
  • rca:
  • test: all subsequent
  • frequency: medium
  • testbed: 2n-icx
  • example: 2n-icx mrr, 2n-icx ndrpdr
  • ticket: CSIT-1881
  • note: Once VPP breaks, all subsequent tests fail. Even all subsequent builds will be failing until Peter makes TB working again. Although it's failing with medium frequency when it happens it breaks all subsequent builds on the TB therefore [H] priority.

(M) 3n-icx hoststack: Udpquicscale tests sometimes fail with various symptoms

  • last update: 2024-08-14
  • work-to-fix: hard
  • rca: Cause unclear a symptoms vary. Maybe memory corruption?
  • test: Udpquicscale
  • frequency: medium
  • testbed: 3n-icx
  • example: [8]
  • ticket: CSIT-1962
  • note: Not visible in trending as udpscale already suffers from CSIT-1935.

(M) 3n-alt: high scale ipsec policy tests may crash VPP

(M) 3nb-spr: Wireguardhw tests are likely to crash

  • last update: before 2024-08-14
  • work-to-fix: hard
  • rca: More investigation needed.
  • test: any wireguardhw
  • frequency: low
  • testbed: 3nb-spr
  • examples: [9]
  • ticket: CSIT-1964
  • note: Symptoms are similar to CSIT-1938 or CSIT-1886, but both of those are frequent in high scale, this is not.

(M) 3n-icx, 3n-snr: wireguard 100 and 1000 tunnels mlrsearch tests failing with 2c and 4c

  • last update: before 2023-01-31
  • work-to-fix: easy
  • rca:
  • test: wireguard 100 tunnels and more
  • frequency: high
  • testbed: 3n-icx, 3n-snr
  • examples: 3n-icx
  • ticket: CSIT-1886

(M) e810cq sometimes reporting link down

Not In Trending

(H) sw_interface_add_del_address: avf process node failed to reply in 5 seconds

  • last update: 2024-08-14
  • work-to-fix: medium
  • rca: AVF processes API commands asynchronously, not prepared to receive multiple messages at once.
  • test: ipsec policy, but any other test using async PAPI processing may be affected.
  • frequency: high on 3n-snr, less frequent on Xeon testbeds
  • testbed: all but with different frwequencies
  • example: [10]
  • ticket: VPP-2121
  • note: Only coverage jobs are running ipsec policy with AFT driver.

Rare Failures

In Trending

(M) 3n-icx, 3n-snr: 1518B IPsec packets not passing

(M) all testbeds: mlrsearch fails to find NDR rate

  • last update: before 2023-06-22
  • work-to-fix: hard
  • rca: On 3n-tsh, the symptom is TRex reporting ierrors, only in one direction. Other testbeds may have a different symptom, but failures there are less frequent.
  • test: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
  • frequency: low
  • testbed: 3n-tsh, 3n-alt, 2n-clx
  • example: [11]
  • ticket: CSIT-1804

(M) all testbeds: AF_XDP mlrsearch fails to find NDR rate

  • last update: before 2023-01-31
  • work-to-fix: hard
  • rca:
  • test: af-xdp multicore tests
  • frequency: low
  • testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx
  • example: 2n-skx, 2n-clx
  • ticket: CSIT-1802
  • note: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100

(L) 2n-zn2: Geneve sometimes loses one direction of traffic

  • last update: 2024-08-14
  • work-to-fix: hard
  • rca: More investigation needed to see the mechanism.
  • test: any geneve
  • frequency: <1%
  • testbed: 2n-zn2
  • example: [12]
  • ticket: CSIT-1960
  • note: Very rare, recently seen only as a failure in report iterative (not weekly trending) and in one run as MRR regression.

(L) Rare VPP crash in nat avf tests

  • last update: 2024-08-14
  • work-to-fix: medium
  • rca: clib_dlist_remove called by nat44_session_update_lru, but probably just a memory corruption symptom
  • test: any NAT test with AVF
  • frequency: low
  • testbed: any
  • example: [13]
  • ticket: CSIT-1947
  • note: More frequent in soak tests.

(L) ipsec hwasync fails with large scale and multiple queues

  • last update: 2024-08-14
  • work-to-fix: hard
  • test: ipsec hwasync
  • frequency: low
  • testbed: all with QAT
  • example: [14]
  • ticket: CSIT-1946
  • note: Frequency decreased when CSIT changes RXQ ratio in 40824.

(L) all testbeds: vpp create avf interface failure in multi-core configs

  • last update: 2023-02-06
  • work-to-fix: hard
  • rca: issue in Intel FVL driver
  • test: multicore AVF
  • frequency: low
  • testbed: all testbeds
  • example: 2n-clx, 3n-icx
  • ticket: CSIT-1782
  • note: A long standing issue without a final permanent fix.

(L) all testbeds: nat44det 4M and 16M scale 1 session not established

  • last update: 2023-02-14
  • work-to-fix: hard
  • rca: unknown
  • test: nat44det udp 4m and 16m (64k is ok, 1m can fail but rarely than bigger scales)
  • frequency: low
  • testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx
  • example: 2n-zn2, 2n-clx
  • ticket: CSIT-1795

(L) TRex may wrongly detect link bandwidth

Not In Trending

(L) Occasional failure on 1518B CX5: Trex failed to send message

  • last update: 2024-08-14
  • work-to-fix: hard
  • rca: More investigation needed.
  • test: Any 1518B on CX5, any driver.
  • frequency: low
  • testbed: 2n-clx
  • example: [15]
  • ticket: CSIT-1965
  • note: Only in coverage and 2n-clx has been decommissioned, but may appear if we move CX5 NICs to another testbed.

Past Failures

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/TestFailuresTracking&oldid=12691"