CSIT/TestFailuresTracking

From fd.io
< CSIT
Revision as of 11:10, 5 December 2022 by Mackonstan (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

CSIT Test Failure Clasification

All known CSIT failures grouped and listed in the following order:

  • Always failing followed by sometimes failing.
  • Always failing tests:
    • Most common use cases followed by less common.
  • Sometimes failing tests:
    • Most frequently failing followed by less frequently failing.
    • Within each sub-group: most common use cases followed by less common.

CSIT Test Fixing Priorities

  • Test fixing work priorities defined as follows
    • (H)igh priority, most common use cases and most common test code.
    • (M)edium priority, specific HW and pervasive test code issue.
    • (L)ow priority, corner cases and external dependencies.

Always Failing Tests

In Trending

(H) 2n-clx, 2n-zn2: VPP RDMA tests no traffic forwarded

  • (H) 2n-clx, 2n-zn2: all RDMA tests failing with cli_inband clear runtime command
    • work-to-fix: easy
    • rca:
    • test: all RDMA with CX556A NIC
    • frequency: always
    • testbed: 2n-clx, 2n-zn2
    • example: 2n-clx, 2n-zn2, 2n-clx
    • ticket: CSIT-1882
    • note:

(M) 3n-snr: hwasync Wireguard failing to verify device

  • (M) 3n-snr: All hwasync wireguard tests failing when trying to verify device
    • work-to-fix: easy
    • rca: Failed to bind PCI device 0000:f4:00.0 to c4xxx on host 10.30.51.93
    • test: hwasync wireguard
    • frequency: always
    • testbed: 3n-snr
    • example: 3n-snr
    • ticket: CSIT-1883
    • note:

(M) 1n-aws: TRex mlrsearch fails to find NDR & PDR due to AWS rate limiting (5min total test duration)

  • (M) 1n-aws: TRex NDR PDR ALL IP4 scale and L2 scale tests failing with 50% packet loss
    • work-to-fix: hard
    • rca:
    • test: ip4scale2m
    • frequency: always
    • testbed: 1n-aws
    • example: 1n-aws
    • ticket: CSIT-1876
    • note: The root cause can be shared environment in aws cloud.

(M) 3n-alt, 3n-snr: testpmd no traffic forwarded

  • (M) 3n-alt, 3n-snr: testpmd tests fail with no traffic
    • work-to-fix: hard
    • rca:
    • test: testpmd
    • frequency: always
    • testbed: 3n-alt, 3n-snr
    • example: 3n-alt, 3n-snr, 3n-snr
    • ticket: CSIT-1848
    • note:

not in trending

(H) 3n-icx: vpp hoststack QUIC vppecho tests failing

  • (H) 3n-icx: QUIC vppecho BPS tests failing on timeout when checking hoststack finished
    • work-to-fix: easy
    • rca:
    • test: Quic vppecho BPS
    • frequency: always
    • testbed: 3n-skx, 3n-icx
    • example: 3n-icx
    • ticket: CSIT-1835
    • note:

(M) all testbeds: vpp 9000B tests with vhostuser, memif, tunnels, avf

(M) 2n-clx, 2n-icx: all Geneve tests with 1024 tunnels fail

(L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail

(L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions

Sometimes failing tests

in trending - high frequency failures

(H) 2n-icx: NFV density VPP does not start in container

(M) 2n-clx: e810 mlrsearch tests packets forwarding in one direction

(M) 3n-snr: 25GE links randomly going down between snr/sut and icx/tg-trex

(M) 3n-icx: wireguard 1k tunnels mlrsearch tests failing with 2c and 4c

(M) 3n-tsh: vpp in VM not starting

in trending - lower frequency failures

(M) 3n-icx, 3n-snr: 1518B IPsec packets not passing

(M) all testbeds: mlrsearch fails to find NDR rate

(M) all testbeds: AF_XDP mlrsearch fails to find NDR rate

(L) all testbeds: vpp create avf interface failure in multi-core configs

(L) 2n-dnv, 3n-dnv: x557 auto-negotiating 1ge instead of 10ge

(L) all testbeds: nat44det 4M and 16M scale 1 session not established

(L) 2n-dnv: nat44ed 1518B 64k sessions not establishing all sessions