Difference between revisions of "CSIT/TestFailuresTracking"

From fd.io
Jump to: navigation, search
(→‎In Trending: CSIT-1928 also confirmed on 3n-snr.)
(→‎Past Failures: Added CSIT-1926 as already fixed.)
Line 296: Line 296:
 
* ticket: [https://jira.fd.io/browse/CSIT-1800 CSIT-1800]
 
* ticket: [https://jira.fd.io/browse/CSIT-1800 CSIT-1800]
 
* fixed by: https://gerrit.fd.io/r/c/vpp/+/39092
 
* fixed by: https://gerrit.fd.io/r/c/vpp/+/39092
 +
 +
==== (L) compute resource values are not constant over DUTs ====
 +
 +
* last update: 2023-08-02
 +
* work-to-fix: easy
 +
* rca: Old CSIT code did not envision two VPPs on different NUMAs.
 +
* test: ipsec swasync scheduler tests
 +
* frequency: always
 +
* testbed: 3na-spr, 3nb-spr
 +
* example: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-master-3na-spr/3/log.html.gz#s1-s1-s1-s1-s1-t3-k2-k11-k1
 +
* ticket: [https://jira.fd.io/browse/CSIT-1926 CSIT-1926]
 +
* fixed by: https://gerrit.fd.io/r/c/csit/+/39267

Revision as of 13:07, 2 August 2023

Contents

CSIT Test Failure Clasification

All known CSIT failures grouped and listed in the following order:

  • Always failing followed by sometimes failing.
  • Always failing tests:
    • Most common use cases followed by less common.
  • Sometimes failing tests:
    • Most frequently failing followed by less frequently failing.
      • High frequency 50%-100%
      • medium frequency 10%-50%
      • low frequency 0%-10%.
    • Within each sub-group: most common use cases followed by less common.

CSIT Test Fixing Priorities

Test fixing work priorities defined as follows:

  • (H)igh priority, most common use cases and most common test code.
  • (M)edium priority, specific HW and pervasive test code issue.
  • (L)ow priority, corner cases and external dependencies.

Current Failures

Deterministic Failures

In Trending

(H) ipsec swasync: VPP can become unresponsive

(M) DPDK 23.03 testpmd startup fails on some testbeds

(M) 2n-spr: zero traffic on cx7 rdma

(M) 3n-icx, 3n-snr: first few swasync scheduler tests timing out in runtime stat

(M) 3n-icx cx6dx hoststack: File descriptor in bad state

(L) 3n-icx: negative ipackets on TB38 AVF 4c l2patch

(L) 2n-tx2: af_xdp mrr failures

Not In Trending

(M) all testbeds: some 9000B tests

  • last update: 2023-02-09
  • work-to-fix: hard
  • rca: VPP code: 34839: dpdk: cleanup MTU handling. CSIT needs to rework how it sets MTU / max frame rate (CSIT-1797). Some tests will continue failing due to missing support on VPP side, we will open specific Jira tickets for those.
  • test: see sub-items
  • frequency: always
  • testbed: all
  • examples: see sub-items
  • ticket: CSIT-1809
  • gerrit: https://gerrit.fd.io/r/c/csit/+/37824
(M) tests with 9000B payload frames not forwarded over vhost interfaces
  • last update: 2023-02-09
  • work-to-fix: hard
  • test: 9000B + vhostuser
  • testbed: 2n-skx, 3n-skx, 2n-clx
  • examples: 3n-skx vhostuser
  • ticket: CSIT-1809
(M) tests with 9000B payload frames not forwarded over memif interfaces
  • last update: 2023-02-09
  • work-to-fix: hard
  • test: 9000B + memif
  • testbed: 2n-skx, 3n-skx, 2n-clx
  • examples: 2n-skx Memif
  • ticket: CSIT-1808
(M) 9000B payload frames not forwarded over tunnels due to violating supported Max Frame Size (VxLAN, LISP, SRv6)
  • last update: 2023-02-09
  • work-to-fix: medium
  • test: 9000B + (IP4 tunnels VXLAN, IP4 tunnels LISP, Srv6, IpSec)
  • testbed: 2n-icx, 3n-icx
  • examples: 2n-icx VXLAN, 3n-icx
  • ticket: CSIT-1801
(M) 9000B all AVF tests are failing to forward traffic
  • last update: 2023-02-09
  • work-to-fix: hard
  • test: 9000B + AVF
  • testbed: 3n-icx
  • examples: 3n-icx ip4base
  • ticket: CSIT-1885
(L) l3fwd error in 200Ge2P1Cx7Veat-Mlx5 test with 9000B
  • last update: 2023-06-28
  • work-to-fix: medium
  • test: 9000B + Cx7 with DPDK DUT
  • testbed: 2n-icx
  • examples: [1]
  • ticket: CSIT-1924

(L) 2n-clx, 2n-icx: nat44ed cps 16M sessions scale fail

(L) 2n-clx, 2n-icx: nat44det imix 1M sessions fails to create sessions

Occasional Failures

In Trending

(H) 2n-icx: NFV density VPP does not start in container

  • last update: before 2023-01-31
  • work-to-fix: hard
  • rca:
  • test: all subsequent
  • frequency: medium
  • testbed: 2n-icx
  • example: 2n-icx mrr, 2n-icx ndrpdr
  • ticket: CSIT-1881
  • note: Once VPP breaks, all subsequent tests fail. Even all subsequent builds will be failing until Peter makes TB working again. Although it's failing with medium frequency when it happens it breaks all subsequent builds on the TB therefore [H] priority.

(M) 2n-clx: e810 mlrsearch tests packets forwarding in one direction

(M) 3n-icx, 3n-snr: wireguard 100 and 1000 tunnels mlrsearch tests failing with 2c and 4c

  • last update: before 2023-01-31
  • work-to-fix: easy
  • rca:
  • test: wireguard 100 tunnels and more
  • frequency: high
  • testbed: 3n-icx, 3n-snr
  • examples: 3n-icx
  • ticket: CSIT-1886

Rare Failures

In Trending

(M) 3n-icx, 3n-snr: 1518B IPsec packets not passing

(M) all testbeds: mlrsearch fails to find NDR rate

  • last update: before 2023-06-22
  • work-to-fix: hard
  • rca: On 3n-tsh, the symptom is TRex reporting ierrors, only in one direction. Other testbeds may have a different symptom, but failures there are less frequent.
  • test: Crypto, Ip4, L2, Srv6, Vm Vhost (all packet sizes, all core configurations affected)
  • frequency: low
  • testbed: 3n-tsh, 3n-alt, 2n-clx
  • example: [2]
  • ticket: CSIT-1804

(M) all testbeds: AF_XDP mlrsearch fails to find NDR rate

  • last update: before 2023-01-31
  • work-to-fix: hard
  • rca:
  • test: af-xdp multicore tests
  • frequency: low
  • testbed: 2n-clx, 2n-skx, 2n-tx2, 2n-icx
  • example: 2n-skx, 2n-clx
  • ticket: CSIT-1802
  • note: This is mainly observed in iterative and coverage. It's very low frequency ~ 1 out of 100

(L) all testbeds: vpp create avf interface failure in multi-core configs

  • last update: 2023-02-06
  • work-to-fix: hard
  • rca: issue in Intel FVL driver
  • test: multicore AVF
  • frequency: low
  • testbed: all testbeds
  • example: 2n-clx, 3n-icx
  • ticket: CSIT-1782
  • note: A long standing issue without a final permanent fix.

(L) all testbeds: nat44det 4M and 16M scale 1 session not established

  • last update: 2023-02-14
  • work-to-fix: hard
  • rca: unknown
  • test: nat44det udp 4m and 16m (64k is ok, 1m can fail but rarely than bigger scales)
  • frequency: low
  • testbed: 2n-zn2, 2n-skx, 2n-icx, 2n-clx
  • example: 2n-zn2, 2n-clx
  • ticket: CSIT-1795

Past Failures

(M) 2n-clx, 2n-icx: all Geneve tests with 1024 tunnels fail

(L) compute resource values are not constant over DUTs

Retrieved from "https://wiki.fd.io/index.php?title=CSIT/TestFailuresTracking&oldid=12349"