CSIT/csit-perf-jobs-reorg-proposal

DRAFT PROPOSAL: CSIT performance jobs re-organisation

Background

Currently csit performance jobs include too many test cases, take long time to execute and generate large logs that are hard to browse and analyze. To address these points, following is proposed:

• modularize csit performance jenkins jobs in terms of amount of tests per job and job execution time.
• the new performance job arrangement should suit main csit perf test consumers within FD.io, specifically:
  • short perf tests per vpp gerrit patch - manually triggered to check against reference NDR rate.
  • daily rolling execution of all perf tests for NDR discovery for subset of VPP setup variations.
  • regular rolling execution of all perf tests for all CSIT VPP setup variations.

Following discussion on csit-dev mailing list and csit project call here is the proposed CSIT performance test job structure for listed FD.io perf-test consumers:

• vpp-dev patch performance verify
• csit-dev trending daily
• csit-dev trending semi-weekly
• csit-dev patch perf verify
• csit-dev perf code base checks

vpp-dev patch performance verify

1. objective:
   • VPP performance verify - per vpp gerrit patch
2. job:
   • name: vpp-csit-perf-verify
     • https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-hw-perf-master/
     • https://git.fd.io/ci-management/tree/jjb/vpp/vpp.yaml, line 336: name: 'vpp-csit-verify-hw-perf-{stream}'
   • duration: < 10 min
     • this is a MUST requirement
   • number of TCs: 3..6
   • trigger: keyword in comment to vpp patch
     • comment-contains-value: 'vpp-verify-perf-l2'
     • comment-contains-value: 'vpp-verify-perf-ip4'
     • comment-contains-value: 'vpp-verify-perf-ip6'
     • comment-contains-value: 'vpp-verify-perf-lisp'
     • comment-contains-value: 'vpp-verify-perf-vxlan'
     • comment-contains-value: 'vpp-verify-perf-vhost'
     • https://git.fd.io/csit/tree/bootstrap-verify-perf.sh
       • e.g. for ip4 line 237: VPP-VERIFY-PERF-IP4 )
       • --include ndrchkANDnic_intel-x520-da2AND1t1cANDip4baseORndrchkANDnic_intel-x520-da2AND1t1cANDip4fwdANDfib_2m tests/
   • production status: in repo and not tested
3. perf test-cases selection:
   • test-cases selected using trigger keywords paired with tags
   • all test-cases are labelled with CSIT RF topic tags
     • defined in /csit/docs/tag_documentation.rst
4. proposed keywords:
   • focus on testing specified vpp packet processing path
   • format: (keyword, [tag_list])
   • (perf_verify_ip4, [IP4Base, IP4Scale2M])
   • (perf_verify_ip6, [IP6Base, IP6Scale2M])
   • (perf_verify_l2, [L2XC, L2BD])
   • (perf_verify_lisp, [4Lisp, 6Lisp])
   • (perf_verify_vxlan, [Vxlan])
   • (perf_verify_vhost, [Vhost])
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • X520: x520 Niantic NIC
   • NDRCHK: check against a reference NDR throughput
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse: no

csit-dev trending daily

1. objective:
   • daily perf trending of vpp master build
2. job:
   • name: csit-vpp-perf-trend-daily
     • https://jenkins.fd.io/view/csit/job/csit-vpp-perf-trend-daily-master/
     • https://git.fd.io/ci-management/tree/jjb/csit/csit.yaml, line 2646: name: 'csit-vpp-perf-trend-daily-master'
   • duration: ~11 hrs (2*5.5 hrs)
   • number of TCs: 188
   • trigger: scheduled daily, between 20:00 and 23:00 UTC
   • production status: in repo and tested
3. perf test-cases selection:
   • all CSIT perf suites
4. default tags for vpp startup config and perf test type
   • 1T1C: one worker thread, one core
   • 2T2C: two worker threads, two cores
   • X520: x520 Niantic NIC
   • NDRDISC: discovery of NDR throughput
5. results:
   • jjb verify: no
   • archive:
     • full results LF jenkins only
     • NDR rates - archived separately in [TBD]
   • present:
     • jenkins trending graphs
     • other options that are prettier - docs.fd.io [TBD]
     • csit rls reports [?] useful as a large data sample [!]
   • analyse:
     • csit project
       • catch degradations
       • catch improvements
     • readouts to vpp project [?]
6. notes:
   • ...

csit-dev trending semi-weekly

1. objective:
   • semi-weekly perf trending of vpp master build
2. job
   • name: csit-vpp-perf-trend-semiweekly
     • https://jenkins.fd.io/view/csit/job/csit-vpp-perf-trend-semi-weekly-master/
   • duration: ~4:45 hrs
   • number of TCs: 40
   • trigger: scheduled semi-weekly - Tuesday and Saturday - between 20:00 and 23:00 UTC
     • https://git.fd.io/ci-management/tree/jjb/csit/csit.yaml, line 3032: name: 'csit-vpp-perf-trend-semi-weekly-master'
   • production status: in repo and tested
3. perf test-cases selection:
   • all CSIT perf suites
4. default tags for vpp startup config and perf test type
   • • 1T1C: one worker thread, one core
     • 2T2C: two worker threads, two cores
     • X710: x710 Fortville NIC
     • XL710: xl710 Fortville NIC
     • NDRDISC: discovery of NDR throughput
5. results:
   • jjb verify: no
   • archive:
     • full results LF jenkins only
     • NDR rates - archived separately in [TBD]
   • present:
     • jenkins trending graphs
     • other options that are prettier - docs.fd.io [TBD]
     • csit rls reports [?] useful as a large data sample [!]
   • analyse:
     • csit project
       • catch degradations
       • catch improvements
     • readouts to vpp project [?]
6. notes:
   • ...

csit-dev patch perf verify

1. objective:
   • perf verify - per csit gerrit patch
2. job:
   • name: csit-vpp-perf-verify
     • https://jenkins.fd.io/view/csit/job/csit-vpp-perf-verify-master/
     • https://git.fd.io/ci-management/tree/jjb/csit/csit.yaml, line 3585 "name: 'csit-vpp-perf-verify-{stream}'"
   • duration: < 60 min
   • number of TCs: 6..12
   • trigger: keyword in comment to csit patch
     • comment-contains-value: 'verify-perf-l2'
     • comment-contains-value: 'verify-perf-ip4'
     • comment-contains-value: 'verify-perf-ip6'
     • comment-contains-value: 'verify-perf-lisp'
     • comment-contains-value: 'verify-perf-vxlan'
     • comment-contains-value: 'verify-perf-vhost'
   • production status: in repo and not tested
3. perf test-cases selection:
   • test-cases selected using trigger keywords paired with tags
   • all test-cases are labelled with CSIT RF topic tags
     • defined in /csit/docs/tag_documentation.rst
4. proposed keywords:
   • focus on testing specified vpp packet processing path
   • format: (keyword, [tag_list])
   • (perf_verify_ip4, [IP4Base, IP4Scale2M])
   • (perf_verify_ip6, [IP6Base, IP6Scale2M])
   • (perf_verify_l2, [L2XC, L2BD])
   • (perf_verify_lisp, [4Lisp, 6Lisp])
   • (perf_verify_vxlan, [Vxlan])
   • (perf_verify_vhost, [Vhost])
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • X520: x520 Niantic NIC
   • NDRDISC: discovery of NDR throughput
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse: no

csit-dev perf code base checks

1. objective: check csit perf code base
2. job:
   • name: csit-vpp-perf-check [TBD]
     • https://jenkins.fd.io/view/csit/job/csit-vpp-perf-check-master/
     • defined in: https://git.fd.io/ci-management/tree/jjb/csit/csit.yaml, line 3485
   • duration: [TBD]
   • number of TCs: [TBD]
   • trigger: by comment
     • "comment-contains-value: 'verify-perf-ndrdisc'"
     • "comment-contains-value: 'verify-perf-pdrdisc'"
     • "comment-contains-value: 'verify-perf-ndrchk'"
     • https://git.fd.io/csit/tree/bootstrap-verify-perf.sh
       • e.g. ndrdisc: line 157
       • VERIFY-PERF-NDRDISC )
       • --include ndrdiscAND1t1cORndrdiscAND2t2c \
   • production status: in repo and not tested
3. perf test-cases selection:
   • all CSIT perf suites
4. proposed keywords:
   • format: (keyword, [tag_list])
   • (perf_verify_ndrdisc, [NDRDISC])
     • 215 TCs 11 hr
   • (perf_verify_pdrdisc, [PDRDISC])
     • 215 TCs 11 hr
   • (perf_verify_ndrchk, [NDRCHK])
     • 132 TCs 3 hrs
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • 2T2C: two worker thread, two core
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse:
     • csit project
       • catch malfunctions
       • catch degradataions

csit-dev perf ndr chk daily

1. objective: check vpp perf ndr chk daily
2. job:
   • name: csit-vpp-perf-ndrchk-daily
     • https://jenkins.fd.io/view/csit/job/csit-vpp-perf-ndrchk-daily-master/
   • duration: ~3.5 hrs
   • number of TCs: 132
   • trigger: daily
   • production status: in repo and tested
3. perf test-cases selection:
   • all CSIT perf suites
4. proposed keywords: (not used as triggered daily)
   • format: (keyword, [tag_list])
   • (perf_verify_ndrchk, [NDRCHK])
     • 132 TCs 3 hrs
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • 2T2C: two worker thread, two core
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse:
     • csit project
       • catch malfunctions
       • catch degradataions