Revision as of 11:22, 12 April 2017

DRAFT PROPOSAL: CSIT performance jobs re-organisation

Background

Currently csit performance jobs include too many test cases, take long time to execute and generate large logs that are hard to browse and analyze. To address these points, following is proposed:

• modularize csit performance jenkins jobs in terms of amount of tests per job and job execution time.
• the new performance job arrangement should suit main csit perf test consumers within FD.io, specifically:
  • short perf tests per vpp gerrit patch - manually triggered to check against reference NDR rate.
  • daily rolling execution of all perf tests for NDR discovery for subset of VPP setup variations.
  • regular rolling execution of all perf tests for all CSIT VPP setup variations.

Following discussion on csit-dev mailing list and csit project call here is the proposed CSIT performance test job structure for listed FD.io perf-test consumers:

• vpp-dev patch performance verify
• csit-dev trending daily
• csit-dev trending semi-weekly
• csit-dev patch perf verify
• csit-dev perf code base checks

vpp-dev patch performance verify

1. objective:
   • VPP performance verify - per vpp gerrit patch
2. job:
   • name: vpp-csit-perf-verify
   • duration: < 10 min
     • this is a MUST requirement
   • number of TCs: 3..6
   • trigger: keyword in comment to vpp patch
3. perf test-cases selection:
   • test-cases selected using trigger keywords paired with tags
   • all test-cases are labelled with CSIT RF topic tags
     • defined in /csit/docs/tag_documentation.rst
4. proposed keywords:
   • focus on testing specified vpp packet processing path
   • format: (keyword, [tag_list])
   • (perf_verify_ip4, [IP4Base, IP4Scale2M])
   • (perf_verify_ip6, [IP6Base, IP6Scale2M])
   • (perf_verify_l2, [L2XC, L2BD])
   • (perf_verify_lisp, [4Lisp, 6Lisp])
   • (perf_verify_vxlan, [Vxlan])
   • (perf_verify_vhost, [Vhost])
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • X520: x520 Niantic NIC
   • NDRCHK: check against a reference NDR throughput
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse: no

csit-dev trending daily

1. objective:
   • daily perf trending of vpp master build
2. job:
   • name: csit-vpp-perf-trend-daily
   • duration: ~9 hrs (2*4.5 hrs)
   • number of TCs: 176
   • trigger: scheduled daily
3. perf test-cases selection:
   • all CSIT perf suites
4. default tags for vpp startup config and perf test type
   • 1T1C: one worker thread, one core
   • 2T2C: two worker threads, two cores
   • X520: x520 Niantic NIC
   • NDRDISC: discovery of NDR throughput
5. results:
   • jjb verify: no
   • archive:
     • full results LF jenkins only
     • NDR rates - archived separately in [TBD]
   • present:
     • jenkins trending graphs
     • other options that are prettier - docs.fd.io [TBD]
     • csit rls reports [?] useful as a large data sample [!]
   • analyse:
     • csit project
       • catch degradations
       • catch improvements
     • readouts to vpp project [?]
6. notes:
   • ...

csit-dev trending semi-weekly

1. objective:
   • semi-weekly perf trending of vpp master build
2. job
   • name: csit-vpp-perf-trend-semiweekly
   • duration: ~ 12hrs (2* 6 hrs)
   • number of TCs: 58
   • trigger: scheduled semi-weekly
3. perf test-cases selection:
   • all CSIT perf suites
4. default tags for vpp startup config and perf test type
   • • 1T1C: one worker thread, one core
     • 2T2C: two worker threads, two cores
     • X710: x710 Fortville NIC
     • XL710: xl710 Fortville NIC
     • NDRDISC: discovery of NDR throughput
5. results:
   • jjb verify: no
   • archive:
     • full results LF jenkins only
     • NDR rates - archived separately in [TBD]
   • present:
     • jenkins trending graphs
     • other options that are prettier - docs.fd.io [TBD]
     • csit rls reports [?] useful as a large data sample [!]
   • analyse:
     • csit project
       • catch degradations
       • catch improvements
     • readouts to vpp project [?]
6. notes:
   • ...

csit-dev patch perf verify

1. objective:
   • perf verify - per csit gerrit patch
2. job:
   • name: csit-vpp-perf-verify
   • duration: < 60 min
   • number of TCs: 6..12
   • trigger: keyword in comment to csit patch
3. perf test-cases selection:
   • test-cases selected using trigger keywords paired with tags
   • all test-cases are labelled with CSIT RF topic tags
     • defined in /csit/docs/tag_documentation.rst
4. proposed keywords:
   • focus on testing specified vpp packet processing path
   • format: (keyword, [tag_list])
   • (perf_verify_ip4, [IP4Base, IP4Scale2M])
   • (perf_verify_ip6, [IP6Base, IP6Scale2M])
   • (perf_verify_l2, [L2XC, L2BD])
   • (perf_verify_lisp, [4Lisp, 6Lisp])
   • (perf_verify_vxlan, [Vxlan])
   • (perf_verify_vhost, [Vhost])
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • X520: x520 Niantic NIC
   • NDRDISC: discovery of NDR throughput
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse: no

csit-dev perf code base checks

1. objective: check csit perf code base
2. job:
   • name: csit-vpp-perf-check [TBD]
   • duration: [TBD]
   • number of TCs: [TBD]
   • trigger: [TBD]
3. perf test-cases selection:
4. proposed keywords:
   • focus on testing specified vpp packet processing path
   • format: (keyword, [tag_list])
   • (perf_verify_ndrdisc, [NDRDISC])
     • 322 TCs 16.5hr
   • (perf_verify_pdrdisc, [PDRDISC])
     • 322 TCs 16.5hr
   • (perf_verify_ndrchk, [NDRCHK])
     • 198 TCs 4.25 hrs
   • (perf_verify_pdrchk, [PDRCHK])
     • 198 TCs 4.25 hrs
5. default tags for vpp startup config and perf test type:
   • 1T1C: one worker thread, one core
   • 2T2C: two worker thread, two core
   • 4T4C: four worker thread, four core
6. results:
   • jjb verify: PASS/FAIL
   • archive: LF jenkins only
   • present: no
   • analyse:
     • csit project
       • catch malfunctions
       • catch degradataions