Difference between revisions of "CSIT/Honeycomb/Summary"

Revision as of 14:09, 26 October 2016

Honeycomb testing Summary

For details on the Honeycomb project itself, check out its own page. This page assumes you are also familiar with the rest of CSIT.

Execution info:

• test suite is run separately from the rest of CSIT
• a test run is triggered by every successful run of "honeycomb-integration" Jenkins job (which includes unit testing)
• in addition to CSIT requirements, test runs require Honeycomb to be set up on the primary DUT as a Linux service

Topology and misc:

• tests are primarily CRUD, as such test cases only utilize a single DUT node with no traffic generator
• planned traffic tests will use standard two-node double-link topology with one DUT and TG
• unlike CSIT core tests, VPP is not restarted before/after every test but only after a test failure - due to time constraints
• all tests are tagged with "honeycomb_sanity" (subject to change)

Test structure:

• separate "honeycomb" folders exist for Honeycomb-specific python libraries, robot files, test data and command templates
• Restconf communication with Honeycomb uses python's Requests library
• Netconf communication with Honeycomb uses python's Paramiko library
• Honeycomb maintains two data stores:
  1. Operational data - read-only, built from VPP-API data dumps and should reflect state of VPP configuration
  2. Configuration data - desired configuration; pushed to VPP whenever a change is made, and on initial connection or reconnection to VPP
• CRUD test flow is generally:
  1. Check current status of Honeycomb operational data
  2. Check current status from VAT dump
  3. Make a change to configuration data
  4. Verify change in operational data
  5. Verify change in VAT dump
  6. [Teardown] Revert changes

Summary of Existing Tests

The Honeycomb suite contains tests for the following areas:

• (physical) Interface management:
  • state changes
  • IP addresses
  • IP neighbors
• Bridge domains:
  • CRUD, interface assignment
  • L2 FIB entries
• Vxlan, VxLan GPE tunnels
• TAP interfaces
• Vhost-user interfaces
• Vlan sub-interfaces
• ACL (access-control-lists)
• PBB (provider-backbone-bridge) sub-interface
• non-CRUD tests:
  • Persistence suite:
    • persist and/or restore configuration through a restart of Honeycomb, VPP or both
  • Netconf suite:
    • basic stress test - replicates a specific issue with executing many commands in quick succession
    • Notifications - listen for notifications that should be sent out on interface creation, deletion and state change
  • High-level ACLs (IETF-ACL):
    • utilizing an alternative ACL node/model, no VAT dump available (would require saving extra metadata on VPP)
    • Traffic tests separately for L2, L3, L4 layers, mixed layer cases and multiple rules

Planned Test Cases

• LISP
• NSH_SFC
• Netconf transaction revert
• security groups