CSIT/Honeycomb/Summary

Honeycomb testing Summary

For details on the Honeycomb project itself, check out its own page. This page assumes you are also familiar with the rest of CSIT.

Execution info:

• test suite is run separately from the rest of CSIT
• a test run is triggered by every successful run of Honeycomb build Jenkins job (which includes unit testing)
• in addition to CSIT requirements, test runs require Honeycomb to be set up on the primary DUT as a Linux service

Topology and misc:

• tests are primarily CRUD, as such test cases only utilize a single DUT node with no traffic generator
• planned traffic tests will use standard two-node double-link topology with one DUT and TG
• unlike CSIT core tests, VPP is not restarted before/after every test but only after a test failure - due to time constraints
• all tests are tagged with "honeycomb_sanity" (subject to change)

Test structure:

• separate "honeycomb" folders exist for Honeycomb-specific python libraries, robot files, test data and command templates
• Restconf communication with Honeycomb uses python's Requests library
• Netconf communication with Honeycomb uses python's Paramiko library
• Honeycomb maintains two data stores:
  1. Operational data - read-only, built from VPP-API data dumps and should reflect state of VPP configuration
  2. Configuration data - desired configuration; pushed to VPP whenever a change is made, and on initial connection or reconnection to VPP
• CRUD test flow is generally:
  1. Check current status of Honeycomb operational data
  2. Check current status from VAT dump
  3. Make a change to configuration data
  4. Verify change in operational data
  5. Verify change in VAT dump
  6. [Teardown] Revert changes

Summary of Existing Tests

The Honeycomb suite contains tests for the following areas:

• (physical) Interface management:
  • state changes
  • IP addresses
  • IP neighbors
• Bridge domains:
  • CRUD, interface assignment
  • L2 FIB entries
• Vxlan, VxLan GPE tunnels
• TAP interfaces
• Vhost-user interfaces
• Vlan sub-interfaces
• ACL (access-control-lists)
• PBB (provider-backbone-bridge) sub-interface
• non-CRUD tests:
  • Persistence suite:
    • persist and/or restore configuration through a restart of Honeycomb, VPP or both
  • Netconf suite:
    • basic stress test - replicates a specific issue with executing many commands in quick succession
    • Notifications - listen for notifications that should be sent out on interface creation, deletion and state change

Planned Test Cases

• High-level ACLs (IETF-ACL):
  • utilizing an alternative ACL node/model, should simplify the process of setting up complex classify chains
  • may require traffic tests instead of CRUD, supposedly a comprehensive VAT dump is "not possible"
• LISP
• NSH_SFC
• Netconf transaction revert