|
|
| Line 1: |
Line 1: |
| − | === Introduction === | + | === Documentation Moved === |
| | | | |
| − | Although every situation is different, this page describes data which will help make efficient use of everyone's time when dealing with vpp bugs.
| + | Please see [[https://fd.io/docs/vpp/master/troubleshooting/reportingissues/reportingissues.html How To Report an Issue]] |
| − | | + | |
| − | Before you press the Jira button to create a bug report - or email vpp-dev@lists.fd.io - please ask yourself whether there's enough information for someone else to understand and possibly to reproduce the issue given a reasonable amount of effort. Unicast emails to maintainers, committers, and the project PTL are strongly discouraged.
| + | |
| − | | + | |
| − | A good strategy for clear-cut bugs: file a detailed Jira ticket, and then send a short description of the issue to vpp-dev@lists.fd.io, perhaps from the Jira ticket description. It's fine to send email to vpp-dev@lists.fd.io to ask a few questions before filing Jira tickets.
| + | |
| − | | + | |
| − | === Data to include in bug reports ===
| + | |
| − | | + | |
| − | ==== Image version and operating environment ====
| + | |
| − | | + | |
| − | Please make sure to include the vpp image version: | + | |
| − | | + | |
| − | sudo vppctl show version verbose
| + | |
| − | vpp v1.0.0-188~geef4d99 built by vagrant on localhost at Wed Feb 24 08:52:13 PST 2016
| + | |
| − | Built in /home/vagrant/git/vpp
| + | |
| − | Compiled with GCC 4.8.4
| + | |
| − | DPDK version is RTE 2.2.0
| + | |
| − | DPDK EAL init arguments: -c 1 -n 4 --socket-mem 1024 --huge-dir /run/vpp/hugepages --file-prefix vpp
| + | |
| − | -b 0000:02:00.0 -b 0000:02:01.0 --master-lcore 0
| + | |
| − |
| + | |
| − | With respect to the operating environment: if misbehavior involving a specific VM / container / bare-metal environment is involved, please describe the environment in detail:
| + | |
| − | | + | |
| − | * Linux Distro (e.g. Ubuntu 14.04.3 LTS, CentOS-7, etc.)
| + | |
| − | * NIC type(s) (ixgbe, i40e, enic, etc. etc.), vhost-user, tuntap
| + | |
| − | * NUMA configuration if applicable
| + | |
| − | | + | |
| − | Please note the CPU architecture (x86_86, aarch64), and hardware platform.
| + | |
| − | | + | |
| − | Please attempt to reproduce issues using unmodified vpp engine images.
| + | |
| − | | + | |
| − | ==== "Show" Command Output ====
| + | |
| − | | + | |
| − | Every situation is different. If the issue involves a sequence of debug CLI command, please enable CLI command logging, and send the sequence involved. Note that the debug CLI is a developer's tool - '''no warranty express or implied''' - and that we may choose not to fix debug CLI bugs.
| + | |
| − | | + | |
| − | Please include "show error" [error counter] output. It's often helpful to "clear error", send a bit of traffic, then "show error" particularly when running vpp on a noisy networks.
| + | |
| − | | + | |
| − | Please include ip4 / ip6 / mpls FIB contents ("show ip fib", "show ip6 fib", "show mpls fib", "show mpls tunnel").
| + | |
| − | | + | |
| − | Please include "show hardware", "show interface", and "show interface address" output
| + | |
| − | | + | |
| − | Here is a consolidated set of commands that are generally useful before/after sending traffic. Before sending traffic:
| + | |
| − | | + | |
| − | vppctl clear hardware
| + | |
| − | vppctl clear interface
| + | |
| − | vppctl clear error
| + | |
| − | vppctl clear run
| + | |
| − | | + | |
| − | Send some traffic and then issue the following commands:
| + | |
| − |
| + | |
| − | vppctl show version verbose
| + | |
| − | vppctl show hardware
| + | |
| − | vppctl show hardware address
| + | |
| − | vppctl show interface
| + | |
| − | vppctl show run
| + | |
| − | sudo vppctl show error
| + | |
| − | | + | |
| − | Here are some protocol specific show commands that may also make sense. Only include those features which have been configured:
| + | |
| − | | + | |
| − | vppctl show l2fib
| + | |
| − | vppctl show bridge-domain
| + | |
| − | | + | |
| − | vppctl show ip fib
| + | |
| − | vppctl show ip arp
| + | |
| − | | + | |
| − | vppctl show ip6 fib
| + | |
| − | vppctl show ip6 neighbors
| + | |
| − | | + | |
| − | vppctl show mpls fib
| + | |
| − | vppctl show mpls tunnel
| + | |
| − | | + | |
| − | ==== Network Topology ====
| + | |
| − | | + | |
| − | Please include a crisp description of the network topology, including L2 / IP / MPLS / segment-routing addressing details. If you expect folks to reproduce and debug issues, this is a must.
| + | |
| − | | + | |
| − | At or above a certain level of topological complexity, it becomes problematic to reproduce the original setup.
| + | |
| − | | + | |
| − | ==== Packet Tracer Output ====
| + | |
| − | | + | |
| − | If you capture packet tracer output which seems relevant, please include it:
| + | |
| − | | + | |
| − | vppctl trace add dpdk-input 100 # or similar
| + | |
| − |
| + | |
| − | <i><send-traffic></i>
| + | |
| − |
| + | |
| − | vppctl show trace
| + | |
| − | | + | |
| − | === Capturing post-mortem data ===
| + | |
| − | | + | |
| − | It should go without saying, but anyhow: '''please put post-mortem data in accessible places.''' Time wasted trying to acquire accounts, credentials, and IP addresses simply delays problem resolution.
| + | |
| − | | + | |
| − | ==== syslog output ====
| + | |
| − | | + | |
| − | The vpp signal handler typically writes a certain amount of data in /var/log/syslog before exiting. Make sure to check for evidence, e.g via "grep /usr/bin/vpp /var/log/syslog" or similar.
| + | |
| − | | + | |
| − | ==== Binary API Trace ====
| + | |
| − | | + | |
| − | If the issue involves a sequence of control-plane API messages - even a very long sequence - please enable control-plane API tracing. Control-plane API post-mortem traces end up in /tmp/api_post_mortem.<pid>. Please provide a pointer [accessible to the general public!] to the binary API trace. These API traces are especially helpful in cases where the vpp engine is throwing traffic on the floor, e.g. for want of a default route or similar.
| + | |
| − | | + | |
| − | Make sure to leave the default stanza "... api-trace { on } ... " in the vpp startup configuration file /etc/vpp/startup.conf, or to include it in the command line arguments passed by orchestration software.
| + | |
| − | | + | |
| − | ==== Core Files ====
| + | |
| − | | + | |
| − | Production systems, as well as long-running pre-production soak-test systems, '''must''' arrange to collect core images. There are various ways to configure core image capture, including e.g. the Ubuntu "corekeeper" package. In a pinch, the following very basic sequence will capture usable vpp core files in /tmp/dumps.
| + | |
| − | | + | |
| − | # mkdir -p /tmp/dumps
| + | |
| − | # sysctl -w debug.exception-trace=1
| + | |
| − | # sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
| + | |
| − | # ulimit -c unlimited
| + | |
| − | #
| + | |
| − | # Since Linux 3.6, if /proc/sys/fs/suid_dumpable is set to 2
| + | |
| − | # ("suidsafe"), the pattern must be either an absolute pathname
| + | |
| − | # (starting with a leading '/' character) or a pipe,
| + | |
| − | # echo 2 > /proc/sys/fs/suid_dumpable
| + | |
| − | | + | |
| − | Vpp core files often appear enormous. Gzip typically compresses them to manageable sizes. A multi-GByte corefile often compresses to 10-20 Mbytes.
| + | |
| − | | + | |
| − | Make sure to leave the default stanza "... unix { ... full-coredump ... } ... " in the vpp startup configuration file /etc/vpp/startup.conf, or to include it in the command line arguments passed by orchestration software.
| + | |
| − | | + | |
| − | Core files from private, modified images are discouraged. If it's necessary to go that route, please copy the '''exact''' Debian packages (or RPMs) corresponding to the core file to the same public place as the core file. In particular:
| + | |
| − | | + | |
| − | vpp_<version>_<arch>.deb # the vpp executable
| + | |
| − | vpp-dbg_<version>_<arch>.deb # debug symbols
| + | |
| − | vpp-dev_<version>_<arch>.deb # development package
| + | |
| − | vpp-lib_<version>_<arch>.deb # shared libraries
| + | |
| − | vpp-plugins_<version>_<arch>.deb # plugins
| + | |
| − | | + | |
| − | Please include the full commit-ID the Jira ticket.
| + | |
| − | | + | |
| − | If we go through the setup process only to discover that the image and core files don't match, it will simply delay resolution of the issue. And it will annoy the heck out of the engineer who just wasted their time. Exact means '''exact''', not "oh, gee, I added a few lines of debug scaffolding since then..." or "I rebuilt the Debian packages when you asked for them." See the [[VPP/CoreFileMismatch | Core File Mismatch Example]].
| + | |
