VPP/CoreFileMismatch

Introduction

Trying to decode a core file given a set of Debian packages which do not exactly match it is typically a fool's errand. Here's a detailed example of what happens 90+% of the time. Note that experienced vpp engineers become quite exasperated when faced with a legitimate issue which they can't even begin to address.

The example below is from a case where the provided Debian packages are from a release image (-g -O2), but the core file is from a debug image (-g -O0). We've seen every imaginable variation on this theme.

Finding Version Info

We install the Debian packages, and use vppctl to "show version verbose"

vpp# sh ver verbose
Version:                  v19.01-rc0~591-g98d6b611
Compiled by:              root
Compile host:             7454ee796c9f
Compile date:             Wed Feb 20 20:40:41 UTC 2019
Compile location:         /vpp/vpp
Compiler:                 GCC 7.3.0
Current PID:              29215

And we look for some semblance of the provided date in the core file, with the "strings" utility:

# strings vpp_main-1550651139 | grep "Wed Feb 20"
vpp v19.01-rc0~591-g98d6b611 built by root on 7454ee796c9f at Wed Feb 20 08:12:32 UTC 2019

Characteristically, the core file date will be earlier than the Debian package date.

GDB backfire

Given these data, gdb simply refuses:

Reading symbols from /usr/bin/vpp...Reading symbols from /usr/lib/debug/.build-id/fd/4c081bd3953ed9a674a096e5b09ee30b672c2b.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 38733]
[New LWP 38735]
[New LWP 38736]
[New LWP 38737]
[New LWP 38734]
[New LWP 38738]
Core was generated by `/vpp/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp unix { log /var/log/vp'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f8d062db3f8 in ?? ()
[Current thread is 1 (LWP 38733)]
(gdb) bt
#0  0x00007f8d062db3f8 in ?? ()
#1  0x1313131313131313 in ?? ()
#2  0x1313131313131313 in ?? ()
#3  0x1313131313131313 in ?? ()
#4  0x1313131313131313 in ?? ()
#5  0x1313131313131313 in ?? ()
#6  0x1313131313131313 in ?? ()
#7  0x1313131313131313 in ?? ()
<snip>

At this point, we ask whether the responsible individual's parents were actually married, etc.