VPP/CodingTips

This page aims to capture some lessons learned based on the VPP programming experience. There are no absolutes of course, just some considerations.

General

use ASSERT()s, but...

use it only for the invariants that must hold true at all times. Do not ever use it for the absent error handling.

And remember that ASSERTs disappear in production images.

You can call abort() in places which you think will cause serious confusion if not announced even in production images.

Multithreading

use per-thread structures rather than global

Use this:

 pool_get(per_thread_pool[my_thread_index], p);

rather than this:

 spinlock;
 pool_get(shared_global_pool, p);
 unlock;
 p->mumble;

The lock protects the pool, but the p becomes obsolete the moment the lock releases - think of another thread trying to get one more item and the pool being full - it will get reallocated and this will result in a dangling reference into already free memory. Writing to that reference will bring a lot of hours of entertaining troubleshooting.

Data structures and memory

format() produces vectors, not C strings

Remember the distinction between “%v” and “%s”. Format produces vectors, not C-strings; specifically to make it easy to append data. If you need a C-string, either

 u8 *s = format (0, “This is a C-string...%c”, 0);

or

 u8 *s = format (0, “This is a C-string...”);
 vec_add1(s, 0);

use foo_elt_at_index(), foo_elt() rather than []

Of course if you validate the vector right before use, the below is perfectly sane way to write:

 vec_validate (v, i);
 v[i] = xxx;

But if you do not validate the vector and assume that the caller must have validated it, Murphy's law will dictate that sometimes they won't (even if the caller is your own code). So, a much safer option is to always use vec_elt() or vec_elt_at_index() - they are a bit more verbose, but you will sleep better.

code data structure display functions with gdb in mind

If you separate the functions displaying the status of your data structures into modules callable with just vlib_main as a parameter, and maybe some primitive arguments - you will be happy when you can call them from gdb during the debugging

Dave B. has added a set of functions in .../src/vnet/unix/gdb_funcs.c, specifically to help people in gdb. If you’ve never tried any of them, try the “show gdb” command:

 DBGvpp# show gdb
 vl(p) returns vec_len(p)
 pe(p) returns pool_elts(p)
 pifi(p, i) returns pool_is_free_index(p, i)
 debug_hex_bytes (ptr, n_bytes) dumps n_bytes in hex
 vlib_dump_frame_ownership() does what it says
 vlib_runtime_index_to_node_name (index) prints NN

Examples:

 (gdb) p vl(x) # prints vec_len (x)
 (gdb) p pifi(pool, i) # prints pool_is_free_index (pool, i)
 (gdb) p pe(pool) # prints pool_elts (pool)

segregating your heap

separate heap might be a good idea if your logic gets complicated. This way you get a good way to track your own memory usage and ensure that the faults are contained. If you create your own heap, remember to set a thread safe flag - the global heap is thread safe, and also to switch the heap back to what it was before calling the library functions! You do get a headache of that, but the bugs that this has the potential to create (crash due to an assertion failing about foo_header being a heap object) are not hard to find and fix.

debugging memory leaks and corruptions

The mheap allocator can trace allocations - with backtrace info - ordinarily I warm up data structures, then turn on leak tracing. After a few <whatevers>, a bit of “show memory” action will give leaked memory sizes, counts, and backtraces. That’s usually 100% effective in tracking down memory leaks.

If the heap is corrupted, sprinkle clib_mem_validate() calls in likely places. Eventually, you’ll find the offending bit of code.

---

This page was created and is maintained by ayourtch, capturing some of his ideas as well as suggestions from Dave Barach.