View source for VPP/How To Use The C API

__TOC__

== Introduction ==

So, VPP is pretty cool and you want to use it to make your own router.  Yeah, me too!
You could write it using the Python, Java or Lua API to VPP.  But why use one of those
classically hip languages when you could go old-school and use the C API?

Now that we are on the same page, let's dig in!

== C API Overview ==

The API at the C source code level
* All API functions are described in C header files.
* All interaction with the VPP API is through direct calls to C functions.

There are several VPP related libraries of code available to you,
the C application developer.  Here is a quick overview of the libraries:

* vlib
* vlibapi
* vlibmemory
* vlibsocket
*: Collectively, the bulk of the VPP C-function API calls.

* vpp
*: VPP messaging API proper.

* vppinfra
*: Some basic infrastructure pieces.  Vectors, hash, bitops, etc.

* vpp_plugins

* dpdk
*: The Data-plane development kit.  The primary purpose is two-fold:
*:* First, to receive networking packets from an interface and place them in a convenient, vector in memory.
*:* Second, to take network packets from an in-memory vector and send them out on an interface.  This library, though built as part of VPP, has its own project and development here: FIXME: DPDK URL ref.

* jvpp-common
* pneum
* svm
* vat
* vnet

=== The Runtime Environment ===

The runtime environment for the application is explicitly multi-threaded.


== Message API Review ==

This section is a quick review of material
from the [[VPP/API_Concepts|API Concepts]] Wiki page.

There are two basic types of message exchanges that you can
have with the VPP library.

First, there is a simple "request and reply" type.  These messages
are a simple exchange of data initiated by you, the Application Writer,
and cause a single reply message to be returned from VPP back to you.
You construct a message, and you get a reply back.

The initiating request message of this form all have names that
DO NOT end in "DUMP".  For example VL_API_SW_INTERFACE_ADD_DEL_ADDRESS.
Its corresponding reply message is the exact same name with "_REPLY"
as a suffix.  You are expected to write a handler for this reply
with the name sw_interface_add_del_address_reply() and prototype

<pre>
void
vl_api_sw_interface_add_del_address_reply_t_handler(
			vl_api_sw_interface_add_del_address_reply *mp);
</pre>

The second type of message exchange is a "dump" request and a repeated
reply message in the form of multiple "details".  The requesting message
has a name ending in the suffix "_DUMP", and a reply message ending
in the suffix "_DETAILS".  For example, the message VL_API_IP_ADDRESS_DUMP
would have a reply handler function with the following prototype:

<pre>
void
vl_api_ip_address_details_t_handler(vl_api_ip_address_details_t *mp);
</pre>


=== Important Information about Reply Handlers ===

There are a few critical pieces of information about
reply handlers (both simple and details) that you need to keep in mind.

First, reply handlers are run asynchronously.

Think about it for a second.  You make some API message request.
Fire it off into the library and forget about it.  And then what?
Well, at *some point*, you are told, the reply handler will run.
When, exactly, will that be?  The answer is simple:  Whenever it can.
In fact, it could be being called in a different thread of execution
while your mainline app just kept slogging along toward the next API
call it wants to make, and maybe even *does* make.

The point is, the reply handlers could be running simultaneously
with other code in your application.  Are you locking your shared
data structures?  Do they need locks?  Are you writing independent
code and data streams that won't interfere with each other?  Good.

Second, the dump/details style messages need to be finished using
an additional, final CONTROL_PING message.

Because a details handler may be called multiple times as the result
of a single dump request, it must be terminated with a CONTROL_PING
message.  The purpose of the control ping message is to ensure that
the queue of sending-side messages waits for completion of all of the
sent details before allowing it to continue with new sent messages.

This waiting-until-complete mechanism is enforced in a few message
send and wait primitives discussed below.


== API Include Files ==

As with all program developers, laziness is considered a virtue.
The VPP development is no exception!  Each module of the VPP library
has a single "API description file" that is the single source for
the definition of messages used by that module.

Scripts then process that API file and turn it into various C header
definitions with types, message names, print functions, etc.
The API files are exported and installed with names like "vpe.api.h"
or "snat.api.h" under /usr/include/vpp/api or /usr/include/vpp_plugins.

Within the various api.h are different sections that supply a vast
set of definitions or data about each message within that group of API
messages.

As of this writing, the various sections are:

   vl_msg_id
   vl_union_id
   vl_printfun
   vl_endianfun
   vl_api_version
   vl_typedefs
   vl_msg_name
   vl_msg_name_crc_list

I'm going to talk about the three of these (vl_msg_id, vl_typedefs,
and vl_msg_name_crc_list) in some detail, and then just blindly reference
the others, wave my hands and vaguely suggest you do something similar.

Here is the trick.  Each of these sections constructs lists of messages
and data about messages within the header files.  Then, you are expected
to *do* something with this list.  There are a couple idioms used here.
You should familiarize yourself with them.

First, the bulk declarations.

If you crack open, say, vpe.api.h, somewhere in there you will
see a section of code all surrounded by an #ifdef like this:

<pre>
/****** Typedefs *****/

#ifdef vl_typedefs

typedef VL_API_PACKED(struct _vl_api_create_vlan_subif {
    u16 _vl_msg_id;
    u32 client_index;
    u32 context;
    u32 sw_if_index;
    u32 vlan_id;
}) vl_api_create_vlan_subif_t;

...
</pre>

You are expected to #define vl_typedefs and #include the vpe.api.h file
before you can use any of these message typedefs.  Like this:

<pre>
#define vl_typedefs
#include <vpp/api/vpe_all_api_h.h>
#undef vl_typedefs
</pre>

In fact, you will need to do something similar for each section of
each *.api.h file you want.  So you could do this:

<pre>
#define vl_typedefs
#include <vpp/api/vpe_all_api_h.h>
#undef vl_typedefs

#define vl_endianfun
#include <vpp/api/vpe_all_api_h.h>
#undef vl_endianfun
...
</pre>

That can be simplified so that just one #include takes place:

<pre>
#define vl_typedefs
#define vl_endianfun
#include <vpp/api/vpe_all_api_h.h>
#undef vl_typedefs
#undef vl_endianfun
...
</pre>

Good, but we have to be careful here.  Indiscriminately applying this
approach to all sections of the various *.api.h won't be right.

Some of the sections expect a context for processing in, say, a C function,
and not just a declaration context at the C file scope.  In fact, some
of the sections expect a #define functional definition that is applied
to each *something* in the list.  I know that sounds a bit vague.
So let's see it in action.

Let's say you want (and you do!) a message-specific printf() function
for debug purposes, you don't have to write them all.
They are almost free:

<pre>
#define vl_print(handle, ...)	vlib_cli_output (handle, __VA_ARGS__)
#define vl_printfun
#include <vpp/api/vpe_all_api_h.h>
#undef vl_printfun
</pre>

Notice here we had to both define a vl_print() function to say what was to be
done with each member within the bl_printfun section of the *.api.h file.
In this case, the buried calls to vl_print() are rewritten as an actual
output routine from VLIB called vlib_cli_output().

If you wanted no output, you could easily have instead written:

<pre>
#define vl_print(handle, ...)
</pre>

While this idiom produces declarations at the C file level, it shows
the use of an additional processing macro that might be defined during the
processing of the entire section of the *.api.h file.

Another idiom frequently employed allows you to apply a #define operation
to each member of a list.  For example, the section that processes
the message names and CRC list looks like this:

<pre>
/****** Message name, crc list ******/

#ifdef vl_msg_name_crc_list
#define foreach_vl_msg_name_crc_vpe \
_(VL_API_CREATE_VLAN_SUBIF, create_vlan_subif, af9ae1e9) \
_(VL_API_CREATE_VLAN_SUBIF_REPLY, create_vlan_subif_reply, 8f36b888) \
...
</pre>

It then lists each and every message, by name, and its CRC value.
Here, you are expected to #define an operation called "_", that will
be applied to each of those message-crc lines.

I'll show where, why and how to do this below.


== VPE vs. Plugin Messages ==

In all of the discussion above, I have meticulously referenced either
the "vpe.api.h" or something vague like "the various *.api.h" files.
That is because I was dodging an issue in order to simplify the
discussion above.

But now it is time to face that fate head on!

The VPP library presents you, the application developer, with two
types of modules: the base module (AKA "vpe"), and plugins.
In fact several plugins.  You must use the base module, vpe, but
you can opt to or opt not to use each plugin.

As of this writing, the plugins that have API module-level support
are ACL and SNAT. DPDK as a plugin is in the works and coming soon.
By library name, there are also plugins for flowerpkt, ioam, ila and lb.

I'm going to use SNAT as a point of discussion here.

=== Message Registration ===

In order to use the messages provided by a *.api.h file, (yes, either
the vpe.api.h, or any of the plugin api files like snat.api.h),
you must first register your intent to use the message by stating
what the handler will be for the reply (or details) message that result

Here is an example registration call for the VL_API_CONTROL_PING_REPLY message.

<pre>
        vl_msg_api_set_handlers(VL_API_CONTROL_PING_REPLY,
                                control_ping_reply,
                                vl_api_control_ping_reply_t_handler,
				vl_noop_handler,
                                vl_api_control_ping_t_endian,
                                vl_api_control_ping_t_print,
                                sizeof(vl_api_control_ping_t),
				1);
</pre>

The prototype for this call is found in "vlibapi/api.h":

<pre>
void vl_msg_api_set_handlers (int msg_id,
                              char *msg_name,
                              void *handler,
                              void *cleanup,
                              void *endian,
                              void *print,
                              int msg_size,
                              int traced);
</pre>

While it is hard to tell even from the prototype, all of the parameters
handler, cleanup, endian, and print are function names.  The endian and
print handlers can be automatically generated from the *.api.h files
using the #define vl_endianfun and vl_printfun, respectively.
The cleanup function was summarily tossed at the vl_noop_handler.

On the other hand, the important function here is the one named
vl_api_control_ping_reply_t_handler().  Like all of the messages you
intend to send to the VPP library as an API call, you must supply a
reply (generic) handler.  That handler must be type-matched to each
message and it must be named either <msg_name>_reply_t_handler() if
it is a simple reply handler, or <msg_name>_details_t_handler() if
it is a "dump" style reply handler.

You are going to write a bunch of reply handlers.  One for every VPP message.
It could get tedious.  To simplify some of the tedium, and yet at the same
time obscure it behind some C token pasting the folowing idiom has become
the accepted norm.

First, list the messages for which you want replies.  This includes
both the simple replies and the details replies.  For the VPE module,
we'll stat with just one message, the so-called "control-ping":

<pre>
#define foreach_vpe_api_reply_msg                               \
        _(CONTROL_PING_REPLY, control_ping_reply)
</pre>

Then, inside a C function, call the message registration API:

<pre>
static void vpp_vpe_init(void)
{
#define _(N,n)                                                  \
        vl_msg_api_set_handlers(VL_API_##N,                     \
                                #n,                             \
                                vl_api_##n##_t_handler,         \
                                vl_noop_handler,                \
                                vl_api_##n##_t_endian,          \
                                vl_api_##n##_t_print,           \
                                sizeof(vl_api_##n##_t), 1);
        foreach_vpe_api_reply_msg;
#undef _
}
</pre>

When you need to handle another message, just add it to the list:

<pre>
#define foreach_vpe_api_reply_msg                               \
        _(CONTROL_PING_REPLY, control_ping_reply)               \
        _(ANOTHER_MSG_REPLY, another_msg_reply)                 \
        _(SOME_OTHER_MSG_REPLY, some_other_msg_reply)
</pre>


=== Will the Real Message Id Please Stand Up? ===

I'd like to say it was all just that easy.  But I'd be lying.
There are two wrinkles swirling around the actual API message numbers
that you have to understand.  First, the enumerated list as found
in the *.api.h files can lie to you unless you have some conditional
knowledge about exactly how the corresponding VPP binary code that
implements the API calls was compiled.  Hint, you don't.
And furthermore, you can't.

The second complicating issue is that each plugin is optional.
It may or may not be present in your application.  And the order
it is loaded may or may not correspond to how the plugin in VPP
libraries are loaded.  So you don't know what their message ids
actually are.  Yet.

So I'll explain how to get the real API message ids.

==== Message Ids for VPE Messages ====

There is no One True Global Message Id space that has all of
the messages laid out once-and-for-all.
Instead, the base messages from the VPE module are always loaded first.
They occupy the message ids starting at 0, and end, well, wherever
they want to. You don't know and I don't know.

In fact, there could be gaps in this message space or shifting of
message ids around as parts are conditionally compiled in or out.
The only way to know is to first call into VPP using a different
C function to request the real message id.  No, really.  So save
this returned value in a map.

Oh, and, to keep everyone honest, and to make sure that both your
app and the vlib implementation are talking about the *same* message,
with the same fields and contents, a CRC hash value of the message
structure is constructed in place in the API file beside each message
that it is willing to name.  And you, the application writer, have
to supply both the message name, and the CRC value to VPP when you
request the *real* message id.

It goes like this.  First declare a global message id map:

<pre>
/*
 * VPP API requires a dynamic mapping for message ids
 * based on the message name and a CRC of the message structure.
 */
struct vpp_msg_map *vmid_map;
</pre>

then modify the vpp_vpe_init() function, above, so that it constructs
the message id map first, then registers it using the *real* id:

<pre>
static void vpp_vpe_init(void)
{
        /*
         * Construct a mapping from <message-name_CRC> to msg ID.
         */
        vmid_map = calloc(VL_MSG_FIRST_AVAILABLE, sizeof(*vmid_map));

#define _(msg_id, name, crc)    \
        vpp_set_msg_id_map(msg_id, #name "_" #crc);

        foreach_vl_msg_name_crc_vpe;
#undef _

#define _(N,n)                                                  \
        vl_msg_api_set_handlers(vmid_map[VL_API_##N].id,                \
                                #n,                             \
                                vl_api_##n##_t_handler,         \
                                vl_noop_handler,                \
                                vl_api_##n##_t_endian,          \
                                vl_api_##n##_t_print,           \
                                sizeof(vl_api_##n##_t), 1);
        foreach_vpe_api_reply_msg;
#undef _
}
</pre>

If we *think* we want to send message VL_API_<something>, we *really*
have to set the message id to be vmid_map[VL_API_<something>] instead.

OK, that takes care of the base message ids from the VPE module.
Let's turn to plugin message ids now.

==== Message Ids for a Plugin's Messages ====

Luckily, VPP remembered where all the VPE messages landed, and kept
track of the last-used-message-id.
Now when the first plugin is loaded, its messages get assigned a base id
somewhere after the VPE messages, and are then linearly present for all of
the plugin's messages, ending somewhere.  Again, we can only surmise it
to be "this module's base id" plus "the number of messages in this module."

So, as each plugin is loaded, it is assigned a base message id.

Ooo, we better keep track of the base message id for each plugin.
So, here is the magic incantation, er, API call sequence to do that
for the SNAT plugin.  Others are similar!

<pre>
        u8 *plugin_name;	/* Yeah, like "char *" */
        u16 base_msg_id;

        /*
         * Determine the base msg-id for SNAT plugin messages.
         */
        plugin_name = format(0, "snat_%08x%c", snat_api_version, 0);
	base_msg_id = vl_client_get_first_plugin_msg_id((char *)plugin_name);
	if (base_msg_id == (u16) ~0) {
                vec_free(plugin_name);
                return -1;
        }
</pre>

Now if you are really following along at home, you, the astute reader
and Application Builder, are hopefully asking yourself a question
similar to this: "So, if we had to register the VPE messages using a
mapped message id, what do we do for the plugin message ids?"

Good question.  It is similar, naturally, but just a bit different.
Again, we begin by declaring a list of the message replies that will
be interesting to us:

<pre>
#define foreach_snat_api_reply_msg                                      \
        _(SNAT_ADD_ADDRESS_RANGE_REPLY, snat_add_address_range_reply)   \
        _(SNAT_INTERFACE_ADD_DEL_FEATURE_REPLY,                         \
          snat_interface_add_del_feature_reply)                         \
        _(SNAT_ADD_STATIC_MAPPING_REPLY, snat_add_static_mapping_reply)
...
</pre>

And then in a C function, determine the base address and call the
message registration, as before-ish:

<pre>
/*
 * SNAT Private Data
 */
struct snat_data {
        u16 base_msg_id;
};

struct snat_data snat_data;


int vpp_snat_init(void)
{
        u8 *plugin_name;
        u16 base_msg_id;

        fprintf(stderr, "%s: Starting\n", __func__);

        /*
         * Determine the base msg-id for SNAT plugin messages.
         */
        plugin_name = format(0, "snat_%08x%c", snat_api_version, 0);
        base_msg_id = vl_client_get_first_plugin_msg_id((char *)plugin_name);
        if (base_msg_id == (u16) ~0) {
                vec_free(plugin_name);
                return -1;
        }

        snat_data.base_msg_id = base_msg_id;
        fprintf(stderr, "%s: Base message_id is %d\n", __func__, base_msg_id);

#define _(N,n)                                                  \
        vl_msg_api_set_handlers(vmid_map_fn(VL_API_##N),        \
                                #n,                             \
                                vl_api_##n##_t_handler,         \
                                vl_noop_handler,                \
                                vl_api_##n##_t_endian,          \
                                vl_api_##n##_t_print,           \
                                sizeof(vl_api_##n##_t), 1);
        foreach_snat_api_reply_msg;
#undef _

        return 0;
}
</pre>

So, notice two things here.  I created a "snat_data" structure
with a base_msg_id in it.  And I captured and stored the base
message id in that global.  What you can't see yet, is that
I also buried the use of that variable inside of the function
called vmid_map_fn(), used as the first parameter to the message
registration API call.

You might correctly surmise that the purpose of this vmid_map_fn()
is to correctly map to a 0-based message set if it is a VPE message,
and a snat_data.base_msg_id-based message set if it is a SNAT message.

You would be correct.

Enough talk.  Let's build an application now.


== Building Your C Application ==

A simple and direct application will likely need to use the
following VPP  libraries during its link:

* vlibmemoryclient
* vlibapi
* svm
* vppinfra
* vlib

And the following standard libraries as well:

*pthread
*m
* rt
* dl
* crypto

That is, use a linker line like this:

<pre>
VPP_LIBS        += -lvlibmemoryclient
VPP_LIBS        += -lsvm
VPP_LIBS        += -lvppinfra
VPP_LIBS        += -lvlib

LDFLAGS         += $(VPP_LIBS)
LDFLAGS         += -lpthread -lm -lrt -ldl -lcrypto

$(SOFILE): ${OBJS}
        $(QUIET_SLINK)$(CC) -o $(SOFILE) -shared $(CFLAGS) $(LDFLAGS) ${OBJS}
</pre>

You recognize Makefile snippets when you see them, right?  Good.

You will want both /usr/include and /usr/include/vpp_plugins on your
list of INCLUDES.

<pre>
INCLUDES += -I/usr/include/vpp_plugins
CFLAGS   += $(INCLUDES)
</pre>

== Initialization ==

To get an application started, let's construct a file where we will
set up the main connection to the VPP library and start the process
of registering the API messages.  I did this in a file called api_setup.c.


The VPP library supplies a global you will need to reference but not declare:

<pre>
api_main_t *api_main;
</pre>

Not surprisingly, it comes from vlibapi/api.h!

These are several more globals that are needed; they are not really
necessary to the VPP librariy itself, but they will be needed to
hold some important information about the API connection with VLIB.

Here is the set I made up and used.
Place these all in a structure, or leave them bare.  Your call:

<pre>
static int api_connected;
static char api_connection_name[API_NAME_LEN];
static unix_shared_memory_queue_t *api_vl_input_queue;
static u32 api_client_index;
static volatile int api_result_ready;
static volatile int api_result_retval;
static clib_time_t api_clib_time;
</pre>

On the other hand, here are two globals that the VPP library must have,
does not supply itself, and instead you must:

<pre>
vlib_main_t vlib_global_main;
vlib_main_t **vlib_mains;
</pre>


The easiest initialization and connection to the VPP library
might look like this:

<pre>
static int vpp_connect_to_vlib(char *name)
{
        api_main_t *am;

        if (vl_client_connect_to_vlib("/vpe-api",
                                      api_connection_name,
                                      API_RX_Q_SIZE) < 0) {
                svm_region_exit();
                return -1;
        }

        am = &api_main;
        api_vl_input_queue = am->shmem_hdr->vl_input_queue;
        api_client_index = am->my_client_index;
        api_connected = 1;

        return 0;
}
</pre>

You might have something that disconnects, reconnects, needs a retry
or whatever.  So keep track of api_connected or not.  Have your message
sending ensure connection prior to launching a message.  Whatever.



== Constructing, Sending, and Receiving API Messages ==

=== Allocating Messages ===

Here's the first dirty little tid-bit that must be part of your message allocation scheme:
all messages are strongly typed.  That is, each individual message has a type definition
that is specific to that particular message.  There is a C typedef for each and every
message.  Now that may seem like a fine idea, and it is, but it has ramifications on
the allocation scheme.  Specifically, it means there is also a per-message size.

There is not, however, an allocation function for each message and each message type.
Instead, there is a single message allocator function, vl_msg_api_alloc(), that accepts
the size, in bytes, of a message to be generically allocated.  LIke this:

<pre>
         mp = vl_msg_api_alloc(size);
</pre>

Here, mp, is the accepted "message pointer" variable for any message.
Any message.  Which message?  And how do you know the size of its type?
Good questions.   In fact, the answer to those questions lead to some
seriously grody, not-even-inline, macro definitions for the main message
allocation "function".

Perhaps something like this:

<pre>

</pre>


=== Using Message Fields ===

All data in the messages and in the replies are in "network byte order".
That means when constructing a message to send to VPP, you must convert
each field of host-byte-ordered data into network-byte-order like this:

    mp = allocate-some-message-structure();

    mp->sw_if_index = htonl(my_sw_if_index);

    send-some-message(mp);

Conversely, upon receiving a reply, do the inverse in a reply or
details handler:

    my_sw_if_index = ntohl(mp->sw_if_index);

Naturally, use the operation of the correct size:

    ntohl(), htonl()	u32 or long data
    ntohs(), htons()	u16 or short data

Single-byte quantities need no byte swapping, right?  Right.

Larger or bytes-stream like values need careful handling too.
As strings are represented as an array of u8 values, simply
supply a copy of the byte array.  Be careful to honor a 0
termination (normal C string) or counted quantity (like an
IP address).

Ooo, yeah, IP addresses.  Let's talk about those for a  minute.
These days, there are both IPv4 and IPv6 addresses.  The former
are 4-bytes long, usually represented as a dotted quad (10.10.220.1)
or as the actual string of a dotted quad ("10.10.220.1").
The latter is a 16-byte mess usually represented as a prefix and
host portion separated by colons constructed in an obscure,
closely guarded, and secret incantation.

I'm not going to describe how to convert from a string-form of
an IP address into the equivalent byte-stream from first principles.
Instead I will leave you with these code snippets.

You might use something like this on a request message:

<pre>
	char *ip_str;
        u8 ip_address[16];
	u8 is_ipv6;

	mp = allocate-some-message-with-an-ip-address-field();
	mp->is_ipv6 = is_ipv6;

	if (is_ipv6) {
	        ip_str = "::1";
		inet_pton(AF_INET6, ip_str, &ip_address);
		cli_memcpy(mp->ip_address, ip_address, sizeof(ip6_address_t))
	} else {
	        ip_str = "192.168.7.13";
		inet_pton(AF_INET, ip_str, &ip_address);
		cli_memcpy(mp->ip_address, ip_address, sizeof(ip4_address_t))
	}
</pre>

And you might have something like this in a reply or details handler:

<pre>
        char addrbuf[40];

	if (is_ipv6) {
                inet_ntop(AF_INET6, mp->ip, addrbuf, sizeof(addrbuf));
        } else {
                inet_ntop(AF_INET, mp->ip, addrbuf, sizeof(addrbuf));
        }
</pre>

If you don't want to use inet_ntop() and inet_pton(), feel free to
write your own, of course.

=== Sending and Waiting on Messages ===

- S(), W()

- talk about cli_timer_start() too