Difference between revisions of "VPP/How To Use The C API"

Revision as of 16:52, 9 March 2017

Introduction

So, VPP is pretty cool and you want to use it to make your own router. Yeah, me too! You could write it using the Python, Java or Lua API to VPP. But why use one of those classically hip languages when you could go old-school and use the C API?

Now that we are on the same page, let's dig in!

C API Overview

The API at the C source code level

• All API functions are described in C header files.
• All interaction with the VPP API is through direct calls to C functions.

There are several VPP related libraries of code available to you, the C application developer. Here is a quick overview of the libraries:

• vlib
• vlibapi
• vlibmemory
• vlibsocket

  Collectively, the bulk of the VPP C-function API calls.

• vpp

  VPP messaging API proper.

• vppinfra

  Some basic infrastructure pieces. Vectors, hash, bitops, etc.

• vpp_plugins

• dpdk

  The Data-plane development kit. The primary purpose is two-fold:

  • First, to receive networking packets from an interface and place them in a convenient, vector in memory.
  • Second, to take network packets from an in-memory vector and send them out on an interface. This library, though built as part of VPP, has its own project and development here: FIXME: DPDK URL ref.

• jvpp-common
• pneum
• svm
• vat
• vnet

The Runtime Environment

The runtime environment for the application is explicitly multi-threaded.

Message API Review

This section is a quick review of material from the API Concepts Wiki page.

There are two basic types of message exchanges that you can have with the VPP library.

First, there is a simple "request and reply" type. These messages are a simple exchange of data initiated by you, the Application Writer, and cause a single reply message to be returned from VPP back to you. You construct a message, and you get a reply back.

The initiating request message of this form all have names that DO NOT end in "DUMP". For example VL_API_SW_INTERFACE_ADD_DEL_ADDRESS. Its corresponding reply message is the exact same name with "_REPLY" as a suffix. You are expected to write a handler for this reply with the name sw_interface_add_del_address_reply() and prototype

void
vl_api_sw_interface_add_del_address_reply_t_handler(
			vl_api_sw_interface_add_del_address_reply *mp);

The second type of message exchange is a "dump" request and a repeated reply message in the form of multiple "details". The requesting message has a name ending in the suffix "_DUMP", and a reply message ending in the suffix "_DETAILS". For example, the message VL_API_IP_ADDRESS_DUMP would have a reply handler function with the following prototype:

void
vl_api_ip_address_details_t_handler(vl_api_ip_address_details_t *mp);

Important Information about Reply Handlers

There are a few critical pieces of information about reply handlers (both simple and details) that you need to keep in mind.

First, reply handlers are run asynchronously.

Think about it for a second. You make some API message request. Fire it off into the library and forget about it. And then what? Well, at *some point*, you are told, the reply handler will run. When, exactly, will that be? The answer is simple: Whenever it can. In fact, it could be being called in a different thread of execution while your mainline app just kept slogging along toward the next API call it wants to make, and maybe even *does* make.

The point is, the reply handlers could be running simultaneously with other code in your application. Are you locking your shared data structures? Do they need locks? Are you writing independent code and data streams that won't interfere with each other? Good.

Second, the dump/details style messages need to be finished using an additional, final CONTROL_PING message.

Because a details handler may be called multiple times as the result of a single dump request, it must be terminated with a CONTROL_PING message. The purpose of the control ping message is to ensure that the queue of sending-side messages waits for completion of all of the sent details before allowing it to continue with new sent messages.

This waiting-until-complete mechanism is enforced in a few message send and wait primitives discussed below.

API Include Files

As with all program developers, laziness is considered a virtue. The VPP development is no exception! Each module of the VPP library has a single "API description file" that is the single source for the definition of messages used by that module.

Scripts then process that API file and turn it into various C header definitions with types, message names, print functions, etc. The API files are exported and installed with names like "vpe.api.h" or "snat.api.h" under /usr/include/vpp/api or /usr/include/vpp_plugins.

Within the various api.h are different sections that supply a vast set of definitions or data about each message within that group of API messages.

As of this writing, the various sections are:

  vl_msg_id
  vl_union_id
  vl_printfun
  vl_endianfun
  vl_api_version
  vl_typedefs
  vl_msg_name
  vl_msg_name_crc_list

I'm going to talk about the three of these (vl_msg_id, vl_typedefs, and vl_msg_name_crc_list) in some detail, and then just blindly reference the others, wave my hands and vaguely suggest you do something similar.

Here is the trick. Each of these sections constructs lists of messages and data about messages within the header files. Then, you are expected to *do* something with this list. There are a couple idioms used here. You should familiarize yourself with them.

First, the bulk declarations.

If you crack open, say, vpe.api.h, somewhere in there you will see a section of code all surrounded by an #ifdef like this:

/****** Typedefs *****/

#ifdef vl_typedefs

typedef VL_API_PACKED(struct _vl_api_create_vlan_subif {
    u16 _vl_msg_id;
    u32 client_index;
    u32 context;
    u32 sw_if_index;
    u32 vlan_id;
}) vl_api_create_vlan_subif_t;

...

You are expected to #define vl_typedefs and #include the vpe.api.h file before you can use any of these message typedefs. Like this:

#define vl_typedefs
#include <vpp/api/vpe_all_api_h.h>
#undef vl_typedefs

In fact, you will need to do something similar for each section of each *.api.h file you want. So you could do this:

#define vl_typedefs
#include <vpp/api/vpe_all_api_h.h>
#undef vl_typedefs

#define vl_endianfun
#include <vpp/api/vpe_all_api_h.h>
#undef vl_endianfun
...

That can be simplified so that just one #include takes place:

#define vl_typedefs
#define vl_endianfun
#include <vpp/api/vpe_all_api_h.h>
#undef vl_typedefs
#undef vl_endianfun
...

Good, but we have to be careful here. Indiscriminately applying this approach to all sections of the various *.api.h won't be right.

Some of the sections expect a context for processing in, say, a C function, and not just a declaration context at the C file scope. In fact, some of the sections expect a #define functional definition that is applied to each *something* in the list. I know that sounds a bit vague. So let's see it in action.

Let's say you want (and you do!) a message-specific printf() function for debug purposes, you don't have to write them all. They are almost free:

#define vl_print(handle, ...)	vlib_cli_output (handle, __VA_ARGS__)
#define vl_printfun
#include <vpp/api/vpe_all_api_h.h>
#undef vl_printfun

Notice here we had to both define a vl_print() function to say what was to be done with each member within the bl_printfun section of the *.api.h file. In this case, the buried calls to vl_print() are rewritten as an actual output routine from VLIB called vlib_cli_output().

If you wanted no output, you could easily have instead written:

#define vl_print(handle, ...)

While this idiom produces declarations at the C file level, it shows the use of an additional processing macro that might be defined during the processing of the entire section of the *.api.h file.

Another idiom frequently employed allows you to apply a #define operation to each member of a list. For example, the section that processes the message names and CRC list looks like this:

/****** Message name, crc list ******/

#ifdef vl_msg_name_crc_list
#define foreach_vl_msg_name_crc_vpe \
_(VL_API_CREATE_VLAN_SUBIF, create_vlan_subif, af9ae1e9) \
_(VL_API_CREATE_VLAN_SUBIF_REPLY, create_vlan_subif_reply, 8f36b888) \
...

It then lists each and every message, by name, and its CRC value. Here, you are expected to #define an operation called "_", that will be applied to each of those message-crc lines.

I'll show where, why and how to do this below.

VPE vs. Plugin Messages

In all of the discussion above, I have meticulously referenced either the "vpe.api.h" or something vague like "the various *.api.h" files. That is because I was dodging an issue in order to simplify the discussion above.

But now it is time to face that fate head on!

The VPP library presents you, the application developer, with two types of modules: the base module (AKA "vpe"), and plugins. In fact several plugins. You must use the base module, vpe, but you can opt to or opt not to use each plugin.

As of this writing, the plugins that have API module-level support are ACL and SNAT. DPDK as a plugin is in the works and coming soon. By library name, there are also plugins for flowerpkt, ioam, ila and lb.

I'm going to use SNAT as a point of discussion here.

Message Registration

In order to use the messages provided by a *.api.h file, (yes, either the vpe.api.h, or any of the plugin api files like snat.api.h), you must first register your intent to use the message by stating what the handler will be for the reply (or details) message that result

Here is an example registration call for the VL_API_CONTROL_PING_REPLY message.

        vl_msg_api_set_handlers(VL_API_CONTROL_PING_REPLY,
                                control_ping_reply,
                                vl_api_control_ping_reply_t_handler,
				vl_noop_handler,
                                vl_api_control_ping_t_endian,
                                vl_api_control_ping_t_print,
                                sizeof(vl_api_control_ping_t),
				1);

The prototype for this call is found in "vlibapi/api.h":

void vl_msg_api_set_handlers (int msg_id,
                              char *msg_name,
                              void *handler,
                              void *cleanup,
                              void *endian,
                              void *print,
                              int msg_size,
                              int traced);

While it is hard to tell even from the prototype, all of the parameters handler, cleanup, endian, and print are function names. The endian and print handlers can be automatically generated from the *.api.h files using the #define vl_endianfun and vl_printfun, respectively. The cleanup function was summarily tossed at the vl_noop_handler.

On the other hand, the important function here is the one named vl_api_control_ping_reply_t_handler(). Like all of the messages you intend to send to the VPP library as an API call, you must supply a reply (generic) handler. That handler must be type-matched to each message and it must be named either <msg_name>_reply_t_handler() if it is a simple reply handler, or <msg_name>_details_t_handler() if it is a "dump" style reply handler.

You are going to write a bunch of reply handlers. One for every VPP message. It could get tedious. To simplify some of the tedium, and yet at the same time obscure it behind some C token pasting the folowing idiom has become the accepted norm.

First, list the messages for which you want replies. This includes both the simple replies and the details replies. For the VPE module, we'll stat with just one message, the so-called "control-ping":

#define foreach_vpe_api_reply_msg                               \
        _(CONTROL_PING_REPLY, control_ping_reply)

Then, inside a C function, call the message registration API:

static void vpp_vpe_init(void)
{
#define _(N,n)                                                  \
        vl_msg_api_set_handlers(VL_API_##N,                     \
                                #n,                             \
                                vl_api_##n##_t_handler,         \
                                vl_noop_handler,                \
                                vl_api_##n##_t_endian,          \
                                vl_api_##n##_t_print,           \
                                sizeof(vl_api_##n##_t), 1);
        foreach_vpe_api_reply_msg;
#undef _
}

When you need to handle another message, just add it to the list:

#define foreach_vpe_api_reply_msg                               \
        _(CONTROL_PING_REPLY, control_ping_reply)               \
        _(ANOTHER_MSG_REPLY, another_msg_reply)                 \
        _(SOME_OTHER_MSG_REPLY, some_other_msg_reply)

Will the Real Message Id Please Stand Up?

I'd like to say it was all just that easy. But I'd be lying. There are two wrinkles swirling around the actual API message numbers that you have to understand. First, the enumerated list as found in the *.api.h files can lie to you unless you have some conditional knowledge about exactly how the corresponding VPP binary code that implements the API calls was compiled. Hint, you don't. And furthermore, you can't.

The second complicating issue is that each plugin is optional. It may or may not be present in your application. And the order it is loaded may or may not correspond to how the plugin in VPP libraries are loaded. So you don't know what their message ids actually are. Yet.

So I'll explain how to get the real API message ids.

Message Ids for VPE Messages

There is no One True Global Message Id space that has all of the messages laid out once-and-for-all. Instead, the base messages from the VPE module are always loaded first. They occupy the message ids starting at 0, and end, well, wherever they want to. You don't know and I don't know.

In fact, there could be gaps in this message space or shifting of message ids around as parts are conditionally compiled in or out. The only way to know is to first call into VPP using a different C function to request the real message id. No, really. So save this returned value in a map.

Oh, and, to keep everyone honest, and to make sure that both your app and the vlib implementation are talking about the *same* message, with the same fields and contents, a CRC hash value of the message structure is constructed in place in the API file beside each message that it is willing to name. And you, the application writer, have to supply both the message name, and the CRC value to VPP when you request the *real* message id.

It goes like this. First declare a global message id map:

/*
 * VPP API requires a dynamic mapping for message ids
 * based on the message name and a CRC of the message structure.
 */
struct vpp_msg_map *vmid_map;

then modify the vpp_vpe_init() function, above, so that it constructs the message id map first, then registers it using the *real* id:

static void vpp_vpe_init(void)
{
        /*
         * Construct a mapping from <message-name_CRC> to msg ID.
         */
        vmid_map = calloc(VL_MSG_FIRST_AVAILABLE, sizeof(*vmid_map));

#define _(msg_id, name, crc)    \
        vpp_set_msg_id_map(msg_id, #name "_" #crc);

        foreach_vl_msg_name_crc_vpe;
#undef _

#define _(N,n)                                                  \
        vl_msg_api_set_handlers(vmid_map[VL_API_##N].id,                \
                                #n,                             \
                                vl_api_##n##_t_handler,         \
                                vl_noop_handler,                \
                                vl_api_##n##_t_endian,          \
                                vl_api_##n##_t_print,           \
                                sizeof(vl_api_##n##_t), 1);
        foreach_vpe_api_reply_msg;
#undef _
}

If we *think* we want to send message VL_API_<something>, we *really* have to set the message id to be vmid_map[VL_API_<something>] instead.

OK, that takes care of the base message ids from the VPE module. Let's turn to plugin message ids now.

Message Ids for a Plugin's Messages

Luckily, VPP remembered where all the VPE messages landed, and kept track of the last-used-message-id. Now when the first plugin is loaded, its messages get assigned a base id somewhere after the VPE messages, and are then linearly present for all of the plugin's messages, ending somewhere. Again, we can only surmise it to be "this module's base id" plus "the number of messages in this module."

So, as each plugin is loaded, it is assigned a base message id.

Ooo, we better keep track of the base message id for each plugin. So, here is the magic incantation, er, API call sequence to do that for the SNAT plugin. Others are similar!

        u8 *plugin_name;	/* Yeah, like "char *" */
        u16 base_msg_id;

        /*
         * Determine the base msg-id for SNAT plugin messages.
         */
        plugin_name = format(0, "snat_%08x%c", snat_api_version, 0);
	base_msg_id = vl_client_get_first_plugin_msg_id((char *)plugin_name);
	if (base_msg_id == (u16) ~0) {
                vec_free(plugin_name);
                return -1;
        }

Now if you are really following along at home, you, the astute reader and Application Builder, are hopefully asking yourself a question similar to this: "So, if we had to register the VPE messages using a mapped message id, what do we do for the plugin message ids?"

Good question. It is similar, naturally, but just a bit different. Again, we begin by declaring a list of the message replies that will be interesting to us:

#define foreach_snat_api_reply_msg                                      \
        _(SNAT_ADD_ADDRESS_RANGE_REPLY, snat_add_address_range_reply)   \
        _(SNAT_INTERFACE_ADD_DEL_FEATURE_REPLY,                         \
          snat_interface_add_del_feature_reply)                         \
        _(SNAT_ADD_STATIC_MAPPING_REPLY, snat_add_static_mapping_reply)
...

And then in a C function, determine the base address and call the message registration, as before-ish:

/*
 * SNAT Private Data
 */
struct snat_data {
        u16 base_msg_id;
};

struct snat_data snat_data;


int vpp_snat_init(void)
{
        u8 *plugin_name;
        u16 base_msg_id;

        fprintf(stderr, "%s: Starting\n", __func__);

        /*
         * Determine the base msg-id for SNAT plugin messages.
         */
        plugin_name = format(0, "snat_%08x%c", snat_api_version, 0);
        base_msg_id = vl_client_get_first_plugin_msg_id((char *)plugin_name);
        if (base_msg_id == (u16) ~0) {
                vec_free(plugin_name);
                return -1;
        }

        snat_data.base_msg_id = base_msg_id;
        fprintf(stderr, "%s: Base message_id is %d\n", __func__, base_msg_id);

#define _(N,n)                                                  \
        vl_msg_api_set_handlers(vmid_map_fn(VL_API_##N),        \
                                #n,                             \
                                vl_api_##n##_t_handler,         \
                                vl_noop_handler,                \
                                vl_api_##n##_t_endian,          \
                                vl_api_##n##_t_print,           \
                                sizeof(vl_api_##n##_t), 1);
        foreach_snat_api_reply_msg;
#undef _

        return 0;
}

So, notice two things here. I created a "snat_data" structure with a base_msg_id in it. And I captured and stored the base message id in that global. What you can't see yet, is that I also buried the use of that variable inside of the function called vmid_map_fn(), used as the first parameter to the message registration API call.

You might correctly surmise that the purpose of this vmid_map_fn() is to correctly map to a 0-based message set if it is a VPE message, and a snat_data.base_msg_id-based message set if it is a SNAT message.

You would be correct.

Enough talk. Let's build an application now.

Building Your C Application

A simple and direct application will likely need to use the following libraries during its link:

vlibmemoryclient vlibapi svm vppinfra pthread m rt

That is, use a linker line like this:

VPP_LIBS   = vlibmemoryclient vlibapi svm vppinfra pthread m rt
VPP_LDADD  = $(foreach lib,$(VPP_LIBS),-l$(lib) )

LDFLAGS    += $(VPP_LDADD)
LDFLAGS    += $(VPP_LDFLAGS)
LDFLAGS    += -Wl,-soname -Wl,$(LIBNAME).so.0

$(SOFILE): ${OBJS}
        $(QUIET_SLINK)$(CC) -o $(SOFILE) -shared $(CFLAGS) $(LDFLAGS) ${OBJS}

You recognize Makefile snippets when you see them, right? Good.

You will want both /usr/include and /usr/include/vpp_plugins on your list of INCLUDES.

INCLUDES += -I/usr/include/vpp_plugins
CFLAGS   += $(INCLUDES)

Initialization

To get an application started, let's construct a file where we will set up the main connection to the VPP library and start the process of registering the API messages. I did this in a file called api_setup.c.

The VPP library supplies a global you will need to reference but not declare:

api_main_t *api_main;

Not surprisingly, it comes from vlibapi/api.h!

These are several more globals that are needed; they are not really necessary to the VPP librariy itself, but they will be needed to hold some important information about the API connection with VLIB.

Here is the set I made up and used. Place these all in a structure, or leave them bare. Your call:

static int api_connected;
static char api_connection_name[API_NAME_LEN];
static unix_shared_memory_queue_t *api_vl_input_queue;
static u32 api_client_index;
static volatile int api_result_ready;
static volatile int api_result_retval;
static clib_time_t api_clib_time;

On the other hand, here are two globals that the VPP library must have, does not supply itself, and instead you must:

vlib_main_t vlib_global_main;
vlib_main_t **vlib_mains;

The easiest initialization and connection to the VPP library might look like this:

static int vpp_connect_to_vlib(char *name)
{
        api_main_t *am;

        if (vl_client_connect_to_vlib("/vpe-api",
                                      api_connection_name,
                                      API_RX_Q_SIZE) < 0) {
                svm_region_exit();
                return -1;
        }

        am = &api_main;
        api_vl_input_queue = am->shmem_hdr->vl_input_queue;
        api_client_index = am->my_client_index;
        api_connected = 1;

        return 0;
}

You might have something that disconnects, reconnects, needs a retry or whatever. So keep track of api_connected or not. Have your message sending ensure connection prior to launching a message. Whatever.

Sending and Receiving Messages

Using Message Fields

All data in the messages and in the replies are in "network byte order". That means when constructing a message to send to VPP, you must convert each field of host-byte-ordered data into network-byte-order like this:

   mp = allocate-some-message-structure();

   mp->sw_if_index = htonl(my_sw_if_index);

   send-some-message(mp);

Conversely, upon receiving a reply, do the inverse in a reply or details handler:

   my_sw_if_index = ntohl(mp->sw_if_index);

Naturally, use the operation of the correct size:

   ntohl(), htonl()	u32 or long data
   ntohs(), htons()	u16 or short data

Single-byte quantities need no byte swapping, right? Right.

Larger or bytes-stream like values need careful handling too. As strings are represented as an array of u8 values, simply supply a copy of the byte array. Be careful to honor a 0 termination (normal C string) or counted quantity (like an IP address).

Ooo, yeah, IP addresses. Let's talk about those for a minute. These days, there are both IPv4 and IPv6 addresses. The former are 4-bytes long, usually represented as a dotted quad (10.10.220.1) or as the actual string of a dotted quad ("10.10.220.1"). The latter is a 16-byte mess usually represented as a prefix and host portion separated by colons constructed in an obscure, closely guarded, and secret incantation.

I'm not going to describe how to convert from a string-form of an IP address into the equivalent byte-stream from first principles. Instead I will leave you with these code snippets.

You might use something like this on a request message:

	char *ip_str;
        u8 ip_address[16];
	u8 is_ipv6;

	mp = allocate-some-message-with-an-ip-address-field();
	mp->is_ipv6 = is_ipv6;

	if (is_ipv6) {
	        ip_str = "::1";
		inet_pton(AF_INET6, ip_str, &ip_address);
		cli_memcpy(mp->ip_address, ip_address, sizeof(ip6_address_t))
	} else {
	        ip_str = "192.168.7.13";
		inet_pton(AF_INET, ip_str, &ip_address);
		cli_memcpy(mp->ip_address, ip_address, sizeof(ip4_address_t))
	}

And you might have something like this in a reply or details handler:

        char addrbuf[40];

	if (is_ipv6) {
                inet_ntop(AF_INET6, mp->ip, addrbuf, sizeof(addrbuf));
        } else {
                inet_ntop(AF_INET, mp->ip, addrbuf, sizeof(addrbuf));
        }

If you don't want to use inet_ntop() and inet_pton(), feel free to write your own, of course.

Allocating, Sending and Waiting on Messages

- Allocation, S(), W()

- talk about cli_timer_start() too