VPP/Command-line Arguments

Introduction

The VPP network stack comes with several configuration options that can be provided either on the command line or in a configuration file.

Specific applications built on the stack have been known to require a dozen arguments, depending on requirements. This page describes commonly-used options and parameters.

You can find command-line argument parsers in the source code by searching for instances of the VLIB_CONFIG_FUNCTION macro. The invocation VLIB_CONFIG_FUNCTION (foo_config, "foo") will receive all parameters given in a block named "foo", for example: "foo { arg1 arg2 arg3 ... }".

Please help maintain and improve this document!

Command-line arguments

Parameters are grouped by a section name. When providing more than one parameter to a section all parameters for that section must be wrapped in curly braces.

# /usr/bin/vpp unix { interactive cli-listen 127.0.0.1:5002 }
<startup diagnostic messages>
    _______    _        _   _____  ___ 
 __/ __/ _ \  (_)__    | | / / _ \/ _ \
 _/ _// // / / / _ \   | |/ / ___/ ___/
 /_/ /____(_)_/\___/   |___/_/  /_/    

vpp# <start-typing>

When providing only one such parameter the braces are optional. For example, the following command argument, unix interactive does not have braces:

# /usr/bin/vpp unix interactive
<startup diagnostic messages>
    _______    _        _   _____  ___ 
 __/ __/ _ \  (_)__    | | / / _ \/ _ \
 _/ _// // / / / _ \   | |/ / ___/ ___/
 /_/ /____(_)_/\___/   |___/_/  /_/    

vpp# <start-typing>

The command line can be presented as a single string or as several; anything given on the command line is concatenated with spaces into a single string before parsing.

VPP applications must be able to locate their own executable images. The simplest way to ensure this will work is to invoke a VPP application by giving its absolute path; for example: /usr/bin/vpp <options>. At startup, VPP applications parse through their own ELF-sections [primarily] to make lists of init, configuration, and exit handlers.

When developing with VPP, in gdb it's often sufficient to start an application like this:

(gdb) run unix interactive

Configuration file

It is also possible to supply parameters in a startup configuration file the path of which is provided to the VPP application on its command line.

The format of the configuration file is a simple text file with the same content as the command line but with the benefit of being able to use newlines to make the content easier to read. For example:

unix {
  nodaemon
  log /tmp/vpp.log
  full-coredump
  cli-listen 0.0.0.0:5002
}

api-trace {
  on
}

dpdk {
  dev 0000:03:00.0
}

VPP is then instructed to load this file with the -c option:

/usr/bin/vpp -c /etc/vpp/startup.conf

Configuration parameters

This is not an exhaustive list of parameters available. Readers who find information is missing or inaccurate are encouraged to add to this wiki page!

"unix" parameters

interactive	Attach CLI to stdin/out and provide a debugging command line interface. Implies nodaemon.
nodaemon	Do not fork / background the vpp process. Typical when invoking VPP applications from a process monitor.
cli-listen localhost:5002	Bind the CLI to listen at address localhost on TCP port 5002.
exec <filename>	Read startup configuration from filename.
log <filename>	Logs the startup configuration and all subsequent CLI commands in filename. Very useful in situations where folks don't remember or can't be bothered to include CLI commands in bug reports.
cli-line-mode	Disable character-by-character I/O on stdin. Useful when combined with, for example, emacs M-x gud-gdb.
cli-history-limit <nn>	Limit commmand history to nn lines. A value of 0 disables command history. Default value: 50
cli-no-banner	Disable the login banner on stdin and Telnet connections.
cli-no-pager	Disable the output pager.
cli-pager-buffer-limit <nn>	Limit pager buffer to nn lines of output. A value of 0 disables the pager. Default value: 100000
full-coredump	Ask the Linux kernel to dump all memory-mapped address regions, instead of just text+data+bss.

"tuntap" parameters

The "tuntap" driver configures a point-to-point interface between the vpp engine and the local Linux kernel stack. This allows e.g. users to ssh to the host / VM / container via vpp "revenue" interfaces. It's marginally useful, and is currently disabled by default. To [dynamically] create TAP interfaces - the preferred scheme - see the "tap_connect" binary API

tuntap disable	Disable the tun/tap driver if desired
tuntap ethernet	Create a tap device (ethernet MAC) instead of a tun device (point-to-point tunnel)
tuntap have-normal-interface	Treat the host Linux stack as a routing peer instead of programming vpp interface L3 addresses onto the tun/tap devices. You'll need to manually configure the Linux network stack "vnet" interface, configure vlan subinterfaces if desired, etc.
tuntap name ishmael	Call the tun/tap device ishmael

Here's a typical multiple parameter invocation:

 tuntap { ethernet have-normal-interface name vpp1 }

"api-trace" parameters

The ability to trace, dump, and replay control-plane API traces makes all the difference in the world when trying to understand what the control-plane has tried to ask the forwarding-plane to do.

on	Enable API trace capture from the beginning of time, and arrange for a post-mortem dump of the API trace if the application terminates abnormally. By default, the (circular) trace buffer will be configured to capture 256K traces
nitems <nnn>	Configure the circular trace buffer to contain the last <nnn> entries. By default, the trace buffer captures the last 256K API messages received.

Typically, one simply enables the API message trace scheme:

api-trace { on }

"cpu" parameters

Command-line CPU configuration controls the creation of named thread types, and the cpu affinity thereof. As of this writing, the cpu configuration function supports these parameters:

workers <nn>	Create <nn> worker threads.
io <nn>	Create nn i/o threads.
main-thread-io	Handle i/o devices from thread 0, hand off traffic to worker threads. Requires "workers <nn>".
skip-cores <nn>	Leave the low nn bits of the process affinity mask clear.

In some cases, it may be necessary to place thread instances manually:

main-core <n>	Assign main thread to specific core.
coremask-workers <hex-mask>	Place worker threads according to the bitmap hex-mask.
corelist-workers <list>	Same as coremask-workers but accepts a list of cores instead of a bitmap.
coremask-io <hex-mask>	Place I/O threads according to the bitmap hex-mask.
corelist-io <list>	Same as coremask-io but accepts a list of cores instead of a bitmap.

Note that the "main" thread always occupies the lowest core-id specified in the DPDK [process-level] coremask.

Here's a full-bore manual placement example:

/cisco/bin/vpe  unix interactive tuntap disable cpu { main-thread-io coremask-workers 18 coremask-stats 4 } dpdk {coremask 1e}

# taskset -a -p <vpe-pid>
pid 16251's current affinity mask: 2   # main thread
pid 16288's current affinity mask: ffffff # DPDK interrupt thread (not bound to a core)
pid 16289's current affinity mask: 4 # stats thread
pid 16290's current affinity mask: 8 # worker thread 0
pid 16291's current affinity mask: 10 # worker thread 1

"dpdk" parameters

Command line DPDK configuration controls a number of parameters, including device whitelisting, the number of CPUs available for launching dpdk-eal-controlled threads, the number of I/O buffers, and the process affinity mask.

In addition, the DPDK configuration function attempts to support all of the DPDK EAL configuration parameters.

Popular options include:

coremask <hex>	Sets the process-level coremask. See previous section on cpu parameters for more details.
no-hugetlb	Don't use huge TLB pages. Potentially useful for running simulator images.
dev <pci-dev>	White-list [as in, attempt to drive] a specific PCI device. PCI-dev is a string of the form "DDDD:BB:SS.F" where DDDD = Domain, BB = Bus Number, SS = Slot number, and F = Function. This is the same format used in the linux sysfs tree (i.e. /sys/bus/pci/devices) for PCI device directory names.
vdev <DPDK EAL command to create link bonding interfaces>	Specify bonded Ethernet interfaces, operating modes and PCI address of slave links. For example, "vdev eth_bond0,mode=2,slave=0000:0f:00.0,slave=0000:11:00.0,xmit_policy=l34 vdev eth_bond1,mode=2,slave=0000:10:00.0,slave=0000:12:00.0,xmit_policy=l34". Only XOR balanced (mode 2) mode is supported.
num-mbufs <nn>	Number of I/O buffers. You need to increase this if you have a lot of physical interfaces with multiple RSS queues (for example, 131072).
kni <nn>	Number of KNI interfaces. Refer to the DPDK documentation
uio-driver <name>	Defaults to "igb_uio" [the Ubuntu default]. The names "uio_pci_generic" or "vfio-pci" are other possible values.

All of the DPDK EAL options should be available. See .../open-repo/vnet/vnet/devices/dpdk/dpdk.h, look at the set of "foreach_eal_XXX" macros.

"l2tp" parameters

ip6-l2tpv3 configuration controls the method used to locate a specific ip6-l2tpv3 tunnel. The following settings are mutually exclusive

lookup-v6-src	Lookup tunnel by ip6 source address.
lookup-v6-dst	Lookup tunnel by ip6 destination address.
lookup-session-id	Lookup tunnel by l2tpv3 session identifier.

Refer to .../open-repo/vnet/vnet/devices/dpdk/{dpdk.h, init.c} for more details.

"oam" parameters

Oam configuration controls the (ip4-icmp) interval, and number of misses allowed before reporting an oam target down to any registered listener.

interval <floating-point-seconds>	Interval between sending oam ip4 icmp messages.
misses-allowed <nn>	Number of misses before declaring an oam target down.

"heapsize" parameter

Heapsize configuration controls the size of the main heap. The heap size is configured very early in the boot sequence, before loading plug-ins or doing much of anything else.

<nn>G

Specifies the size of the heap in mb or gb. Setting the main heap size to 4gb or more requires recompilation of the entire system with CLIB_VEC64 > 0. See .../clib/clib/vec_bootstrap.h.

"plugin_path" parameter

Plugin_path configuration controls the set of directories searched for vlib plugins. Supply a colon-separated list of (absolute) directory names:

plugin_path dir1:dir2:...:dirN

Configure the search path for plugins.

"cj" parameters

The cj thread-safe circular log buffer scheme is occasionally useful when chasing bugs. Calls to it should not be checked in. See .../vlib/vlib/unix/cj.c.

records <nn>	Configure the number of cj records in the circular buffer.
on	Turns on logging at the earliest possible moment.

"vhost-user" parameters

Vhost-user configuration parameters control the vhost-user driver.

coalesce-frames <nn>	Subject to deadline-timer expiration - see next item - attempt to transmit at least nn packet frames.
coalesce-time <seconds>	Hold packets no longer than (floating-point) seconds before transmitting them.
dont-dump-memory	vhost-user shared-memory segments can add up to a large amount of memory, so it's handy to avoid adding them to corefiles when using a significant number of such interfaces.

"l2learn" parameter

Configures the number of L2 (MAC) addresses in the L2 FIB at any one time. Defaults to 1M entries.

limit <nn>

Limits the size of the L2 FIB to nn concurrent entries.