VPP/Command-line Arguments

Introduction

The vpp network stack comes with a command-line interface that can be used for debugging purposes.

Please help maintain and improve this document!

Command-line arguments

The vpp stack can be configured from the command-line in a number of ways. Specific applications built on the stack have been known to require a dozen arguments, depending on requirements. This section describes commonly-used command-line options. These options are parsed by individual bits of code, similar to the action functions of the CLI parser.

You can find command-line argument parsers in the source code by searching for instances of the VLIB_CONFIG_FUNCTION macro. The actual command-line invocation syntax is extremely flexible.

macro invocation: VLIB_CONFIG_FUNCTION (foo_config, "foo")
command-line: "foo { arg1 arg2 arg3 ... }"

When using the command on the command line, surround the list of arguments with curly braces.

NOTE: If you supply a single argument to a specific configuration function, you do not need to surround the argument with curly braces. For example, the following command argument does not have braces:

# /cisco/bin/swt unix interactive
SWT: <start-typing..>

Vpp applications must be able to locate their own executable images. The simplest way to ensure this will work: invoke a vpp application by giving its absolute path. At startup, vpp aplications parse through their own Elf-sections [primarily] to make lists of init, configuration, and exit handlers. In gdb, it's often sufficient to start an application like this:

(gdb) r unix interactive

"unix" parameters

• interactive -- attach CLI to stdin/out
• nodaemon -- do not fork / background the vpp process, implied by "interactive". Typical when invoking vpp applications from a process monitor.
• cli-listen localhost:5002 -- bind CLI to localhost TCP-port 5002
• exec <filename> -- read startup configuration from <filename>
• log <filename> -- logs the startup configuration and all subsequent CLI commands in <filename>. Very useful in situations where folks don't remember of can't be bothered to include CLI commands in bug reports.
• cli-line-mode -- disable character-by-character I/O, useful when combined with emacs M-x telnet
• cli-history-limit <nn> -- limit commmand history to nn lines. Default value: 50
• full-coredump -- ask the Linux kernel to dump all memory-mapped address regions, instead of just text+data+bss.

"tuntap" parameters

The "tuntap" driver configures a point-to-point interface between the vpp engine and the local Linux kernel stack. This allows e.g. users to ssh to the host / VM / container via vpp "revenue" interfaces. It's marginally useful, and is currently disabled by default. To [dynamically] create TAP interfaces - the preferred scheme - see the "tap_connect" binary API

• "tuntap disable" -- disable the tun/tap driver if desired
• "tuntap ethernet" -- create a tap device (ethernet MAC) instead of a tun device (point-to-point tunnel)
• "tuntap have-normal-interface" -- treat the host Linux stack as a routing peer instead of programming vpp interface L3 addresses onto the tun/tap devices. You'll need to manually configure the Linux network stack "vnet" interface, configure vlan subinterfaces if desired, etc.
• "tuntap name ishmael" -- call the tun/tap device ishmael

Here's a typical multiple parameter invocation:

 tuntap { ethernet have-normal-interface name vpp1 }

"api-trace" parameters

The ability to trace, dump, and replay control-plane API traces makes all the difference in the world when trying to understand what the control-plane has tried to ask the forwarding-plane to do.

• "on" -- enable API trace capture from the beginning of time, and arrange for a post-mortem dump of the API trace if the application terminates abnormally. By default, the (circular) trace buffer will be configured to capture 256K traces

• "nitems <nnn>" -- Configure the circular trace buffer to contain the last <nnn> entries. By default, the trace buffer captures the last 256K API messages received.

Typically, one simply enables the API message trace scheme:

api-trace { on }

"cpu" parameters

Command-line CPU configuration controls the creation of named thread types, and the cpu affinity thereof. As of this writing, the cpu configuration function supports these parameters:

• "workers <nn>" - create <nn> worker threads.
• "io <nn>" - create <nn> i/o threads.
• "main-thread-io" - handle i/o devices from thread 0, hand off traffic to worker threads. Requires "workers <nn>".
• "skip-cores <nn>" - leave the low <nn> bits of the process affinity mask clear.

In some cases, it may be necessary to place thread instances manually:

• "main-core <n>" - assign main thread to specific core 
• "coremask-workers <hex-mask>" - place worker threads according to <hex-mask>
• "corelist-workers <list>" - same like coremask-workers but accepts list of cores instead of bitmap
• "coremask-io <hex-mask>" - place i/o threads according to <hex-mask>
• "corelist-io <list>" - same like coremask-io but accepts list of cores instead of bitmap

Note that the "main" thread always occupies the lowest core-id specified in the dpdk [process-level] coremask.

Here's a full-bore manual placement example:

/cisco/bin/vpe  unix interactive tuntap disable cpu { main-thread-io coremask-workers 18 coremask-stats 4 } dpdk {coremask 1e}

# taskset -a -p <vpe-pid>
pid 16251's current affinity mask: 2   # main thread
pid 16288's current affinity mask: ffffff # DPDK interrupt thread (not bound to a core)
pid 16289's current affinity mask: 4 # stats thread
pid 16290's current affinity mask: 8 # worker thread 0
pid 16291's current affinity mask: 10 # worker thread 1

"dpdk" parameters

Command line DPDK configuration controls a number of parameters, including device whitelisting, the number of CPUs available for launching dpdk-eal-controlled threads, the number of I/O buffers, and the process affinity mask.

In addition, the dpdk configuration function attempts to support all of the dpdk eal configuration parameters.

Popular options include:

• coremask <hex> - sets the process-level coremask. See previous section on cpu parameters for more details.
• no-hugetlb - don't use huge TLB pages. Potentially useful for running simulator images.
• dev <pci-dev> - white-list [as in, attempt to drive] a specific PCI device. PCI-dev is a string of the form "DDDD:BB:SS.F" where DDDD = Domain, BB = Bus Number, SS = Slot number, and F = Function. This is the same format used in the linux sysfs tree (i.e. /sys/bus/pci/devices) for PCI device directory names.
• vdev <DPDK EAL command to create link bonding interfaces> - specify bonded Ethernet interfaces, operating modes and PCI address of slave links. For example, "vdev eth_bond0,mode=2,slave=0000:0f:00.0,slave=0000:11:00.0,xmit_policy=l34 vdev eth_bond1,mode=2,slave=0000:10:00.0,slave=0000:12:00.0,xmit_policy=l34". Only XOR balanced (mode 2) mode is supported.
• num-mbufs <nn> - number of I/O buffers. You need to increase this if you have a lot of physical interfaces with multiple RSS queues (for example, 131072).
• kni <nn> - number of KNI interfaces. Refer to the dpdk documentation
• uio-driver <name> - defaults to "igb_uio" [the Ubuntu default]. The names "uio_pci_generic" or "vfio-pci" are other possible values.

All of the DPDK EAL options should be available. See .../open-repo/vnet/vnet/devices/dpdk/dpdk.h, look at the set of "foreach_eal_XXX" macros.

"l2tp" parameters

ip6-l2tpv3 configuration controls the method used to locate a specific ip6-l2tpv3 tunnel. The following settings are mutually exclusive

• lookup-v6-src - lookup tunnel by ip6 source address
• lookup-v6-dst - lookup tunnel by ip6 destination address
• lookup-session-id - lookup tunnel by l2tpv3 session identifier

Refer to .../open-repo/vnet/vnet/devices/dpdk/{dpdk.h, init.c} for more details

"oam" parameters

Oam configuration controls the (ip4-icmp) interval, and number of misses allowed before reporting an oam target down to any registered listener

• interval <floating-point-seconds> - interval between sending oam ip4 icmps
• misses-allowed <nn> - number of misses before declaring an oam target down

"heapsize" parameter

Heapsize configuration controls the size of the main heap. The heap size is configured very early in the boot sequence, before loading plug-ins or doing much of anything else.

• heapsize <nn>M | <nn>G - specifies the size of the heap in mb or gb. Setting the main heap size to 4gb or more requires recompilation of the entire system with CLIB_VEC64 > 0. See .../clib/clib/vec_bootstrap.h.

"plugin_path" parameter

Plugin_path configuration controls the set of directories searched for vlib plugins. Supply a colon-separated list of (absolute) directory names:

• plugin_path dir1:dir2:...:dirN

"cj" parameters

The cj thread-safe circular log buffer scheme is occasionally useful when chasing bugs. Calls to it should not be checked in. See .../vlib/vlib/unix/cj.c.

• records <nn> - configure the number of cj records in the circular buffer
• on - turns on logging at the earliest possible moment

"vhost-user" parameters

Vhost-user configuration parameters control the vhost-user driver.

• coalesce-frames <nn> - Subject to deadline-timer expiration - see next item - attempt to transmit at least NN packet frames.
• coalesce-time <fp-seconds> - hold packets no longer than <floating-point-seconds> before transmitting them
• dont-dump-memory - vhost-user shared-memory segments can add up to a large amount of memory, so it's handy to avoid adding them to corefiles when using a significant number of such interfaces.

"l2learn" parameter

Configures the number of L2 (MAC) addresses in the L2 FIB at any one time. Defaults to 1M entries.

limit <nn> - limits the size of the L2 FIB to <nn> concurrent entries.