View source for VPP/Command-line Arguments

=== Introduction  ===

The VPP network stack comes with several configuration options that can be provided either on the command line or in a configuration file. 

Specific applications built on the stack have been known to require a dozen arguments, depending on requirements. This page describes commonly-used options and parameters.

You can find command-line argument parsers in the source code by searching for instances of the <code>VLIB_CONFIG_FUNCTION</code> macro. The invocation <code>VLIB_CONFIG_FUNCTION (foo_config, "foo")</code> will cause the function <code>foo_config</code> to receive all parameters given in a parameter block named "<code>foo</code>", for example: "<code>foo { arg1 arg2 arg3 ... }</code>".

Please help maintain and improve this document!

=== Command-line arguments  ===

Parameters are grouped by a section name. When providing more than one parameter to a section all parameters for that section must be wrapped in curly braces.

 # /usr/bin/vpp unix { interactive cli-listen 127.0.0.1:5002 }
 &lt;startup diagnostic messages&gt;
 <span style="color:red">    _______    _     </span>   _   _____  ___ 
 <span style="color:red"> __/ __/ _ \  (_)__  </span>  | | / / _ \/ _ \
 <span style="color:red"> _/ _// // / / / _ \ </span>  | |/ / ___/ ___/
 <span style="color:red"> /_/ /____(_)_/\___/ </span>  |___/_/  /_/    
 
 vpp# &lt;start-typing&gt;

When providing only one such parameter the braces are optional. For example, the following command argument, <code>unix interactive</code> does not have braces: 

 # /usr/bin/vpp unix interactive
 &lt;startup diagnostic messages&gt;
 <span style="color:red">    _______    _     </span>   _   _____  ___ 
 <span style="color:red"> __/ __/ _ \  (_)__  </span>  | | / / _ \/ _ \
 <span style="color:red"> _/ _// // / / / _ \ </span>  | |/ / ___/ ___/
 <span style="color:red"> /_/ /____(_)_/\___/ </span>  |___/_/  /_/    
 
 vpp# &lt;start-typing&gt;

The command line can be presented as a single string or as several; anything given on the command line is concatenated with spaces into a single string before parsing.

VPP applications must be able to locate their own executable images. The simplest way to ensure this will work is to invoke a VPP application by giving its absolute path; for example: <code>/usr/bin/vpp &lt;options&gt;</code>. At startup, VPP applications parse through their own ELF-sections [primarily] to make lists of init, configuration, and exit handlers.

When developing with VPP, in gdb it's often sufficient to start an application like this: 

 (gdb) run unix interactive

=== Configuration file ===

It is also possible to supply parameters in a startup configuration file the path of which is provided to the VPP application on its command line.

The format of the configuration file is a simple text file with the same content as the command line but with the benefit of being able to use newlines to make the content easier to read. For example:

<code>
 unix {
   nodaemon
   /var/log/vpp/vpp.log
   full-coredump
   cli-listen localhost:5002
 }
 
 api-trace {
   on
 }
 
 dpdk {
   dev 0000:03:00.0
 }
</code>

VPP is then instructed to load this file with the <code>-c</code> option:

 /usr/bin/vpp -c /etc/vpp/startup.conf

'''Note:''' In VPP 18.04, the default log file location was moved from '' '/tmp/vpp.log' '' to '' '/var/log/vpp/vpp.log' ''.  The VPP code is indifferent to the file location. However, if SELinux is enabled, then the new location is required for the file to be properly labelled. Check your local startup.conf file for the log file location on your system.

=== Configuration parameters ===

This is not an exhaustive list of parameters available. Readers who find information is missing or inaccurate are encouraged to add to this wiki page!


==== "unix" parameters  ====

{| class="wikitable"
| <code>interactive</code>
| Attach CLI to stdin/out and provide a debugging command line interface. Implies <code>nodaemon</code>.
|-
| <code>nodaemon</code>
| Do not fork / background the vpp process. Typical when invoking VPP applications from a process monitor. 
|-
| <code>exec &lt;filename&gt;</code>
<code>startup-config &lt;filename&gt;</code>
| Read startup operational configuration from <code>filename</code>. The contents of the file will be performed as though entered at the CLI.
The two keywords are aliases for the same function; if both are specified, only the last will have an effect.
|-
| <code>log &lt;filename&gt;</code>
| Logs the startup configuration and all subsequent CLI commands in <code>filename</code>.
Very useful in situations where folks don't remember or can't be bothered to include CLI commands in bug reports. 
|-
| <code>full-coredump</code>
| Ask the Linux kernel to dump all memory-mapped address regions, instead of just text+data+bss.
|-
| <code>cli-listen localhost:5002</code>
| Bind the CLI to listen at address <code>localhost</code> on TCP port <code>5002</code>.
This will accept an <code>ipaddress:port</code> pair or a filesystem path; in the latter case
a local Unix socket is opened instead.
|-
| <code>cli-line-mode</code>
| Disable character-by-character I/O on stdin. Useful when combined with, for example, <code>emacs M-x gud-gdb</code>.
|-
| <code>cli-prompt &lt;string&gt;</code>
| Configure the CLI prompt to be <code>string</code>.
|-
| <code>cli-history-limit &lt;nn&gt;</code>
| Limit commmand history to <code>nn</code> lines. A value of <code>0</code> disables command history. Default value: <code>50</code>
|-
| <code>cli-no-banner</code>
| Disable the login banner on stdin and Telnet connections.
|-
| <code>cli-no-pager</code>
| Disable the output pager.
|-
| <code>cli-pager-buffer-limit &lt;nn&gt;</code>
| Limit pager buffer to <code>nn</code> lines of output. A value of 0 disables the pager. Default value: <code>100000</code>
|}

==== "tuntap" parameters  ====

The "tuntap" driver configures a point-to-point interface between the vpp engine and the local Linux kernel stack. This allows e.g. users to ssh to the host / VM / container via vpp "revenue" interfaces. It's marginally useful, and is currently disabled by default. To [dynamically] create TAP interfaces - the preferred scheme - see the "tap_connect" binary API 

{| class="wikitable"
| <code>tuntap disable</code>
| Disable the tun/tap driver if desired 
|-
| <code>tuntap ethernet</code>
| Create a tap device (ethernet MAC) instead of a tun device (point-to-point tunnel) 
|-
| <code>tuntap have-normal-interface</code>
| Treat the host Linux stack as a routing peer instead of programming vpp interface L3 addresses onto the tun/tap devices.
You'll need to manually configure the Linux network stack "vnet" interface, configure vlan subinterfaces if desired, etc. 
|-
| <code>tuntap name ishmael</code>
| Call the tun/tap device ishmael
|}

Here's a typical multiple parameter invocation: 

  tuntap { ethernet have-normal-interface name vpp1 }

==== "api-trace" parameters  ====

The ability to trace, dump, and replay control-plane API traces makes all the difference in the world when trying to understand what the control-plane has tried to ask the forwarding-plane to do. 

{| class="wikitable"
| <code>on</code>
| Enable API trace capture from the beginning of time, and arrange for a post-mortem dump of the API trace if the application terminates abnormally.
By default, the (circular) trace buffer will be configured to capture 256K traces
|-
| <code>nitems &lt;nnn&gt;</code>
| Configure the circular trace buffer to contain the last &lt;nnn&gt; entries. By default, the trace buffer captures the last 256K API messages received.
|}

Typically, one simply enables the API message trace scheme: 

 api-trace { on }

==== "api-segment" parameters ====

These values control various aspects of the binary API interface to VPP.

{| class="wikitable"
| <code>prefix &lt;path&gt;</code>
| Sets the prefix prepended to the name used for shared memory (SHM) segments.
The default is empty, meaning shared memory segments are created directly in the SHM directory <code>/dev/shm</code>.

It is worth noting that on many systems <code>/dev/shm</code> is a symbolic link to somewhere else in the file system;

Ubuntu links it to <code>/run/shm</code>.
|-
| <code>uid &lt;number<nowiki>|</nowiki>name&gt;</code>
| Sets the user ID or name that should be used to set the ownership of the shared memory segments.
Defaults to the same user that VPP is started with, probably <code>root</code>.
|-
| <code>gid &lt;number<nowiki>|</nowiki>name&gt;</code>
| Sets the group ID or name that should be used to set the ownership of the shared memory segments.
Defaults to the same group that VPP is started with, probably <code>root</code>.
|}

==== "cpu" parameters  ====

Command-line CPU configuration controls the creation of named thread types, and the cpu affinity thereof. As of this writing, the cpu configuration function supports these parameters: 

{| class="wikitable"
| <code>workers &lt;nn&gt;</code>
| Create &lt;nn&gt; worker threads.
|-
| <code>io &lt;nn&gt;</code>
| Create <code>nn</code> i/o threads.
|-
| <code>main-thread-io</code>
| Handle i/o devices from thread 0, hand off traffic to worker threads. Requires "workers &lt;nn&gt;". 
|-
| <code>skip-cores &lt;nn&gt;</code>
| Leave the low <code>nn</code> bits of the process affinity mask clear. 
|}

In some cases, it may be necessary to place thread instances manually: 

{| class="wikitable"
| <code>main-core &lt;n&gt;</code>
| Assign main thread to specific core.
|-
| <code>coremask-workers &lt;hex-mask&gt;</code>
| Place worker threads according to the bitmap <code>hex-mask</code>.
|-
| <code>corelist-workers &lt;list&gt;</code>
| Same as <code>coremask-workers</code> but accepts a list of cores instead of a bitmap.
|-
| <code>coremask-io &lt;hex-mask&gt;</code>
| Place I/O threads according to the bitmap <code>hex-mask</code>.
|-
| <code>corelist-io &lt;list&gt;</code>
| Same as <code>coremask-io</code> but accepts a list of cores instead of a bitmap.
|}

Note that the "main" thread always occupies the lowest core-id specified in the DPDK [process-level] coremask. 

Here's a full-bore manual placement example: 

 /cisco/bin/vpe  unix interactive tuntap disable cpu { main-thread-io coremask-workers 18 coremask-stats 4 } dpdk {coremask 1e}
 
 # taskset -a -p &lt;vpe-pid&gt;
 pid 16251's current affinity mask: 2   # main thread
 pid 16288's current affinity mask: ffffff # DPDK interrupt thread (not bound to a core)
 pid 16289's current affinity mask: 4 # stats thread
 pid 16290's current affinity mask: 8 # worker thread 0
 pid 16291's current affinity mask: 10 # worker thread 1

==== "dpdk" parameters  ====

Command line DPDK configuration controls a number of parameters, including device whitelisting, the number of CPUs available for launching dpdk-eal-controlled threads, the number of I/O buffers, and the process affinity mask. 

In addition, the DPDK configuration function attempts to support all of the DPDK EAL configuration parameters. 

Popular options include: 

{| class="wikitable"
| <code>coremask &lt;hex&gt;</code>
| Sets the process-level coremask. See [[VPP/Command-line_Arguments#.22cpu.22_parameters|previous section on cpu parameters]] for more details. 
|-
| <code>no-hugetlb</code>
| Don't use huge TLB pages. Potentially useful for running simulator images. 
|-
| <code>dev &lt;pci-dev&gt;</code>
| White-list [as in, attempt to drive] a specific PCI device. PCI-dev is a string of the form "<code>DDDD:BB:SS.F</code>" where:
* <code>DDDD</code> = Domain
* <code>BB</code> = Bus Number
* <code>SS</code> = Slot number
* <code>F</code> = Function
This is the same format used in the linux sysfs tree (i.e. <code>/sys/bus/pci/devices</code>) for PCI device directory names. 

VLAN stripping for each PCI device on the white-list can be specified using [{vlan-strip-offload on|off}]. VLAN stripping is off by default for all NICs except VICs, using ENIC driver, which has VLAN stripping on by default. To disable VLAN stripping for a VIC vNIC on PCI address 0000:0c:00.0, for example, specify "dev 0000:0c:00.0 {vlan-strip-offload off}" on the white-list.
|-
| <code>vdev &lt;eal-command&gt;</code>
| Provide a DPDK EAL command to specify bonded Ethernet interfaces, operating modes and PCI addresses of slave links. For example:
 vdev eth_bond0,mode=2,slave=0000:0f:00.0,slave=0000:11:00.0,xmit_policy=l34
 vdev eth_bond1,mode=2,slave=0000:10:00.0,slave=0000:12:00.0,xmit_policy=l34
Only XOR balanced (mode 2) mode is supported. 
|-
| <code>num-mbufs &lt;nn&gt;</code>
| Number of I/O buffers. The default is <code>16384</code>.
You need to increase this if you have a lot of physical interfaces with multiple RSS queues.
|-
| <code>kni &lt;nn&gt;</code>
| Number of KNI interfaces. Refer to the DPDK documentation 
|-
| <code>uio-driver &lt;name&gt;</code>
| Defaults to "<code>igb_uio</code>" [the Ubuntu default]. The names "<code>uio_pci_generic</code>" or "<code>vfio-pci</code>" are other possible values.
|}

All of the DPDK EAL options should be available. See <code>.../open-repo/vnet/vnet/devices/dpdk/dpdk.h</code>, look at the set of "<code>foreach_eal_XXX</code>" macros.

==== "l2tp" parameters  ====

ip6-l2tpv3 configuration controls the method used to locate a specific ip6-l2tpv3 tunnel. The following settings are mutually exclusive 

{| class="wikitable"
| <code>lookup-v6-src</code>
| Lookup tunnel by ip6 source address.
|-
| <code>lookup-v6-dst</code>
| Lookup tunnel by ip6 destination address.
|-
| <code>lookup-session-id</code>
| Lookup tunnel by l2tpv3 session identifier.
|}

Refer to <code>.../open-repo/vnet/vnet/devices/dpdk/{dpdk.h, init.c}</code> for more details.

==== "oam" parameters  ====

Oam configuration controls the (ip4-icmp) interval, and number of misses allowed before reporting an oam target down to any registered listener.

{| class="wikitable"
| <code>interval &lt;floating-point-seconds&gt;</code>
| Interval between sending oam ip4 icmp messages.
|-
| <code>misses-allowed &lt;nn&gt;</code>
| Number of misses before declaring an oam target down.
|}

==== "heapsize" parameter  ====

Heapsize configuration controls the size of the main heap. The heap size is configured very early in the boot sequence, before loading plug-ins or doing much of anything else. 

{| class="wikitable"
| <code>heapsize &lt;nn&gt;M <nowiki>|</nowiki> &lt;nn&gt;G</code>
| Specifies the size of the heap in mb or gb.
Setting the main heap size to 4gb or more requires recompilation of the entire system with <code>CLIB_VEC64 &gt; 0</code>. See <code>.../clib/clib/vec_bootstrap.h</code>.
|}

==== "plugin_path" parameter  ====

Plugin_path configuration controls the set of directories searched for vlib plugins. Supply a colon-separated list of (absolute) directory names: 

{| class="wikitable"
| <code>plugin_path dir1:dir2:...:dirN</code>
| Configure the search path for plugins.
|}

==== "cj" parameters  ====

The cj thread-safe circular log buffer scheme is occasionally useful when chasing bugs. Calls to it should not be checked in. See <code>.../vlib/vlib/unix/cj.c</code>.

{| class="wikitable"
| <code>records &lt;nn&gt;</code>
| Configure the number of cj records in the circular buffer.
|-
| <code>on</code>
| Turns on logging at the earliest possible moment.
|}

==== "vhost-user" parameters  ====

Vhost-user configuration parameters control the vhost-user driver. 

{| class="wikitable"
| <code>coalesce-frames &lt;nn&gt;</code>
| Subject to deadline-timer expiration - see next item - attempt to transmit at least <code>nn</code> packet frames. 
|-
| <code>coalesce-time &lt;seconds&gt;</code>
| Hold packets no longer than (floating-point) <code>seconds</code> before transmitting them.
|-
| <code>dont-dump-memory</code>
| vhost-user shared-memory segments can add up to a large amount of memory, so it's handy to avoid adding them to corefiles when using a significant number of such interfaces. 
|}

==== "l2learn" parameter  ====

Configures the number of L2 (MAC) addresses in the L2 FIB at any one time. Defaults to 1M entries. 

{| class="wikitable"
| <code>limit &lt;nn&gt;</code>
| Limits the size of the L2 FIB to <code>nn</code> concurrent entries.
|}

==== "socksvr" parameters  ====

Create a socket server for API <code>.../vlibmemory/socket_api.c</code>.

{| class="wikitable"
| <code>socket-name &lt;filename&gt;</code>
| Configure API socket filename
|-
| <code>default</code>
| Default API filename (/run/vpp-api.sock)
|}

==== "statseg" parameters  ====

Configure the shared memory stats segment <code>.../vpp/stats/stat_segment.c</code>.

{| class="wikitable"
| <code>socket-name &lt;filename&gt;</code>
| Configure stats segment socket filename
|-
| <code>default</code>
| Use default socket filename (/run/vpp/stats.sock)
|-
| <code>per-node-counters on off</code>
| Enable or disable per node performance counters /sys/node
|}