VPP/HostStack/LDP/nginx

Example of how to run nginx via ldp and vcl on top of vpp's host stack.

VPP configuration

In addition to the typical vpp startup config parameters, the host stack requires the one lower for enabling the session layer socket api for app attachment. For a list of startup parameters see here.

session { use-app-socket-api }

Nginx

First start vpp and ensure that the network to the http client is working as expected. Then define the following environment variables with the appropriate paths

VCL_CFG=/path/to/vcl.conf
LDP_PATH=/path/to/vpp/build-root/build-vpp[_debug]-native/vpp/lib/libvcl_ldpreload.so

And create a minimal startup configuration in /path/to/vcl.conf like the one lower:

vcl {
  heapsize 64M
  segment-size 4000000000
  add-segment-size 4000000000
  rx-fifo-size 4000000
  tx-fifo-size 4000000
  app-socket-api /var/run/vpp/app_ns_sockets/default
}

The above configures vcl to request 4MB receive and transmit fifo sizes and it provides the path to vpp's session layer socket api.

To start nginx:

sudo LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG nginx -c nginx.conf

A simple, known to work nginx.conf file includes:

worker_processes 4;
daemon off;
master_process on;

Performance testing

For high scale loads testing make sure to properly configure fifo sizes and the number of nginx workers. Additionally, configurations known to improve RPS include:

worker_rlimit_nofile 10240;
events {
       use epoll;
       worker_connections  10240;
       accept_mutex       off;
       multi_accept       off;
}
http {
       keepalive_timeout 300s;
       keepalive_requests 1000000;
       access_log off;
}

Also, it's recommended to start nginx pinned to cores on the numa used by vpp's workers and the nic using taskset.