Difference between revisions of "VPP/HostStack/LDP/nginx"

Revision as of 21:00, 22 November 2021

Example of how to run nginx via ldp and vcl on top of vpp's host stack.

VPP configuration

In addition to the typical vpp startup config parameters, the host stack requires the ones lower for enabling the use of memfds segments for vpp message queues and to configure the binary api socket transport. For a list of startup parameters see here.

session { use-app-socket-api }

Nginx

First start vpp and ensure that the network to the http client is working as expected. Then define the following environment variables with the appropriate paths

VCL_CFG=/path/to/vcl.conf
LDP_PATH=/path/to/vpp/build-root/build-vpp[_debug]-native/vpp/lib/libvcl_ldpreload.so

And create a minimal startup configuration in /path/to/vcl.conf like the one lower:

vcl {
  heapsize 64M
  segment-size 4000000000
  add-segment-size 4000000000
  rx-fifo-size 4000000
  tx-fifo-size 4000000
  app-socket-api /var/run/vpp/app_ns_sockets/default
}

The above configures vcl to request 4MB receive and transmit fifo sizes and it provides the path to vpp's binary api socket.

To start nginx:

sudo LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG nginx -c nginx.conf

A simple, known to work nginx.conf file includes:

worker_processes 4;
daemon off;
master_process on;

Performance testing

For high scale loads testing make sure to properly configure fifo sizes and the number of nginx workers. Additionally, configurations known to improve RPS include:

worker_rlimit_nofile 10240;
events {
       use epoll;
       worker_connections  10240;
       accept_mutex       off;
       multi_accept       off;
}
http {
       keepalive_timeout 300s;
       keepalive_requests 1000000;
       access_log off;
}

Also, it's recommended to start nginx pinned to cores on the numa used by vpp's workers and the nic using taskset.