Difference between revisions of "VPP/HostStack/LDP/nginx"
Florin.coras (Talk | contribs) (→Nginx) |
Florin.coras (Talk | contribs) (→Nginx) |
||
| (9 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
== VPP configuration == | == VPP configuration == | ||
| − | In addition to the typical vpp startup config parameters, the host stack requires the | + | In addition to the typical vpp startup config parameters, the host stack requires the one lower for enabling the session layer socket api for app attachment. For a list of startup parameters see [https://fdio-vpp.readthedocs.io/en/latest/gettingstarted/users/configuring/startup.html# here]. |
| − | + | session { use-app-socket-api } | |
| − | session { | + | |
| − | + | ||
== Nginx == | == Nginx == | ||
| Line 19: | Line 17: | ||
vcl { | vcl { | ||
heapsize 64M | heapsize 64M | ||
| + | segment-size 4000000000 | ||
| + | add-segment-size 4000000000 | ||
rx-fifo-size 4000000 | rx-fifo-size 4000000 | ||
tx-fifo-size 4000000 | tx-fifo-size 4000000 | ||
| − | + | app-socket-api /var/run/vpp/app_ns_sockets/default | |
} | } | ||
| − | The above configures vcl to request 4MB receive and transmit fifo sizes and it provides the path to vpp's | + | The above configures vcl to request 4MB receive and transmit fifo sizes and it provides the path to vpp's session layer socket api. |
To start nginx: | To start nginx: | ||
| Line 36: | Line 36: | ||
master_process on; | master_process on; | ||
| − | For high scale loads testing make sure to properly configure fifo sizes and the number of nginx workers. | + | == Performance testing == |
| + | |||
| + | For high scale loads testing make sure to properly configure fifo sizes and the number of nginx workers. Additionally, configurations known to improve RPS include: | ||
| + | |||
| + | worker_rlimit_nofile 10240; | ||
| + | events { | ||
| + | use epoll; | ||
| + | worker_connections 10240; | ||
| + | accept_mutex off; | ||
| + | multi_accept off; | ||
| + | } | ||
| + | http { | ||
| + | keepalive_timeout 300s; | ||
| + | keepalive_requests 1000000; | ||
| + | access_log off; | ||
| + | } | ||
| + | |||
| + | Also, it's recommended to start nginx pinned to cores on the numa used by vpp's workers and the nic using taskset. | ||
Latest revision as of 21:03, 22 November 2021
Example of how to run nginx via ldp and vcl on top of vpp's host stack.
VPP configuration
In addition to the typical vpp startup config parameters, the host stack requires the one lower for enabling the session layer socket api for app attachment. For a list of startup parameters see here.
session { use-app-socket-api }
Nginx
First start vpp and ensure that the network to the http client is working as expected. Then define the following environment variables with the appropriate paths
VCL_CFG=/path/to/vcl.conf LDP_PATH=/path/to/vpp/build-root/build-vpp[_debug]-native/vpp/lib/libvcl_ldpreload.so
And create a minimal startup configuration in /path/to/vcl.conf like the one lower:
vcl {
heapsize 64M
segment-size 4000000000
add-segment-size 4000000000
rx-fifo-size 4000000
tx-fifo-size 4000000
app-socket-api /var/run/vpp/app_ns_sockets/default
}
The above configures vcl to request 4MB receive and transmit fifo sizes and it provides the path to vpp's session layer socket api.
To start nginx:
sudo LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG nginx -c nginx.conf
A simple, known to work nginx.conf file includes:
worker_processes 4; daemon off; master_process on;
Performance testing
For high scale loads testing make sure to properly configure fifo sizes and the number of nginx workers. Additionally, configurations known to improve RPS include:
worker_rlimit_nofile 10240;
events {
use epoll;
worker_connections 10240;
accept_mutex off;
multi_accept off;
}
http {
keepalive_timeout 300s;
keepalive_requests 1000000;
access_log off;
}
Also, it's recommended to start nginx pinned to cores on the numa used by vpp's workers and the nic using taskset.
