VPP/Build System Deep Dive
Contents
Introduction
The vpp build system consists of a data-driven main makefile, and a set of makefile fragments. The various parts come together as the result of a set of well-thought-out conventions.
Repository Groups and Source Paths
The vpp build system brings together sets of git "repository groups." In a typical product development case, one would use two repository groups: a group that we will call for the purpose of this explanation "open-repo" (including the build system itself) and a group that we will call "closed-repo" comprising vendor-specific closed source. Each repository group consists of a number of independent git repositories.
The current vpp workspaces comprise a single repository group, ordinarily called "vpp". The file vpp/build-root/build-config.mk defines a key variable called SOURCE_PATH. The SOURCE_PATH variable names the set of repository groups. We recommend that you examine that file to understand the repositories that are being built.
Dependencies and Components
The vpp build system caters to components built with GNU autoconf / automake. Adding such components is a simple process. Dealing with components which use BSD-style raw Makefiles is a bit more difficult. Dealing with toolchain components such as GCC and GLIBC can be considerably more complicated.
The vpp build system is a single-pass build system. A partial order must exist for any set of components: the set of (a before b) tuples must resolve to an ordered list. If you create a circular dependency of the form; (a,b) (b,c) (c,a), gmake will try to build the target list, but there’s a 0.0% chance that the results will be pleasant. Cut-n-paste mistakes in .../build-data/packages/<oops>.mk can produce confusing failures.
In a single-pass build system, it’s best to separate libraries and applications which instantiate them. For example, if vpp depends on libfoo.a, and myapp depends on both vpp and libfoo.a, it's best to place libfoo.a and myapp in separate components. The build system will build libfoo.a, vpp, and then (as a separate component) myapp. If you try to build libfoo.a and myapp from the same component, it won’t work.
If you absolutely, positively insist on having myapp and libfoo.a in the same source tree, you can create a pseudo-component in a separate .mk file in the .../build-data/packages/ directory. Define the code phoneycomponent_source = realcomponent, and provide manual configure/build/install targets.
Separate components for myapp, libfoo.a, and vpp is the best and easiest solution. However, the “mumble_source = realsource” degree of freedom exists to solve intractable circular dependencies, such as: to build gcc-bootstrap, followed by glibc, followed by “real” gcc/g++ [which depends on glibc too].
.../vpp/build-root
The .../vpp/build-root directory contains the repository group specification build-config.mk, the main Makefile, and the system-wide set of autoconf/automake variable overrides in config.site. We'll describe these files in some detail. To be clear about expectations, the main Makefile and config.site file are subtle and complex. It's unlikely that you'll need or want to modify them. Poorly planned changes in either place typically cause bugs that are difficult to solve.
.../vpp/build-root/build-config.mk
As described above, the build-config.mk file is straightforward: it sets the make variable SOURCE_PATH to a list of repository group absolute paths.
The SOURCE_PATH variable
If you choose to move a workspace, make sure to modify the paths defined by the SOURCE_PATH variable. Those paths need to match changes you make in the workspace paths. For example, if you place the vpp directory in the workspace of a user named jsmith, you might need to change the SOURCE_PATH to:
SOURCE_PATH = /home/jsmithuser/workspace/vpp
.../vpp/build-root/Makefile
The main Makefile is complex. If you think you need to modify it, it's a good idea to do some research, or ask for advice before you change it.
The main Makefile was organized and designed to provide the following characteristics: excellent performance, absolutely accurate dependency processing, cache enablement, timestamp optimizations, git integration, extensibility, builds with cross-compilation tool chains, and builds with embedded Linux distributions.
If you need to do so, you can build double-cross tools. For example, you could: compile gdb on x86_64, to run on PowerPC, to debug the Xtensa instruction set.
The PLATFORM variable
The PLATFORM make/environment variable controls a number of important characteristics, primarily:
- cpu architecture, and
- the list of images to build.
The list of images to build is specified by the target: make PLATFORM=xxx install-packages. (See below for a description of that target.)
In the vpp case, we use "PLATFORM=vpp".
The main Makefile interprets $PLATFORM by attempting to -include the file <repo-group-root>/build-data/platforms.mk:
$(foreach d,$(FULL_SOURCE_PATH), \ $(eval -include $(d)/platforms.mk))
By convention, we don't define platforms in the .../<repo-group-root>/build-data/platforms.mk file. In the vpp case, we search for platform definition makefile fragments in .../vpp/build-data/platforms.mk, as follows:
$(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS), \ $(eval -include $(d)/platforms/*.mk))
With vpp, which uses the "vpp" platform as discussed above, we end up "-include"-ing .../vpp/build-data/platforms/vpp.mk. To make it easier to read here in this page, the following snippet omits several variable settings. Here are some of the contents of vpp.mk:
# vector packet processor vpp_arch = native vpp_native_tools = vppapigen vppversion vpp_root_packages = vpp vlib vlib-api vnet svm dpdk vpp-api-test \ vpp-japi vpp_configure_args_vpp = --with-dpdk vnet_configure_args_vpp = --with-dpdk # Set these parameters carefully. The vlib_buffer_t is 128 bytes, i.e. # dpdk_headroom = uiotarball_headroom = vlib_pre_data + 128 dpdk_configure_args_vpp = --with-headroom=256 vlib_configure_args_vpp = --with-pre-data=128 vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector-all -fPIC vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector-all -fPIC vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector -fPIC -pie vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector -fPIC -pie
The variable <platform-name>_arch sets the CPU architecture used to build the per-platform cross-compilation toolchain. With the exception of the "native" architecture - used in our example - the vpp build system produces cross-compiled binaries.
The variable <platform-name>_native_tools lists the required set of self-compiled build tools.
The variable <platform-name>_root_packages lists the set of images to build when specifying the target: make PLATFORM=<platform-name> TAG=<tag-name> [install-deb | install-rpm].
The TAG variable
The TAG variable indirectly sets CFLAGS and LDFLAGS, as well as the build and install directory names in the .../vpp/build-root directory. Here is the current "vpp" tag definition specified in the .../vpp/build-data/platforms/vpp.mk file:
vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector-all -fPIC vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector-all -fPIC vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector -fPIC -pie vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector -fPIC -pie
Executables built with PLATFORM=vpp TAG=vpp are built in .../vpp/build-root/build-vpp-x86_64, with executables installed in .../vpp/build-root/install-vpp-x86_64.
Important targets in the Makefile
The main Makefile and the various makefile fragments implement the following user-invoked targets:
| Target | Environment Variable Settings | Notes |
|---|---|---|
| bootstrap-tools | none | Builds the set of native tools needed by the vpp build system to build tool chains and images. Examples include "make", "git", "find", and "tar." |
| install-tools | PLATFORM | Builds the tool chain for the indicated <platform>. Often fails once in the middle of building glibc, simply issue "make PLATFORM=xxx install-tools" again. Complaints to the glibc maintainers, effectively "cat complaint > /dev/null". |
| distclean | none | Roto-rooters everything in sight: toolchains, images, and so forth. After issuing "make distclean", you'll need to rebuild the bootstrap-tools and the indicated platform toolchain. |
| wipe-all | PLATFORM=xxx [TAG=<tag>] | Recursively deletes .../vpp/build-root/build-<tag>-<cpu-arch> and .../vpp/build-root/install-<tag>-<cpu-arch>. The gold standard in recompiling all components listed in the <platform>_root_packages list. |
| <component>-find-source | none | Pull a source tree for <component> from the git server. Scan the list of repository groups, looking for .../build-data/packages/<component>.mk to decide which repo group will have the sources for the indicated <component>. Grubs around in .../vpp/build-root/.git/config to decide which git server / git protocol to use. |
| <component>-pull-all | none | Update <component> and all components upon which it depends from the git server. |
| <component>-install | PLATFORM and TAG | Build and install <component>, and all components upon which <component> depends. Example: to build a vanilla vpp forwarder, use "make PLATFORM=vpp TAG=debug vpp-install". Note the vpp-specific debug and non-debug tags defined in .../vpp/build-data/platforms/vpp.mk. |
| install-packages | PLATFORM and TAG | Build and install all components listed in <platform>_root_packages, use compile / link options defined by TAG. Example: to build the production vpp forwarder packages, use "make PLATFORM=vpp TAG=vpp install-packages" |
| install-deb | PLATFORM and TAG | Build a Debian package comprising components listed in <platform>_root_packages, using compile / link options defined by TAG. This is a .PHONY target. Issue "make ... install-packages" first to avoid accidentally building a brain-dead debian package. |
Additional Makefile environment variables
| Variable Setting | Notes |
|---|---|
| is_build_tool=yes | Directs Makefile to look in <repo-group-root>/build-root/packages/<component>.mk for instructions to build the indicated <component>. Results in .../vpp/build-root/{build,install}-tools. Related to setting TAG=xxx. Example: "make is_build_tool=yes apigen-install". |
| BUILD_DEBUG=vx | Directs Makefile et al. to make a good-faith effort to show what's going on in excruciating detail. Use it as follows: "make ... BUILD_DEBUG=vx". Fairly effective in Makefile debug situations. |
| strip_symbols=yes | Strip symbols from generated binaries prior to installation and packaging. Typically used with the "install-packages" and "install-deb" targets. |
| sign_executables=yes | Appends public-key encrypted SHA256 signatures to ELF binaries. Used when building embedded Linux systems. |
config.site
The .../vpp/build-root/config.site file contains a set of per-component configuration variable overrides. If you look at ../vpp/build-tool-x86_64/glibc/config.log as an example, you'll see something like this:
configure:2233: loading site script /scratch/nightly-builds/latest/vpp/build-root/config.site | # glibc needs this for cross compiling | libc_cv_forced_unwind=yes | libc_cv_c_cleanup=yes
You can see in the above example that the .../vpp/build-root/config.site file defines values for certain variables that glibc might need. Without those values, glibc might assign or assume other default values.
The benefit of defining settings in config.site is that it keeps specifc component "configure" scripts from trying to guess the values of specific variables. Those guesses (or assumptions) made by component scripts often lead to unpredictable results. In an ideal world, there would be no need to do this. The config.site file provides a more predictable alternative than trusting that component scripts are going to make the correct assumptions.
.../xxx/build-data
The set of build-data directories contains detailed build and packaging data for components which are defined in each repo group.
platforms.mk
Each repo group includes the platforms.mk file, which is included by the main Makefile. The vpp/build-data/platforms.mk file is not complex. The closed-repo/build-data/platforms.mk file accomplishes two tasks.
First, it includes vpp/build-data/platforms/*.mk:
$(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS), \ $(eval -include $(d)/platforms/*.mk))
This collects the set of platform definition makefile fragments, as discussed above.
Second, the platforms.mk file implements a set of make targets.
packages/*.mk
Each component needs a makefile fragment in order for the build system to recognize it. The per-component makefile fragments vary considerably in complexity. For a component built with GNU autoconf / automake which does not depend on other components, the make fragment can be empty. See .../vpp/build-data/packages/vpp.mk for an uncomplicated but fully realistic example.
Here are some of the important variable settings in per-component makefile fragments:
| Variable | Notes |
|---|---|
| xxx_configure_depend | Lists the set of component build dependencies for the xxx component. In plain English: don't try to configure this component until you've successfully built the indicated targets. Almost always, xxx_configure_depend will list a set of "yyy-install" targets. Note the pattern: "variable names contain underscores, make target names contain hyphens" |
| xxx_configure_args
(optional) |
Lists any additional arguments to pass to the xxx component "configure" script. The main Makefile %-configure rule adds the required settings for --libdir, --prefix, and --host (when cross-compiling, aka most of the time) |
| xxx_CPPFLAGS | Adds -I stanzas to CPPFLAGS for components upon which xxx depends. Almost invariably "xxx_CPPFLAGS = $(call installed_includes_fn, dep1 dep2 dep3)", where dep1, dep2, and dep3 are listed in xxx_configure_depend. It is bad practice to set "-g -O3" here. Those settings belong in a TAG. |
| xxx_LDFLAGS | Adds -Wl,-rpath -Wl,depN stanzas to LDFLAGS for components upon which xxx depends. Almost invariably "xxx_CPPFLAGS = $(call installed_lib_fn, dep1 dep2 dep3)", where dep1, dep2, and dep3 are listed in xxx_configure_depend. It is bad manners to set "-liberty-or-death" here. Those settings belong in Makefile.am. |
When dealing with "irritating" components built with raw Makefiles which only work when building in the source trees, we use a specific strategy in the xxx.mk file. Examples include the Intel / 6wind DPDK.
The strategy is simple for those "irritating" components: We copy the source tree into .../vpp/build-root/build-xxx. This works, but completely defeats dependency processing. This strategy is acceptable only for 3rd party software which won't need extensive (or preferably any) modifications.
Take a look at .../vpp/build-data/packages/dpdk.mk. When invoked, the dpdk_configure variable copies source code into $(PACKAGE_BUILD_DIR), and performs the BSD equivalent of "autoreconf -i -f" to configure the build area. The rest of the file is similar: a bunch of hand-rolled glue code which manages to make the dpdk act like a good vpp build citizen even though it is not. There are worse examples, notably the historical tail-f "confd" integration.
per-component GNU autotools input files
This section is not an introduction to the GNU autotools. If you are unfamiliar with the GNU autotools you can: stick to programming-by-example, or spend time learning about autotools. A bit of both seems the likely way to get started. There is a lot of information online from various sites on how to use GNU autotools. We recommend that you look at the canonical reference on gnu.org for autotools introduction.
<repo>/<component>/configure.ac
The configure.ac file directs the GNU autotools to generate the "configure" script which the main Makefile invokes through the xxx-configure target.
The configure script receives the set of options specified in <repo>/build-data/packages/<component>/<component>.mk> as specified by the xxx_configure_args argument. When cross-compiling, this is in addition to the arguments --libdir, --prefix, and --host.
Here is short but reasonable example:
AC_INIT(foo, 1.1) AM_INIT_AUTOMAKE AC_PROG_LIBTOOL AM_PROG_AS AC_PROG_CC AM_PROG_CC_C_O AC_OUTPUT([Makefile])
In rough terms, this says "please build a configure script for version 1.1 of the foo component. Make sure that you can find gcc, gas, and gld. The component may build a dynamic library in addition to a command binary."
For an uncomplicated new component, the above configuration is about all you need to specify.
For a more sophisticated example, look at the .../vpp/vpp/configure.ac file. The settings in that file set automake conditional variables (used to control the set of files compiled). It is also arranged to add some -Dfoo variable settings to CFLAGS.
In principle, less is more. The more tools, libraries, etc. that you list in the configure.ac file, the slower the resulting configure script will run. The vpp build systems is ccache-enabled, timestamp enabled, and tuned to the point where the build system rate-limiter is the amount of time spent running "configure" scripts.
