VPP/Build System Deep Dive
Contents
Introduction
The vpp build system comprises a data-driven main makefile, and a set of makefile fragments. All of hangs together nicely, the result of a set of well-thought-out conventions.
The vpp build system stitches together sets of git "repository groups." In a typical product development case, one uses two repository groups: an "open-repo" (including the build system itself) and a "closed-repo" comprising vendor-specific closed source. Each repository group comprises a number of independent git repositories.
Current open-vpp workspaces comprise a single repo group, ordinarily called "vpp". Vpp/build-root/build-config.mk defines a key variable - SOURCE_PATH - which names the set of repo groups.
The vpp build system caters to components built with GNU autoconf / automake. Adding such components is a snap. Dealing with components which use BSD-style raw Makefiles is a bit of a pain. Dealing with toolchain components such as GCC and GLIBC can be seriously painful.
The vpp build system is a single-pass build system. A partial order must exist for any set of components: the set of (a before b) tuples must resolve to an ordered list. If you create a circular dependency of the form; (a,b) (b,c) (c,a), gmake will try to build the target list, but there’s a 0.0% chance that the results will be pleasant. Cut-n-paste mistakes in .../build-data/packages/<oops>.mk can produce the most confusing failures imaginable.
In a single-pass build system, it’s best to separate libraries and applications which instantiate them. If vpp depends on libfoo.a, and myapp depends on both vpp and libfoo.a, place libfoo.a and myapp in separate components. The build system will build libfoo.a, vpp, and then myapp. If you try to build libfoo.a and myapp from the same component, it won’t work.
If you absolutely, positively insist on having myapp and libfoo.a in the same source tree, you can create a pseudo-component in a separate .mk file in .../build-data/packages/. Code “phoneycomponent_source = realcomponent”, and provide manual configure/build/install targets.
Separate components for myapp, libfoo.a, and vpp are the gold-standard cheap / easy solution. The “mumble_source = realsource” degree of freedom exists to solve intractable circular dependencies: to build gcc-bootstrap, followed by glibc, followed by “real” gcc/g++ [which depend on glibc].
.../vpp/build-root
.../vpp/build-root contains the repository group specification build-config.mk, the main Makefile, and the system-wide set of autoconf/automake variable overrides in config.site. We'll describe these files in some detail. To level-set: the main Makefile and config.site are subtle and complex. It's unlikely that you'll need or want to modify them. Ill-considered changes in either place typically cause difficult bugs.
.../vpp/build-root/build-config.mk
As described above, build-config.mk is completely straightforward: it sets the make variable SOURCE_PATH to a list of repository group absolute paths. If you choose to move a workspace, you must modify it to match "reality 2.0." For example:
SOURCE_PATH = /home/joeuser/workspace/vpp
.../vpp/build-root/Makefile
The main Makefile is complex. If you think you need to modify it, please ask for advice before you invest or send money. Seriously.
Pleasant characteristics include: excellent performance, absolutely accurate dependency processing, ccache enablement, timestamp optimizations, git integration, extensibility, builds cross-compilation tool chains, builds embedded Linux distributions. If you need to do so, you can build double-cross tools: compile gdb on x86_64, to run on PowerPC, to debug the Xtensa instruction set.
Unpleasant characteristic: you'll break it if you look cross-eyed at it.
PLATFORM variable
The PLATFORM make / environment variable controls a number of important image characteristics, primarily: cpu architecture, the list of images to build via the "make PLATFORM=xxx install-packages" target. See below for a description of that target. In the vpp case, we use "PLATFORM=vpp".
The main Makefile interprets $PLATFORM by attempting to "-include" <repo-group-root>/build-data/platforms.mk:
$(foreach d,$(FULL_SOURCE_PATH), \ $(eval -include $(d)/platforms.mk))
By convention, we don't define platforms in .../<repo-group-root>/build-data/platforms.mk. In the vpp case, we search for platform definition makefile fragments in .../vpp/build-data/platforms.mk, as follows:
$(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS), \ $(eval -include $(d)/platforms/*.mk))
With vpp, which uses the "vpp" platform as discussed above, we end up "-include"ing .../vpp/build-data/platforms/vpp.mk. With a bunch of secondary "exercise for the reader" variable settings omitted, here is the contents of vpp.mk:
# vector packet processor vpp_arch = native vpp_native_tools = vppapigen vppversion vpp_root_packages = vpp vlib vlib-api vnet svm dpdk vpp-api-test \ vpp-japi vpp_configure_args_vpp = --with-dpdk vnet_configure_args_vpp = --with-dpdk # Set these parameters carefully. The vlib_buffer_t is 128 bytes, i.e. # dpdk_headroom = uiotarball_headroom = vlib_pre_data + 128 dpdk_configure_args_vpp = --with-headroom=256 vlib_configure_args_vpp = --with-pre-data=128 vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector-all -fPIC vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector-all -fPIC vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector -fPIC -pie vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ -fstack-protector -fPIC -pie
The variable <platform-name>_arch sets the CPU architecture used to build the per-platform cross-compilation toolchain. With the exception of the "native" architecture - used in our example - the vpp build system produces cross-compiled binaries.
The variable <platform-name>_native_tools lists the required set of self-compiled build tools.
The variable <platform-name>_root_packages lists the set of images to build via the "make PLATFORM=<platform-name> TAG=<tag-name> [install-deb | install-rpm]" target.
TAG variable
The TAG variable indirectly sets CFLAGS and LDFLAGS, as well as the build and install directory names in .../vpp/build-root. Here is the current "vpp" tag definition from .../vpp/build-data/platforms/vpp.mk:
Executables built with PLATFORM=vpp TAG=vpp are built in .../vpp/build-root/build-vpp-x86_64, with executables installed in .../vpp/build-root/install-vpp-x86_64
Makefile important targets
The main Makefile and the various makefile fragments implement the following user-invoked targets:
| Target | Environment Variable Settings | Notes |
|---|---|---|
| bootstrap-tools | none | Builds the set of native tools needed by the vpp build system to build tool chains and images. Examples include "make", "git", "find", and "tar." |
| install-tools | PLATFORM | Builds the tool chain for the indicated <platform>. Often fails once in the middle of building glibc, simply issue "make PLATFORM=xxx install-tools" again. Complaints to the glibc maintainers, effectively "cat complaint > /dev/null". |
| distclean | none | Roto-rooters everything in sight: toolchains, images, and so forth. After issuing "make distclean", you'll need to rebuild the bootstrap-tools and the indicated platform toolchain. |
| wipe-all | PLATFORM=xxx [TAG=<tag>] | Recursively deletes .../vpp/build-root/build-<tag>-<cpu-arch> and .../vpp/build-root/install-<tag>-<cpu-arch>. The gold standard in recompiling all components listed in the <platform>_root_packages list. |
| <component>-find-source | none | Pull a source tree for <component> from the git server. Scan the list of repository groups, looking for .../build-data/packages/<component>.mk to decide which repo group will have the sources for the indicated <component>. Grubs around in .../vpp/build-root/.git/config to decide which git server / git protocol to use. |
| <component>-pull-all | none | Update <component> and all components upon which it depends from the git server. |
| <component>-install | PLATFORM and TAG | Build and install <component>, and all components upon which <component> depends. Example: to build a vanilla vpp forwarder, use "make PLATFORM=vpp TAG=debug vpp-install". Note the vpp-specific debug and non-debug tags defined in .../vpp/build-data/platforms/vpp.mk. |
| install-packages | PLATFORM and TAG | Build and install all components listed in <platform>_root_packages, use compile / link options defined by TAG. Example: to build the production vpp forwarder packages, use "make PLATFORM=vpp TAG=vpp install-packages" |
| install-deb | PLATFORM and TAG | Build a Debian package comprising components listed in <platform>_root_packages, using compile / link options defined by TAG. This is a .PHONY target. Issue "make ... install-packages" first to avoid accidentally building a brain-dead debian package. |
Additional Makefile environment variables
| Variable Setting | Notes |
|---|---|
| is_build_tool=yes | Directs Makefile to look in <repo-group-root>/build-root/packages/<component>.mk for instructions to build the indicated <component>. Results in .../vpp/build-root/{build,install}-tools. Related to setting TAG=xxx. Example: "make is_build_tool=yes apigen-install". |
| BUILD_DEBUG=vx | Directs Makefile et al. to make a good-faith effort to show what's going on in excruciating detail. Use it as follows: "make ... BUILD_DEBUG=vx". Fairly effective in Makefile debug situations. |
| strip_symbols=yes | Strip symbols from generated binaries prior to installation and packaging. Typically used with the "install-packages" and "install-deb" targets. |
| sign_executables=yes | Appends public-key encrypted SHA256 signatures to ELF binaries. Used when building embedded Linux systems. |
config.site
.../vpp/build-root/config.site contains a set of per-component configuration variable overrides. If you look at e.g. ../vpp/build-tool-x86_64/glibc/config.log, you'll see something like this:
configure:2233: loading site script /scratch/nightly-builds/latest/vpp/build-root/config.site | # glibc needs this for cross compiling | libc_cv_forced_unwind=yes | libc_cv_c_cleanup=yes
and so forth.
Settings in config.site keep specifc component "configure" scripts from trying to guess the setting of specific variables which they always screw up. In an ideal world, there would be no need to do this. People who love sausage and respect the law should never watch either one being made.
.../xxx/build-data
The set of build-data directories contains detailed build and packaging data for components which are defined in each repo group.
platforms.mk
Each repo group includes platforms.mk, which are included by the main Makefile. vpp/build-data/platforms.mk is not interesting. closed-repo/build-data/platforms.mk accomplishes two tasks. First, it includes vpp/build-data/platforms/*.mk:
$(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS), \ $(eval -include $(d)/platforms/*.mk))
This collects the set of platform definition makefile fragments, as discussed above.
Second, platforms.mk implements a set of make targets. Certain of these targets could move to the main Makefile at some point in the indefinite future.
packages/*.mk
Each component needs a makefile fragment e.g. so the build system recognizes it. The per-component makefile fragments vary considerably in complexity. For a component built with GNU autoconf / automake which does not depend on other components, the make fragment can be empty. See .../vpp/build-data/packages/vpp.mk for an uncomplicated but fully realistic example.
Important variable settings in per-component makefile fragments:
| Variable | Notes |
|---|---|
| xxx_configure_depend | Lists the set of component build dependencies for the xxx component. In plain English: don't try to configure this component until you've successfully built the indicated targets. Almost always, xxx_configure_depend will list a set of "yyy-install" targets. Note the pattern: "variable names contain underscores, make target names contain hyphens" |
| xxx_configure_args
(optional) |
Lists any additional arguments to pass to the xxx component "configure" script. The main Makefile %-configure rule adds the required settings for --libdir, --prefix, and --host (when cross-compiling, aka most of the time) |
| xxx_CPPFLAGS | Adds -I stanzas to CPPFLAGS for components upon which xxx depends. Almost invariably "xxx_CPPFLAGS = $(call installed_includes_fn, dep1 dep2 dep3)", where dep1, dep2, and dep3 are listed in xxx_configure_depend. It is bad manners to set "-g -O3" here. Those settings belong in a TAG. |
| xxx_LDFLAGS | Adds -Wl,-rpath -Wl,depN stanzas to LDFLAGS for components upon which xxx depends. Almost invariably "xxx_CPPFLAGS = $(call installed_lib_fn, dep1 dep2 dep3)", where dep1, dep2, and dep3 are listed in xxx_configure_depend. It is bad manners to set "-liberty-or-death" here. Those settings belong in Makefile.am. |
When dealing with "irritating" components built with raw Makefiles which only work when building in the source trees, we use a specific strategy in xxx.mk. Examples include the Intel / 6wind DPDK. The strategy is simple: "OK, be that way. We simply copy the source tree into .../vpp/build-root/build-xxx." This works, but completely defeats dependency processing. This strategy is acceptable only for 3rd party software which won't need extensive (or preferably any) modifications.
Take a look at .../vpp/build-data/packages/dpdk.mk. When invoked, the dpdk_configure variable copies source code into $(PACKAGE_BUILD_DIR), and performs the BSD equivalent of "autoreconf -i -f" to configure the build area. The rest of the file is similar: a bunch of hand-rolled glue code which manages to make the dpdk act like a good vpp build citizen even though it is not. There are worse examples, notably the historical tail-f "confd" integration.
per-component GNU autotools input files
This section is not an introduction to the GNU autotools. Folks unfamiliar with the GNU autotools have two choices: stick to programming-by-example, or spend time learning about autotools. A bit of both seems like the way to go. Google is your friend. Here's a pointer to the canonical introduction.
<repo>/<component>/configure.ac
Configure.ac directs the GNU autotools to generate the "configure" script which the main Makefile invokes via its "xxx-configure" target.
The configure scripts receives the set of options specified in <repo>/build-data/packages/<component>/<component>.mk> via xxx_configure_args, in addition to --libdir, --prefix, and --host (when cross-compiling).
A minimal reasonable example:
AC_INIT(foo, 1.1) AM_INIT_AUTOMAKE AC_PROG_LIBTOOL AM_PROG_AS AC_PROG_CC AM_PROG_CC_C_O AC_OUTPUT([Makefile])
In rough terms, this says "please build a configure script for version 1.1 of the foo component. Make sure that you can find gcc, gas, gld. The component may build a dynamic library in addition to a command binary." For an uncomplicated new component, this is about all you need to specify.
See .../vpp/vpp/configure.ac for a more sophisticated example, which sets automake conditional variables (used to control the set of files compiled) and arranged to add some -Dfoo variable settings to CFLAGS.
Less is more. The more tools, libraries, etc. you list in configure.ac, the slower the resulting configure script will run. The vpp build systems is ccache-enabled, timestamp enabled, and tuned to the point where the build system rate-limiter is the amount of time spent running "configure" scripts.
