Difference between revisions of "VPP/Build System Deep Dive"

Latest revision as of 22:28, 28 February 2020

DEPRECATION NOTICE

This page is no longer maintained. Please see the [VPP Build System Documentation]

Introduction

The vpp build system consists of a data-driven main makefile, and a set of makefile fragments. The various parts come together as the result of a set of well-thought-out conventions.

Repository Groups and Source Paths

The vpp build system brings together sets of git "repository groups." In a typical product development case, one would use two repository groups: a group that we will call for the purpose of this explanation "open-repo" (including the build system itself) and a group that we will call "closed-repo" comprising vendor-specific closed source. Each repository group consists of a number of independent git repositories.

The current vpp workspaces comprise a single repository group, ordinarily called "vpp". The file vpp/build-root/build-config.mk defines a key variable called SOURCE_PATH. The SOURCE_PATH variable names the set of repository groups. We recommend that you examine that file to understand the repositories that are being built.

Dependencies and Components

The vpp build system caters to components built with GNU autoconf / automake. Adding such components is a simple process. Dealing with components which use BSD-style raw Makefiles is a bit more difficult. Dealing with toolchain components such as GCC and GLIBC can be considerably more complicated.

The vpp build system is a single-pass build system. A partial order must exist for any set of components: the set of (a before b) tuples must resolve to an ordered list. If you create a circular dependency of the form; (a,b) (b,c) (c,a), gmake will try to build the target list, but there’s a 0.0% chance that the results will be pleasant. Cut-n-paste mistakes in .../build-data/packages/<oops>.mk can produce confusing failures.

In a single-pass build system, it’s best to separate libraries and applications which instantiate them. For example, if vpp depends on libfoo.a, and myapp depends on both vpp and libfoo.a, it's best to place libfoo.a and myapp in separate components. The build system will build libfoo.a, vpp, and then (as a separate component) myapp. If you try to build libfoo.a and myapp from the same component, it won’t work.

If you absolutely, positively insist on having myapp and libfoo.a in the same source tree, you can create a pseudo-component in a separate .mk file in the .../build-data/packages/ directory. Define the code phoneycomponent_source = realcomponent, and provide manual configure/build/install targets.

Separate components for myapp, libfoo.a, and vpp is the best and easiest solution. However, the “mumble_source = realsource” degree of freedom exists to solve intractable circular dependencies, such as: to build gcc-bootstrap, followed by glibc, followed by “real” gcc/g++ [which depends on glibc too].

.../vpp/build-root

The .../vpp/build-root directory contains the repository group specification build-config.mk, the main Makefile, and the system-wide set of autoconf/automake variable overrides in config.site. We'll describe these files in some detail. To be clear about expectations, the main Makefile and config.site file are subtle and complex. It's unlikely that you'll need or want to modify them. Poorly planned changes in either place typically cause bugs that are difficult to solve.

.../vpp/build-root/build-config.mk

As described above, the build-config.mk file is straightforward: it sets the make variable SOURCE_PATH to a list of repository group absolute paths.

The SOURCE_PATH variable

If you choose to move a workspace, make sure to modify the paths defined by the SOURCE_PATH variable. Those paths need to match changes you make in the workspace paths. For example, if you place the vpp directory in the workspace of a user named jsmith, you might need to change the SOURCE_PATH to:

SOURCE_PATH = /home/jsmithuser/workspace/vpp

.../vpp/build-root/Makefile

The main Makefile is complex. If you think you need to modify it, it's a good idea to do some research, or ask for advice before you change it.

The main Makefile was organized and designed to provide the following characteristics: excellent performance, absolutely accurate dependency processing, cache enablement, timestamp optimizations, git integration, extensibility, builds with cross-compilation tool chains, and builds with embedded Linux distributions.

If you need to do so, you can build double-cross tools. For example, you could: compile gdb on x86_64, to run on PowerPC, to debug the Xtensa instruction set.

The PLATFORM variable

The PLATFORM make/environment variable controls a number of important characteristics, primarily:

• cpu architecture, and
• the list of images to build.

The list of images to build is specified by the target: make PLATFORM=xxx install-packages. (See below for a description of that target.)

In the vpp case, we use "PLATFORM=vpp".

The main Makefile interprets $PLATFORM by attempting to -include the file <repo-group-root>/build-data/platforms.mk:

$(foreach d,$(FULL_SOURCE_PATH), \
  $(eval -include $(d)/platforms.mk))

By convention, we don't define platforms in the .../<repo-group-root>/build-data/platforms.mk file. In the vpp case, we search for platform definition makefile fragments in .../vpp/build-data/platforms.mk, as follows:

$(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS),	\
  $(eval -include $(d)/platforms/*.mk))

With vpp, which uses the "vpp" platform as discussed above, we end up "-include"-ing .../vpp/build-data/platforms/vpp.mk. To make it easier to read here in this page, the following snippet omits several variable settings. Here are some of the contents of vpp.mk:

# vector packet processor
vpp_arch = native
vpp_native_tools = vppapigen vppversion
 
vpp_root_packages = vpp vlib vlib-api vnet svm dpdk vpp-api-test \ 
	vpp-japi 
 
vpp_configure_args_vpp = --with-dpdk
vnet_configure_args_vpp = --with-dpdk

# Set these parameters carefully. The vlib_buffer_t is 128 bytes, i.e.
# dpdk_headroom = uiotarball_headroom = vlib_pre_data + 128
dpdk_configure_args_vpp = --with-headroom=256
vlib_configure_args_vpp = --with-pre-data=128

vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \
  	-fstack-protector-all -fPIC
vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \  
	-fstack-protector-all -fPIC
  
vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
	-fstack-protector -fPIC -pie
vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
	-fstack-protector -fPIC -pie

The variable <platform-name>_arch sets the CPU architecture used to build the per-platform cross-compilation toolchain. With the exception of the "native" architecture - used in our example - the vpp build system produces cross-compiled binaries.

The variable <platform-name>_native_tools lists the required set of self-compiled build tools.

The variable <platform-name>_root_packages lists the set of images to build when specifying the target: make PLATFORM=<platform-name> TAG=<tag-name> [install-deb | install-rpm].

The TAG variable

The TAG variable indirectly sets CFLAGS and LDFLAGS, as well as the build and install directory names in the .../vpp/build-root directory. Here is the current "vpp" tag definition specified in the .../vpp/build-data/platforms/vpp.mk file:

vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \
  	-fstack-protector-all -fPIC
vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \  
	-fstack-protector-all -fPIC
  
vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
	-fstack-protector -fPIC -pie
vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
	-fstack-protector -fPIC -pie

Executables built with PLATFORM=vpp TAG=vpp are built in .../vpp/build-root/build-vpp-x86_64, with executables installed in .../vpp/build-root/install-vpp-x86_64.

Important targets in the Makefile

The main Makefile and the various makefile fragments implement the following user-invoked targets:

Additional Makefile environment variables

config.site

The .../vpp/build-root/config.site file contains a set of per-component configuration variable overrides. If you look at ../vpp/build-tool-x86_64/glibc/config.log as an example, you'll see something like this:

configure:2233: loading site script /scratch/nightly-builds/latest/vpp/build-root/config.site
| # glibc needs this for cross compiling 
| libc_cv_forced_unwind=yes
| libc_cv_c_cleanup=yes

You can see in the above example that the .../vpp/build-root/config.site file defines values for certain variables that glibc might need. Without those values, glibc might assign or assume other default values.

The benefit of defining settings in config.site is that it keeps specifc component "configure" scripts from trying to guess the values of specific variables. Those guesses (or assumptions) made by component scripts often lead to unpredictable results. In an ideal world, there would be no need to do this. The config.site file provides a more predictable alternative than trusting that component scripts are going to make the correct assumptions.

.../xxx/build-data

The set of build-data directories contains detailed build and packaging data for components which are defined in each repo group.

platforms.mk

Each repo group includes the platforms.mk file, which is included by the main Makefile. The vpp/build-data/platforms.mk file is not complex. The closed-repo/build-data/platforms.mk file accomplishes two tasks.

First, it includes vpp/build-data/platforms/*.mk:

$(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS),	\
  $(eval -include $(d)/platforms/*.mk))

This collects the set of platform definition makefile fragments, as discussed above.

Second, the platforms.mk file implements a set of make targets.

packages/*.mk

Each component needs a makefile fragment in order for the build system to recognize it. The per-component makefile fragments vary considerably in complexity. For a component built with GNU autoconf / automake which does not depend on other components, the make fragment can be empty. See .../vpp/build-data/packages/vpp.mk for an uncomplicated but fully realistic example.

Here are some of the important variable settings in per-component makefile fragments:

When dealing with "irritating" components built with raw Makefiles which only work when building in the source trees, we use a specific strategy in the xxx.mk file. Examples include the Intel / 6wind DPDK.

The strategy is simple for those "irritating" components: We copy the source tree into .../vpp/build-root/build-xxx. This works, but completely defeats dependency processing. This strategy is acceptable only for 3rd party software which won't need extensive (or preferably any) modifications.

Take a look at .../vpp/build-data/packages/dpdk.mk. When invoked, the dpdk_configure variable copies source code into $(PACKAGE_BUILD_DIR), and performs the BSD equivalent of "autoreconf -i -f" to configure the build area. The rest of the file is similar: a bunch of hand-rolled glue code which manages to make the dpdk act like a good vpp build citizen even though it is not. There are worse examples, notably the historical tail-f "confd" integration.

per-component GNU autotools input files

This section is not an introduction to the GNU autotools. If you are unfamiliar with the GNU autotools you can: stick to programming-by-example, or spend time learning about autotools. A bit of both seems the likely way to get started. There is a lot of information online from various sites on how to use GNU autotools. We recommend that you look at the canonical reference on gnu.org for autotools introduction.

<repo>/<component>/configure.ac

The configure.ac file directs the GNU autotools to generate the "configure" script which the main Makefile invokes through the xxx-configure target.

The configure script receives the set of options specified in <repo>/build-data/packages/<component>/<component>.mk> as specified by the xxx_configure_args argument. When cross-compiling, this is in addition to the arguments --libdir, --prefix, and --host.

Here is short but reasonable example:

AC_INIT(foo, 1.1)
AM_INIT_AUTOMAKE

AC_PROG_LIBTOOL
AM_PROG_AS
AC_PROG_CC
AM_PROG_CC_C_O 

AC_OUTPUT([Makefile])

In rough terms, this says "please build a configure script for version 1.1 of the foo component. Make sure that you can find gcc, gas, and gld. The component may build a dynamic library in addition to a command binary."

For an uncomplicated new component, the above configuration is about all you need to specify.

For a more sophisticated example, look at the .../vpp/vpp/configure.ac file. The settings in that file set automake conditional variables (used to control the set of files compiled). It is also arranged to add some -Dfoo variable settings to CFLAGS.

In principle, less is more. The more tools, libraries, etc. that you list in the configure.ac file, the slower the resulting configure script will run. The vpp build systems is ccache-enabled, timestamp enabled, and tuned to the point where the build system rate-limiter is the amount of time spent running "configure" scripts.