View source for VPP/Build System Deep Dive

__TOC__

== Introduction ==

The vpp build system consists of a data-driven main makefile, and a set of makefile fragments. The various parts come together as the result of a set of well-thought-out conventions. 

==== Repository Groups and Source Paths ====

The vpp build system brings together sets of git "repository groups." In a typical product development case, one would use two repository groups: a group that we will call for the purpose of this explanation "open-repo" (including the build system itself) and a group that we will call "closed-repo" comprising vendor-specific closed source. Each repository group consists of a number of independent git repositories.

The current vpp workspaces comprise a single repository group, ordinarily called "vpp". The file <tt>vpp/build-root/build-config.mk</tt> defines a key variable called <code>SOURCE_PATH</code>. The <code>SOURCE_PATH</code> variable names the set of repository groups. We recommend that you examine that file to understand the repositories that are being built.

==== Dependencies and Components ====

The vpp build system caters to components built with GNU autoconf / automake. Adding such components is a simple process. Dealing with components which use BSD-style raw Makefiles is a bit more difficult. Dealing with toolchain components such as GCC and GLIBC can be considerably more complicated.

The vpp build system is a single-pass build system. A partial order must exist for any set of components: the set of (a before b) tuples must resolve to an ordered list. If you create a circular dependency of the form; (a,b) (b,c) (c,a), gmake will try to build the target list, but there’s a 0.0% chance that the results will be pleasant. Cut-n-paste mistakes in <tt>.../build-data/packages/<oops>.mk</tt> can produce confusing failures. 

In a single-pass build system, it’s best to separate libraries and applications which instantiate them. For example, if vpp depends on <tt>libfoo.a</tt>, and myapp depends on both vpp and <tt>libfoo.a</tt>, it's best to place <tt>libfoo.a</tt> and myapp in separate components. The build system will build <tt>libfoo.a</tt>, vpp, and then (as a separate component) myapp. If you try to build <tt>libfoo.a</tt> and myapp from the same component, it won’t work.

If you absolutely, positively insist on having myapp and <tt>libfoo.a</tt> in the same source tree, you can create a pseudo-component in a separate <tt>.mk</tt> file in the <tt>.../build-data/packages/</tt> directory. Define the code <code>phoneycomponent_source = realcomponent</code>, and provide manual configure/build/install targets. 

Separate components for myapp, <tt>libfoo.a</tt>, and vpp is the best and easiest solution. However, the “mumble_source = realsource” degree of freedom exists to solve intractable circular dependencies, such as: to build gcc-bootstrap, followed by glibc, followed by “real” gcc/g++ [which depends on glibc too].

== .../vpp/build-root ==

The .../vpp/build-root directory contains the repository group specification <tt>build-config.mk</tt>, the main Makefile, and the system-wide set of autoconf/automake variable overrides in <tt>config.site</tt>. We'll describe these files in some detail. To be clear about expectations, the main Makefile and <tt>config.site</tt> file are subtle and complex. It's unlikely that you'll need or want to modify them. Poorly planned changes in either place typically cause bugs that are difficult to solve.

=== .../vpp/build-root/build-config.mk ===

As described above, the <tt>build-config.mk</tt> file is straightforward: it sets the make variable <code>SOURCE_PATH</code> to a list of repository group absolute paths. 

==== The SOURCE_PATH variable ====
If you choose to move a workspace, make sure to modify the paths defined by the <code>SOURCE_PATH</code> variable.  Those paths need to match changes you make in the workspace paths. For example, if you place the vpp directory in the workspace of a user named jsmith, you might need to change the <code>SOURCE_PATH</code> to:

 SOURCE_PATH = /home/jsmithuser/workspace/vpp

=== .../vpp/build-root/Makefile ===

The main Makefile is complex. If you think you need to modify it, it's a good idea to do some research, or ask for advice before you change it. 

The main Makefile was organized and designed to provide the following characteristics: excellent performance, absolutely accurate dependency processing, cache enablement, timestamp optimizations, git integration, extensibility, builds with cross-compilation tool chains, and builds with embedded Linux distributions. 

If you need to do so, you can build double-cross tools. For example, you could: compile gdb on x86_64, to run on PowerPC, to debug the Xtensa instruction set. 


==== The PLATFORM variable ====

The <code>PLATFORM</code> make/environment variable controls a number of important characteristics, primarily: 

* cpu architecture, and
* the list of images to build.

The list of images to build is specified by the target: <code>make PLATFORM=xxx install-packages</code>. (See below for a description of that target.) 

In the vpp case, we use <code>"PLATFORM=vpp"</code>. 

The main Makefile interprets <code>$PLATFORM</code> by attempting to <code>-include</code> the file <tt><repo-group-root>/build-data/platforms.mk</tt>:

 $(foreach d,$(FULL_SOURCE_PATH), \
   $(eval -include $(d)/platforms.mk))

By convention, we don't define platforms in the <tt>.../<repo-group-root>/build-data/platforms.mk</tt> file. In the vpp case, we search for platform definition makefile fragments in <tt>.../vpp/build-data/platforms.mk</tt>, as follows:

 $(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS),	\
   $(eval -include $(d)/platforms/*.mk))

With vpp, which uses the "vpp" platform as discussed above, we end up "-include"-ing <tt>.../vpp/build-data/platforms/vpp.mk</tt>. To make it easier to read here in this page, the following snippet omits several variable settings. Here are some of the contents of <tt>vpp.mk</tt>:

 # vector packet processor
 vpp_arch = native
 vpp_native_tools = vppapigen vppversion
  
 vpp_root_packages = vpp vlib vlib-api vnet svm dpdk vpp-api-test \ 
 	vpp-japi 
  
 vpp_configure_args_vpp = --with-dpdk
 vnet_configure_args_vpp = --with-dpdk
 
 # Set these parameters carefully. The vlib_buffer_t is 128 bytes, i.e.
 # dpdk_headroom = uiotarball_headroom = vlib_pre_data + 128
 dpdk_configure_args_vpp = --with-headroom=256
 vlib_configure_args_vpp = --with-pre-data=128
 
 vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \
   	-fstack-protector-all -fPIC
 vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \  
 	-fstack-protector-all -fPIC
   
 vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
 	-fstack-protector -fPIC -pie
 vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
 	-fstack-protector -fPIC -pie

The variable <code><platform-name>_arch</code> sets the CPU architecture used to build the per-platform cross-compilation toolchain. With the exception of the "native" architecture - used in our example - the vpp build system produces cross-compiled binaries. 

The variable <code><platform-name>_native_tools</code> lists the required set of self-compiled build tools.

The variable <code><platform-name>_root_packages</code> lists the set of images to build when specifying the target: <code>make PLATFORM=<platform-name> TAG=<tag-name> [install-deb | install-rpm]</code>.


==== The TAG variable ====

The TAG variable indirectly sets <tt>CFLAGS</tt> and <tt>LDFLAGS</tt>, as well as the build and install directory names in the <tt>.../vpp/build-root</tt> directory. Here is the current "vpp" tag definition specified in the  <tt>.../vpp/build-data/platforms/vpp.mk</tt> file:

 vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \
   	-fstack-protector-all -fPIC
 vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 -march=$(MARCH) \  
 	-fstack-protector-all -fPIC
   
 vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
 	-fstack-protector -fPIC -pie
 vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -march=$(MARCH) \ 
 	-fstack-protector -fPIC -pie

Executables built with <code>PLATFORM=vpp TAG=vpp</code> are built in <tt>.../vpp/build-root/build-vpp-x86_64</tt>, with executables installed in <tt>.../vpp/build-root/install-vpp-x86_64</tt>.


==== Important targets in the Makefile ====

The main Makefile and the various makefile fragments implement the following user-invoked targets:

{| class="wikitable" border="1"
|-
! Target
! Environment Variable Settings
! Notes
|-
| bootstrap-tools
| none
| Builds the set of native tools needed by the vpp build system to build tool chains and images. Examples include "make", "git", "find", and "tar."
|-
| install-tools
| PLATFORM
| Builds the tool chain for the indicated <platform>. Often fails once in the middle of building glibc, simply issue "make PLATFORM=xxx install-tools" again. Complaints to the glibc maintainers, effectively "cat complaint > /dev/null".
|-
| distclean
| none
| Roto-rooters everything in sight: toolchains, images, and so forth. After issuing "make distclean", you'll need to rebuild the bootstrap-tools and the indicated platform toolchain.
|-
| wipe-all
| PLATFORM=xxx [TAG=<tag>]
| Recursively deletes .../vpp/build-root/build-<tag>-<cpu-arch> and .../vpp/build-root/install-<tag>-<cpu-arch>. The gold standard in recompiling all components listed in the <platform>_root_packages list.
|-
|style=white-space:nowrap|<component>-find-source
| none
| Pull a source tree for <component> from the git server. Scan the list of repository groups, looking for .../build-data/packages/<component>.mk to decide which repo group will have the sources for the indicated <component>. Grubs around in .../vpp/build-root/.git/config to decide which git server / git protocol to use.
|-
| <component>-pull-all
| none
| Update <component> and all components upon which it depends from the git server.
|- 
| <component>-install
| PLATFORM and TAG
| Build and install <component>, and all components upon which <component> depends. Example: to build a vanilla vpp forwarder, use "make PLATFORM=vpp TAG=debug vpp-install". Note the vpp-specific debug and non-debug tags defined in .../vpp/build-data/platforms/vpp.mk.
|-
| install-packages
| PLATFORM and TAG
| Build and install all components listed in <platform>_root_packages, use compile / link options defined by TAG. Example: to build the production vpp forwarder packages, use "make PLATFORM=vpp TAG=vpp install-packages"
|-
| install-deb
| PLATFORM and TAG
| Build a Debian package comprising components listed in <platform>_root_packages, using compile / link options defined by TAG. This is a .PHONY target. Issue "make ... install-packages" first to avoid accidentally building a brain-dead debian package.     
|}

==== Additional Makefile environment variables ====

{| class="wikitable" border="1"
|-
! Variable Setting
! Notes
|-
| is_build_tool=yes
| Directs Makefile to look in <repo-group-root>/build-'''root'''/packages/<component>.mk for instructions to build the indicated <component>. Results in .../vpp/build-root/{build,install}-tools. Related to setting TAG=xxx. Example: "make is_build_tool=yes apigen-install".
|-
| BUILD_DEBUG=vx
| Directs Makefile et al. to make a good-faith effort to show what's going on in excruciating detail. Use it as follows: "make ... BUILD_DEBUG=vx". Fairly effective in Makefile debug situations.
|-
| strip_symbols=yes
| Strip symbols from generated binaries prior to installation and packaging. Typically used with the "install-packages" and "install-deb" targets.
|-
| sign_executables=yes
| Appends public-key encrypted SHA256 signatures to ELF binaries. Used when building embedded Linux systems.
|}

=== config.site ===

The <tt>.../vpp/build-root/config.site</tt> file contains a set of per-component configuration variable overrides. If you look at <tt>../vpp/build-tool-x86_64/glibc/config.log</tt> as an example, you'll see something like this:

 configure:2233: loading site script /scratch/nightly-builds/latest/vpp/build-root/config.site
 | # glibc needs this for cross compiling 
 | libc_cv_forced_unwind=yes
 | libc_cv_c_cleanup=yes


You can see in the above example that the <tt>.../vpp/build-root/config.site</tt> file defines values for certain variables that <tt>glibc</tt> might need. Without those values, <tt>glibc</tt> might assign or assume other default values.

The benefit of defining settings in <tt>config.site</tt> is that it keeps specifc component "configure" scripts from trying to guess the values of specific variables. Those guesses (or assumptions) made by component scripts often lead to unpredictable results. In an ideal world, there would be no need to do this. The <tt>config.site</tt> file provides a more predictable alternative than trusting that component scripts are going to make the correct assumptions.

== .../xxx/build-data ==

The set of build-data directories contains detailed build and packaging data for components which are defined in each repo group.

=== platforms.mk ===

Each repo group includes the <tt>platforms.mk</tt> file, which is included by the main Makefile. The <tt>vpp/build-data/platforms.mk</tt> file is not complex. The <tt>closed-repo/build-data/platforms.mk</tt> file accomplishes two tasks. 

First, it includes <tt>vpp/build-data/platforms/*.mk</tt>:

 $(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS),	\
   $(eval -include $(d)/platforms/*.mk))

This collects the set of platform definition makefile fragments, as discussed [[VPP/Build_System_Deep_Dive#The_PLATFORM_variable|above]].

Second, the <tt>platforms.mk</tt> file implements a set of make targets.

=== packages/*.mk ===

Each component needs a makefile fragment in order for the build system to recognize it. The per-component makefile fragments vary considerably in complexity. For a component built with GNU autoconf / automake which does not depend on other components, the make fragment can be empty. See <tt>.../vpp/build-data/packages/vpp.mk</tt> for an uncomplicated but fully realistic example. 

Here are some of the important variable settings in per-component makefile fragments:

{| class="wikitable" border="1"
|-
! Variable
! Notes
|-
| xxx_configure_depend
| Lists the set of component build dependencies for the xxx component. In plain English: don't try to configure this component until you've successfully built the indicated targets. Almost always, xxx_configure_depend will list a set of "yyy-install" targets. Note the pattern: "variable names contain underscores, make target names contain hyphens"
|-
| xxx_configure_args
(optional)
| Lists any additional arguments to pass to the xxx component "configure" script. The main Makefile %-configure rule adds the required settings for --libdir, --prefix, and --host (when cross-compiling, aka most of the time)
|-
| xxx_CPPFLAGS
| Adds -I stanzas to CPPFLAGS for components upon which xxx depends. Almost invariably "xxx_CPPFLAGS = $(call installed_includes_fn, dep1 dep2 dep3)", where dep1, dep2, and dep3 are listed in xxx_configure_depend. It is bad practice to set "-g -O3" here. Those settings belong in a [[VPP/Build_System_Deep_Dive#The_TAG_variable|TAG]].
|-
| xxx_LDFLAGS
| Adds -Wl,-rpath -Wl,depN stanzas to LDFLAGS for components upon which xxx depends. Almost invariably "xxx_CPPFLAGS = $(call installed_lib_fn, dep1 dep2 dep3)", where dep1, dep2, and dep3 are listed in xxx_configure_depend. It is bad manners to set "-liberty-or-death" here. Those settings belong in Makefile.am. 
|}


When dealing with "irritating" components built with raw Makefiles which only work when building in the source trees, we use a specific strategy in the <tt>xxx.mk</tt> file. Examples include the Intel / 6wind DPDK. 

The strategy is simple for those "irritating" components: We copy the source tree into <tt>.../vpp/build-root/build-xxx</tt>. This works, but completely defeats dependency processing. This strategy is acceptable '''only''' for 3rd party software which won't need extensive (or preferably any) modifications.

Take a look at <tt>.../vpp/build-data/packages/dpdk.mk</tt>. When invoked, the <code>dpdk_configure</code> variable copies source code into <code>$(PACKAGE_BUILD_DIR)</code>, and performs the BSD equivalent of "autoreconf -i -f" to configure the build area. The rest of the file is similar: a bunch of hand-rolled glue code which manages to make the dpdk act like a good vpp build citizen even though it is not. There are worse examples, notably the historical tail-f "confd" integration.

== per-component GNU autotools input files ==

This section is not an introduction to the GNU autotools. If you are unfamiliar with the GNU autotools you can: stick to programming-by-example, or spend time learning about autotools. A bit of both seems the likely way to get started. There is a lot of information online from various sites on how to use GNU autotools. We recommend that you look at the canonical reference on gnu.org for autotools [http://www.gnu.org/savannah-checkouts/gnu/automake/manual/html_node/Autotools-Introduction.html introduction].

=== <repo>/<component>/configure.ac ===

The <tt>configure.ac</tt> file directs the GNU autotools to generate the "configure" script which the main Makefile invokes through the <code>xxx-configure</code> target.  

The configure script receives the set of options specified in <tt><repo>/build-data/packages/<component>/<component>.mk></tt> as specified by the [[VPP/Build_System_Deep_Dive#packages.2F.2A.mk|xxx_configure_args]] argument.  When cross-compiling, this is in addition to the arguments <code>--libdir</code>, <code>--prefix</code>, and <code>--host</code>.

Here is short but reasonable example:

 AC_INIT(foo, 1.1)
 AM_INIT_AUTOMAKE
 
 AC_PROG_LIBTOOL
 AM_PROG_AS
 AC_PROG_CC
 AM_PROG_CC_C_O 
 
 AC_OUTPUT([Makefile])

In rough terms, this says "please build a configure script for version 1.1 of the <tt>foo</tt> component. Make sure that you can find gcc, gas, and gld. The component may build a dynamic library in addition to a command binary." 

For an uncomplicated new component, the above configuration is about all you need to specify.

For a more sophisticated example, look at the <tt>.../vpp/vpp/configure.ac</tt> file. The settings in that file set automake conditional variables (used to control the set of files compiled). It is also arranged to add some <code>-Dfoo</code> variable settings to CFLAGS.

In principle, less is more. The more tools, libraries, etc. that you list in the <tt>configure.ac</tt> file, the slower the resulting configure script will run. The vpp build systems is ccache-enabled, timestamp enabled, and tuned to the point where the build system rate-limiter is the amount of time spent running "configure" scripts.