The GAP build system

This file is meant to give an overview of how the GAP build system works. It is targeted at people who need to work on the build system itself (to extend it, fixes bugs in it, etc.). It should not be necessary to read this if all you want to do is compile GAP and work on the GAP library and kernel.

Note that this really is just an overview; for details, please refer to the comments inside the various parts of the build system.

Prerequisites

In order to work on the build system, you need at least the following:

• GNU autoconf (we recommend 2.69 or later)
• GNU make

Note that we extensively use features provided by GNU make, so in general another version of make, such as BSD make, is not suitable.

Quick start: building GAP with no frills

If you are working with a fresh clone of the GAP repository, you need to run the autogen.sh script first, which will generate the configure script. Afterwards, or if you are using a release version of GAP, you can follow the standard procedure:

./configure
make

== Overview of the files constituting the GAP build system

• autogen.sh: sets up the build system; typically the first thing to run in a fresh clone of the GAP repository. It runs autoconf and autoheader.

• configure: generated by autogen.sh from configure.ac.

• configure.ac: the GNU autoconf source of our configure script.

• GNUmakefile, GNUmakefile.in: The former file is generated from the latter by configure. It is the primary Makefile (GNU make prefers it over Makefile). It only contains variables and vpath settings, and includes Makefile.rules for the actual build rules.

• Makefile: This is a placeholder file, and serves two purposes:

  1. If the user runs make before configure, it prints a warning.
  2. If configure did run, but make is not GNU make, it produces a corresponding error message.

• Makefile.rules: This is the core of the build system. If you want to add or remove a kernel C source file, you need to add or remove its name here and only here.

• cnf/: All files in this directory are part of the build system.

• extern/: External libraries we bundle with GAP (such as GMP) are put in here.

• build/: Generated code (such as config.h and version.c) is put into this directory.

• build/obj/: All *.o resp. *.lo files are placed into this directory.

• build/deps/ directories contain *.d files generated by the build system, and which are used to track dependencies, e.g. of C source files on header files.

Out-of-tree builds

The old GAP build system had a concept of "configs" and "CONFIGNAME", which allowed you to build GAP in different configurations from a single set of sources. This is gone in the current build system. However, a similar goal can be achieved by using so-called "out-of-tree builds".

In the following and also in the files that make up the build system, "srcdir" refers to the directory containing the GAP files, i.e. it contains this README, the src and lib directories and more.

To create a new out-of-tree build, create a new directory anywhere in your filesystem. A typical setup places the out-of-tree dirs into subdirectories of a "build" directory inside the srcdir. So you might have directories

srcdir/build/default
srcdir/build/default32
srcdir/build/hpcgap
srcdir/build/hpcgap32
...

We will refer to this directory from now on as the "builddir".

To initialize the out-of-tree build, change into the builddir and execute the configure script from the srcdir, like this:

cd $builddir
$srcdir/configure

You can pass any additional options you like to configure, e.g. ABI=32 or --enable-hpcgap.

Once the configure script has completed, you can run make as usual, and all the object files and the gap executable will be placed inside builddir. Your srcdir will remain untouched.

Dependency tracking

The build system tracks dependencies between files, such as between C source and header files, via *.d files stored in build/deps/. These files are mostly generated by the compiler; for this, the compiler needs to support the relevant flags (gcc, clang, icc all do so).

For a detailed explanation of a very similar scheme, see here: https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

HPC-GAP integration

One of the main features of the new build system is that it optionally allows to build HPC-GAP instead of plain GAP. HPC-GAP is an experimental fork of GAP which implements concurrent programming, multi-threading, etc..

The HPC-GAP kernel and library were forked from the GAP kernel and library and developed semi-independently for several years, with occasional merges between the two. In order to recombine the two, we merged the HPC-GAP fork into a subdirectory hpcgap of the GAP repository. Then, all files inside hpcgap which were identical to their counterparts in the GAP repository were deleted (e.g. hpcgap/src/ariths.c was deleted as it was identical to src/ariths.c). At this point, hpcgap/src has been fully merged, but there are still files in hpcgap/lib/ which differ from their counterparts in lib/

The new build system can optionally be instructed to build HPC-GAP, by passing the --enable-hpcgap flag to the configure script. For the resulting HPC-GAP binary to work, a trick is used: HPC-GAP mode uses multiple GAP root paths. Specifically, the GAP kernel function SySetGapRootPath was modified so that for every root directory FOO that gets added, we first add FOO/hpcgap to the list of root directories. This way, GAPROOT/hpcgap/lib is searched first for files, and only if no matching file is found there does GAP also search in GAPROOT/lib.

Cross compilation

The GAP build system supports cross compilation by leveraging the GNU autoconf support for this (please consult the GNU autoconf manual for details). When building GAP from a release archive, everything should just work (if it doesn't, please report it to us as a bug).

However, for development versions of GAP built directly from its git sources, there is a complication: GAP uses a few C source files which are the output of the GAP-to-C compiler gac, which in turn needs a working gap executable. This requires a bootstrapping process, which works because GAP can actually be built and used without those files, they are merely a performance optimization. So what our build system does is to first compile a version of GAP without those files, then run that to generate those files (if they are missing or not up-to-date). Then finally the actual GAP executable is compiled.

Unfortunately, this poses a problem for cross compilation: when compiling GAP on one computer (the "build" architecture) for a different computer with a different OS / CPU architecture / whatnot (the "host" architecture), then the first GAP executable needs to be built for the "build" architecture, so that it can run during the build process and generated code; while the second GAP executable needs to be compiled for the "host" architecture.

A second cross compilation obstacle is that there files build/ffdata.c and build/ffdata.h are generated by a C program ffgen that thus needs to be built for the "build" architecture.

To overcome these obstacles, you may proceed as follows:

1. First build GAP for the "build" architecture as usual.
2. This produces files build/c_*.c and build/ffdata.* which you should copy into the src directory.
3. Clean all build artifacts (alternatively, perform steps 1 & 2 in an out-of-tree build directory)
4. Build GAP a second time, for the "host" architecture. The build system will detect the files you copied into src in step 2, and will then not attempt to generate them.

Here is a minimal example to illustrate this:

./configure && make                 # build native GAP
cp build/c_*.c build/ffdata.* src   # copy the generated code
make clean
./configure --host=HOST_ARCH        # build for the "host" architecture
make

Note that this is really a minimalistic example; for actual cross compilation, you may need to provide further flags to the configure script and/or set environment variables, e.g. to ensure it finds dependencies like GMP or GNU readline, or to specify an installation prefix, and so on.

One final remark: some of the generated files differ for HPC-GAP. If you want to cross compile HPC-GAP, you need to generate them with HPC-GAP, and place them into src/hpc instead of src, like in this example:

./configure --enable-hpcgap
make
cp build/c_*.c src/hpc/
cp build/ffdata.* src/
make clean
./configure --host=HOST_ARCH --enable-hpcgap
make

Open tasks

There are many things that still need to be done in the new build system. For an overview, see https://github.com/gap-system/gap/issues?q=is%3Aopen+is%3Aissue+label%3A%22topic%3A+build+system%22