Part II: Automake

The GCS defines standard target names to be included in Makefiles, such that the behavior of make becomes uniform among multiple packages. Here is a short list of the main targets:

make all

Builds all the targets sufficient to declare the package built.

make clean

Removes all derived files.

make check

Runs unit tests provided by the package.

make install

Installs the package

make uninstall

Uninstalls the package

make html

Builds the documentation in HTML format

make check

Performs self-tests

The Filesystem Hierarchy Standard (FHS) defines the directory structure and contents in Linux distributions. All Unix-like systems follow more or less the FHS, which defines standard places for files like:

It is important to know what these directories contain, because it will let you know where to install your software and the extra files that are distributed with it such as the documentation or the configuration files.

Installation directories are named by standard variables, so it gives the user the flexibility to install the package in a non-standard place. For example, this feature is required for packaging where the software is installed in a temporary directory.

The main variables are:

In the Makefile.in, these variables may be used between @ symbols, like @bindir@ for instance.

These variables can take different values by specifying them as arguments of the configure script:

./configure --prefix=$HOME/install --sysconfdir=$HOME/.config/

The GCS also defines a set of user-defined environment variables allowing to change the compilation behavior of the package.

• Pattern rules, such as

  %_test.o: %_test.c
          $(CC) $< -o $@
  

• Special functions $(wildcard *.c), $(patsubst %.f, %.o, $(SRC)), …
• The ability to have one rule with multiple output files

• Don't use any other automatic variable than $@ (the name of the current target)
• Write your scripts with _bash{/bin/sh} instead of /bin/bash
• Don't put your compiled files (.o) in a separate location than the source files
• Give up writing your Makefiles by hand. Use scripts (Bash, Python) to write them for you.
• Install bmake as an alternative to GNU make to check the portability of your Makefiles.
• Use $(MAKE) in your Makefiles instead of make for recursive builds
• Similarly, use POSIX constructs for your shell scripts or shell constructs in configure.ac (tests with a single square bracket).

There are two types of people who will try to compile the source code:

1. The end-users having downloaded a tarball with sources
2. The developers of the code, who produce the source distribution of the code

These two profiles should be differentiated, because their system requirements are usually different. The first difference is that the code developers will need to install Autotools as they will produce the configure script, but Autotools will not be needed by the end-users.

A more critical example is when complex Makefiles are produced by a script. The script generating the Makefiles should be stored in the version control system for developers, as opposed to the generated Makefiles. For the end-user's point of view, only the generated Makefiles are needed for the compilation of the sources, and the script should not be present in the source distribution.

To handle these two types of profiles, Autotools provides the maintainer mode. For instance, in maintainer mode the configure script is re-generated when configure.ac has changed, as opposed to standard mode. All these rules are activated in the generated Makefiles by running

./configure --enable-maintainer-mode

Automake provides automatically the dist target. make dist creates a .tar.gz file which contains all the files necessary to install the package: installation instructions, configure script, source code, documentation, man pages, unit tests, etc.

$ make dist
[...]
$ tar --list -f sherman-morrison-0.0.1.tar.gz
sherman-morrison-0.0.1/
sherman-morrison-0.0.1/m4/
sherman-morrison-0.0.1/m4/ax_lib_hdf5.m4
sherman-morrison-0.0.1/src/
sherman-morrison-0.0.1/src/SM_Maponi.cpp
sherman-morrison-0.0.1/src/SM_Standard.cpp
sherman-morrison-0.0.1/src/Woodbury.cpp
sherman-morrison-0.0.1/src/SMWB.cpp
sherman-morrison-0.0.1/src/Helpers.cpp
sherman-morrison-0.0.1/tests/
sherman-morrison-0.0.1/tests/test_h5.cpp
sherman-morrison-0.0.1/Makefile.am
sherman-morrison-0.0.1/configure
sherman-morrison-0.0.1/configure.ac
sherman-morrison-0.0.1/aclocal.m4
sherman-morrison-0.0.1/Makefile.in
sherman-morrison-0.0.1/compile
sherman-morrison-0.0.1/depcomp
sherman-morrison-0.0.1/install-sh
sherman-morrison-0.0.1/missing

To answer this question, we can use the power of Automake and run the distcheck target:

$ make distcheck
[...]
../../src/SM_Maponi.cpp:4:10: fatal error: SM_Maponi.hpp: No such file or directory
 #include "SM_Maponi.hpp"
          ^~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:449: src/SM_Maponi.o] Error 1
make[1]: Leaving directory '/home/test/Sherman-Morrison-0.1-demo/sherman-morrison-0.0.1/_build/sub'
make: *** [Makefile:613: distcheck] Error 1

Oh no! This archive is not valid. By inspecting the content of the archive, we notice we forgot to add the header files in the sources. Although the project builds on our platform, the Makefile is incomplete because it lacks some dependencies.

We can update the SOURCES variables as

SOURCES = src/SM_Maponi.cpp       \
          src/SM_Standard.cpp     \
          src/Woodbury.cpp        \
          src/SMWB.cpp            \
          src/Helpers.cpp         \
          tests/test_h5.cpp       \
          include/Helpers.hpp     \
          include/SM_Maponi.hpp   \
          include/SM_Standard.hpp \
          include/SMWB.hpp        \
          include/Woodbury.hpp

and run make dist again to rebuild the source distribution. Note that the current Makefile detects that Makefile.am has changed, so it re-runs Automake and also re-generates the Makefile using config.status before producing the updated archive.

$ tar --list -f sherman-morrison-0.0.1.tar.gz
sherman-morrison-0.0.1/
sherman-morrison-0.0.1/include/
sherman-morrison-0.0.1/include/Helpers.hpp
sherman-morrison-0.0.1/include/SM_Maponi.hpp
sherman-morrison-0.0.1/include/SM_Standard.hpp
sherman-morrison-0.0.1/include/SMWB.hpp
sherman-morrison-0.0.1/include/Woodbury.hpp
sherman-morrison-0.0.1/m4/
sherman-morrison-0.0.1/m4/ax_lib_hdf5.m4
sherman-morrison-0.0.1/src/
sherman-morrison-0.0.1/src/SM_Maponi.cpp
sherman-morrison-0.0.1/src/SM_Standard.cpp
sherman-morrison-0.0.1/src/Woodbury.cpp
sherman-morrison-0.0.1/src/SMWB.cpp
sherman-morrison-0.0.1/src/Helpers.cpp
sherman-morrison-0.0.1/tests/
sherman-morrison-0.0.1/tests/test_h5.cpp
sherman-morrison-0.0.1/Makefile.am
sherman-morrison-0.0.1/configure
sherman-morrison-0.0.1/configure.ac
sherman-morrison-0.0.1/aclocal.m4
sherman-morrison-0.0.1/Makefile.in
sherman-morrison-0.0.1/compile
sherman-morrison-0.0.1/depcomp
sherman-morrison-0.0.1/install-sh
sherman-morrison-0.0.1/missing

We can now test again the source distribution:

$ make distcheck
[...]
make[1]: Entering directory '/home/test/Sherman-Morrison-0.1-demo/sherman-morrison-0.0.1/_build/sub'
depbase=`echo src/SM_Maponi.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DPACKAGE_NAME=\"sherman-morrison\" -DPACKAGE_TARNAME=\"sherman-morrison\" -DPACKAGE_VERSION=\"0.0.1\" -DPACKAGE_STRING=\"sherman-morrison\ 0.0.1\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"sherman-morrison\" -DVERSION=\"0.0.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_HDF5=1 -I. -I../..   -I/usr/include -I/usr/include/hdf5/serial  -I./include/  -g -O2 -std=c++11 -MT src/SM_Maponi.o -MD -MP -MF $depbase.Tpo -c -o src/SM_Maponi.o ../../src/SM_Maponi.cpp &&\
mv -f $depbase.Tpo $depbase.Po
../../src/SM_Maponi.cpp:4:10: fatal error: SM_Maponi.hpp: No such file or directory
 #include "SM_Maponi.hpp"
          ^~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:449: src/SM_Maponi.o] Error 1
make[1]: Leaving directory '/home/test/Sherman-Morrison-0.1-demo/sherman-morrison-0.0.1/_build/sub'
make: *** [Makefile:613: distcheck] Error 1

Oh no! Still the same error! But now we know that the header file is present.

Inspecting the compilation line, we notice that the compilation takes place in the sherman-morrison-0.0.1/_build/sub directory. Automake is now trying to make an out of source build: building the package outside of the source directory. In the previous tutorial, we have introduced -I./include to the CPPFLAGS variable in the configure.ac file. Using such a relative path implies that we are running make in the source directory, which is not desirable if you want to be able to make RPM packages for instance.

So now we need to move this definition into the Makefile.am file. configure.ac becomes:

CPPFLAGS="${HDF5_CPPFLAGS} ${CPPFLAGS}"

and Makefile.am has the extra line:

AM_CPPFLAGS =  -I$(srcdir)/include/

You can remark that we have now introduced the srcdir variable which points the the root of the source. For in source builds, srcdir is . but for out of source builds it is set to the relative path to the root. If for some reason you need the absolute path, you can use abs_srcdir instead.

Now, let us test our source distribution:

$ make distcheck
[...]
========================================================
sherman-morrison-0.0.1 archives ready for distribution:
sherman-morrison-0.0.1.tar.gz
========================================================

Success! the distribution compiles and we now have something that we can give to our users!

Here is the list of the main variables that are used in Makefile.am:

AM_CPPFLAGS = -I/dir1 -I/dir2 -I$(srcdir)/include ...

List the -I flags that need to be passed to the compiler to find the include files

AM_LDFLAGS = -L/dir1 -L/dir2 ...

Flags to be passed to the linker

LDADD = file.o $(top_builddir)/dir1/libmylib.la -lmylib ...

Object files and built libraries to be linked with all executables

CLEANFILES = my_file.mod ...

List of extra files to remove with make clean

EXTRA_DIST

Additional files to be shipped with the binary distribution.

User variables such as CFLAGS or CPPFLAGS should never be overwritten in a Makefile.am. This explains why these variables have the AM_ prefix.

In the previous section, we have seen two special variables bin_PROGRAMS and test_h5_SOURCES.

bin_PROGRAMS contains the names of the programs we are building, and these will be installed in $(exec_prefix)/bin upon invocation of make install.

Product list variables (PLV) conform to the following template:

[modifier-list]prefix_PRIMARY = product1 product2 ... productN

In our example, the name of the program is test_h5, so by defining test_h5_SOURCES we can list the source files needed to build test_h5. Automake will then do everything necessary in the Makefile.in to build test_h5.

Product source variables (PSV) conform to the following template:

[modifier-list]product_SOURCES = file1 file2 ... fileN

product refers to the built product. If the product contains special characters, these should be replaced by underscores. For example:

lib_LIBRARIES = libc++.a
libc___a_SOURCES = ...

Modifiers change the default behavior of the variable. The main modifiers are dist and nodist, which indicate if the files should be part of the distributed package or not.

The following macros will be useful to display messages to the users running configure:

The usage of an external software can be controlled by the user by specifying some options to the configure script: --with-<package>=<arg> or --without-<package>.

Let us now add the possibility to compile our program with or without MPI. To do that, we start by adding to the configure.ac file:

AC_ARG_WITH(mpi, [AS_HELP_STRING([--with-mpi=yes/no], [Activate MPI])])

After regenerating the configure script, we see that now configure --help shows the --with-mpi option. If the --with-mpi is absent of the command line, the with_mpi variable will be empty. Otherwise, it will contain yes.

AC_ARG_WITH(mpi, [AS_HELP_STRING([--with-mpi], [Activate MPI])])
AS_IF([test "x$with_mpi" == "xyes"], [
  AC_DEFINE([HAVE_MPI], [], [MPI is activated])
  AC_MSG_NOTICE([MPI is configured])
  AC_CHECK_PROGS(MPICC, [mpicc])
  AC_CHECK_PROGS(MPICXX, [mpicxx])
  CC=$MPICC
  CXX=$MPICXX
])

It is convenient to define a variable that allows to know if the configuration should be for a maintainer or for an end user. The simplest recipe is to check for the existence of the .git directory at the root of the project. If it exists, we are not in the source distribution but in a directory obtained by git clone.

AC_CHECK_FILE([.git], [IS_MAINTAINER="yes"], [IS_MAINTAINER="no"])

Then, the commands specific to maintainer mode can be executed in configure.ac by checking the value of this variable:

AS_IF([test "$IS_MAINTAINER" == "yes"],
      [AC_MSG_NOTICE([Maintainer configuration])],
      [AC_MSG_NOTICE([User configuration])])

To propagate this knowledge to Makefile.am, you can use in configure.ac

AM_CONDITIONAL([IS_MAINTAINER], [test "$IS_MAINTAINER" == "yes"])

and then, in Makefile.am

if IS_MAINTAINER
[...]
endif

Inside configure.ac, it is rather simple to get the git hash and store it into a defined variable:

GIT_HASH=`git log | head -1 | cut -d ' ' -f 2`

When the distribution package is made, the information relative to the Git repository is not stored in the tarball. So it will not be possible to get the Git hash by executing this command for the end user. The Git hash is available only for maintainers and needs to be stored somewhere in the distribution package to be propagated to the users.

In configure.ac:

AS_IF([test "$IS_MAINTAINER" = "yes"],
      [GIT_HASH=`git log | head -1 | cut -d ' ' -f 2 | tee ${srcdir}/.git_hash`],
      [GIT_HASH=`cat ${srcdir}/.git_hash`])

AC_DEFINE_UNQUOTED(GIT_HASH, ["${GIT_HASH}"], [Git SHA1 Hash])

In maintainer mode, we fetch the Git hash and store it in .git_hash. In user mode, we read the content of .git_hash. Then, we propagate this variable to config.h with AC_DEFINE_UNQUOTED.

In Makefile.am, we add the new file to the source distribution tarball:

EXTRA_DIST += .git_hash

and we add rules to always regenerate it in case we make a commit:

if IS_MAINTAINER

CLEANFILES = .git_hash

.git_hash: FORCE
        git log | head -1 | cut -d ' ' -f 2 > .git_hash

all: .git_hash

.PHONY: FORCE
endif