Part I: Autoconf

Building software on different platforms is a challenging task. Re-building it a few years later is even more challenging.

Things get more complicated when the project introduces external dependencies (e.g. libraries that are installed independently and outside of the source tree). Such dependencies can be located with some built-in tools like pkg-config. One could then modify the compilation flags to explicitly include installation paths, thus making the code compile on the current machine. This approach works fine as long as the code has to be compiled only locally and as long as the dependency is not moved to a different directory. However, the installation paths are likely to be different on another machine and the aforementioned build procedure might break.

Autoconf was designed to solve this problem by creating configuration scripts that work correctly on most systems, even on systems not initially considered by the developer. As opposed to CMake, all the burden is put on the shoulders of the developer to make the life simpler for the end user.

In this tutorial, we will learn how to configure a particular project using Autoconf. This project consists of several C++ source files, which are compiled into a single program using custom Makefile. It is worth mentioning that you don't need to be a C++ programmer to be able to do this tutorial. We will learn how to configure and test the C++ compiler before building the project. The source files depend on the HDF5 library, which is a good example of external dependency. HDF5 is a binary file format, which is designed for high-performance data input/output. We will learn how to automatically locate the HDF5 library using macros.

Below is a source tree of our project. Header and source files can be found in the include/ and src/ directories, respectively. The tests/ directory contains the test_h5.cpp and dataset.hdf5 files needed to test compilation of the aforementioned source codes.

For this tutorial, the contents of the source and header files are not important. However, it is worth mentioning that they depend on the HDF5 library, which is not included in the source tree. Our goal is to demonstrate how cumbersome configuration/compilation steps can be facilitated by the use of Autoconf.

Sherman-Morrison/
├─ bin/
│  ├─ .gitignore
├─ include/
│  ├─ Helpers.hpp
│  ├─ SMWB.hpp
│  ├─ SM_Maroni.hpp
│  ├─ SM_Standard.hpp
│  ├─ Woodbury.hpp
├─ src/
│  ├─ Helpers.cpp
│  ├─ SMWB.cpp
│  ├─ SM_Maroni.cpp
│  ├─ SM_Standard.cpp
│  ├─ Woodbury.cpp
├─ tests/
│  ├─ test_h5.cpp
│  ├─ dataset.hdf5
├─ LICENSE
├─ Makefile
├─ README.md
├─ .gitignore

The source code comes from the Sherman-Morrison GitHub repository of the TREX-CoE. It is distributed under BSD 3-Clause License.

Below is the Makefile of our project:

## Specify the C++ compiler as well as preprocessor, compiler and linking flags
CXX      = g++
CXXFLAGS = -g -O2 -std=c++11
CPPFLAGS = -I/usr/include -I/usr/include/hdf5/serial  -I./include/
LIBS     = -lhdf5_hl -lhdf5  -lpthread -lsz -lz -ldl -lm  -lhdf5_cpp
LDFLAGS  = -L/usr/lib/x86_64-linux-gnu/hdf5/serial

## Directory structure
SRC_DIR := src
TST_DIR := tests
INC_DIR := include
BIN_DIR := bin

EXEC    := $(BIN_DIR)/test_h5

## Dependencies
DEPS_CXX = $(SRC_DIR)/SM_Maponi.o \
           $(SRC_DIR)/SM_Standard.o \
           $(SRC_DIR)/Woodbury.o \
           $(SRC_DIR)/SMWB.o \
           $(SRC_DIR)/Helpers.o

## Build tagets
.PHONY: all clean

all: $(EXEC)

clean:
        @rm -rf -- $(SRC_DIR)/*.o $(TST_DIR)/*.o $(EXEC)

## Compiling dependencies
.SUFFIXES: .cpp .o

## Linking
$(BIN_DIR)/test_h5: $(DEPS_CXX) $(TST_DIR)/test_h5.o
        $(CXX) $(CPPFLAGS) $(CXXFLAGS) $(TST_DIR)/test_h5.o $(DEPS_CXX) -o $(EXEC) $(LDFLAGS) $(LIBS)

Let's have a look at the following part of our Makefile (from now on we will only focus on the following lines):

CXX      = g++
CXXFLAGS = -g -O2 -std=c++11
CPPFLAGS = -I/usr/include -I/usr/include/hdf5/serial  -I./include/
LIBS     = -lhdf5_hl -lhdf5  -lpthread -lsz -lz -ldl -lm  -lhdf5_cpp
LDFLAGS  = -L/usr/lib/x86_64-linux-gnu/hdf5/serial

Among other things, the crucial issue here is the portability of the hard-coded paths to HDF5 (see CPPFLAGS and LDFLAGS).

Let's execute make command and examine the output. Normally, you should get a lot of errors, which are impossible to debug. So what happened?

The compilation crashed due to several issues. One of them is that the aforementioned Makefile was adapted for one single machine. Another issues comes from the fact that compiler flags are overwritten in Makefile instead of being appended to the existing ones. It means that even if the current environment (e.g. the one created with conda) did some smart job and pre-set some compiler options, this information is still lost.

Let's append to the existing flags instead of overwriting them. For this we will use += syntax of Makefile.

CXX       = g++
CXXFLAGS += -std=c++11
CPPFLAGS += -I./include/
LIBS     += -lhdf5 -lhdf5_cpp -lz
LDFLAGS  += -L/home/user/.conda/envs/autotools/lib

If we now run make command the output may look like this:

$ make

g++ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11   -I./include/  -c -o src/SM_Maponi.o src/SM_Maponi.cpp
g++ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11   -I./include/  -c -o src/SM_Standard.o src/SM_Standard.cpp
g++ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11   -I./include/  -c -o src/Woodbury.o src/Woodbury.cpp
g++ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11   -I./include/  -c -o src/SMWB.o src/SMWB.cpp
g++ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11   -I./include/  -c -o src/Helpers.o src/Helpers.cpp
g++ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11   -I./include/  -c -o tests/test_h5.o tests/test_h5.cpp
g++  -I./include/ -fPIC -isystem /home/user/.conda/envs/autotools/include -std=c++11 tests/test_h5.o src/SM_Maponi.o src/SM_Standard.o src/Woodbury.o src/SMWB.o src/Helpers.o -o bin/test_h5  -L/home/user/.conda/envs/autotools/lib -lhdf5 -lhdf5_cpp -lz

So it works.

Unfortunately, our solution does not resolve the portability issue. In this example, we demonstrated that defining compiler flags is challenging, especially for programs with external dependencies.

Would it be better if we could automate this process? Well, this is where Autoconf comes into play.

In this section, we will talk about several files:

• configure.ac
• configure
• Makefile.in
• Makefile

This might be confusing at the beginning. Let's create the simplest configure.ac file for our project with the following contents:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

# Initialize autoconf
AC_INIT([sherman-morrison], [0.0.1], [])

AC_OUTPUT

Here we provided the name of our project [sherman-morrison] and the current version [0.0.1] to the AC_INIT macro. These are now stored in the $PACKAGE_NAME and $PACKAGE_VERSION variables, respectively. We will not cover the details of the M4 syntax that is used internally by the Autoconf. For more information, see the Autoconf documentation page.

We know that the source code is written in C++ programming language (it can also be guessed from the .cpp extension). Thus, we would like the Autoconf to find and possibly run some checks using the C++ compiler that is available on the current machine.

Let's add a few additional macros for that and examine their output:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

# Initialize autoconf
AC_INIT([sherman-morrison], [0.0.1], [])

# do the tests with C++ flags
AC_LANG(C++)
# search for the C++ compiler
AC_PROG_CXX
# check if the C++ compiler accepts -c and -o simultaneously
AC_PROG_CXX_C_O

AC_OUTPUT

echo \
"-------------------------------------------------

${PACKAGE_NAME} Version ${PACKAGE_VERSION}

CXX ...........:  ${CXX}
CXXFLAGS ......:  ${CXXFLAGS}

--------------------------------------------------"

The contents of the current directory should be as follows:

bin/  configure.ac  include/  LICENSE  Makefile  README.md  src/  tests/

Now run autoreconf command in your terminal and examine the directory again. After succesfull execution, a configure shell script should appear.

Let's run it by calling ./configure. Several checks have been performed by Autoconf and in the end one might get the following (the CXX output will be different on your machine):

-------------------------------------------------

sherman-morrison Version 0.0.1

CXX ...........:  /home/user/.conda/envs/autotools/bin/x86_64-conda-linux-gnu-c++
CXXFLAGS ......:  -fPIC -isystem ${CONDA_PREFIX}/include

--------------------------------------------------

In the example above, the configure script has detected the GNU C++ compiler and expanded the ${CXX} variable with the corresponding name.

From now on we will focus mainly on $-prefixed variables and calls to the Autoconf macros.

The configure script can also define its own variables (e.g. $PACKAGE_NAME above) and pass them to other files. To benefit from the previously detected C++ compiler and its flags, let's propagate them to Makefile.

This requires 2 actions:

• Move Makefile to Makefile.in (this is the standard naming convention in Autotools) and replace the hard-coded variables with the ones defined by the configure script. For demonstration purposes, we only replace CXX and CXXFLAGS at the moment.
• Indicate to Autoconf that producing Makefile is now its responsibility (due to the variables that will be substituted by ./configure call). This can be achieved by adding AC_CONFIG_FILES([Makefile]).

configure.ac:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

#initialize autoconf
AC_INIT([sherman-morrison], [0.0.1], [])

# do the tests with C++ flags
AC_LANG(C++)
# search for the C++ compiler
AC_PROG_CXX
# check if the C++ compiler accepts -c and -o simultaneously
AC_PROG_CXX_C_O

AC_CONFIG_FILES([Makefile])

AC_OUTPUT

echo \
"-------------------------------------------------

${PACKAGE_NAME} Version ${PACKAGE_VERSION}

CXX ...........:  ${CXX}
CXXFLAGS ......:  ${CXXFLAGS}

--------------------------------------------------"

Makefile.in (remember, only the top part has to be changed):

CXX       = @CXX@
CXXFLAGS  = @CXXFLAGS@
CPPFLAGS += -I./include/
LIBS     += -lhdf5 -lhdf5_cpp -lz
LDFLAGS  += -L/home/user/.conda/envs/autotools/lib

Let's reconfigure (autoreconf -i && ./configure) and examine the changes between the input Makefile.in and output Makefile (you can use diff or vimdiff for example):

$ diff Makefile.in Makefile

< CXX      = @CXX@
< CXXFLAGS = @CXXFLAGS@
---
> CXX       = /home/user/.conda/envs/autotools/bin/x86_64-conda-linux-gnu-c++
> CXXFLAGS  = -fPIC -isystem ${CONDA_PREFIX}/include

This tells us that the substitution variables @CXX@ and @CXXFLAGS@ have been replaced with the values detected by the configure script.

Good, but we still have the hard-coded path to lib directory in LDFLAGS. Among other things, this directory contains the HDF5 library. Let's try to find this library using Autoconf functionality.

It is possible to write your our own macros to detect external dependencies. However, in this example we will try to locate the HDF5 library, which is widely used. Luckily for us, the GNU website has an archive of Autoconf macros for many packages, including HDF5. The ax_lib_hdf5.m4 file can be downloaded/copied from this webpage.

Once the file is in the m4 directory, one can add the following lines to configure.ac in order to locate and call the ax_lib_hdf5.m4 macro:

# tell Autoconf the name of directory with external M4 macros
AC_CONFIG_MACRO_DIR([m4])
# call the m4 macro to locate the HDF5 library
AX_LIB_HDF5()

The description of this macro can be found here. Notably, it says:

If HDF5 is successfully found, this macro calls

  AC_SUBST(HDF5_VERSION)
  AC_SUBST(HDF5_CC)
  AC_SUBST(HDF5_CFLAGS)
  AC_SUBST(HDF5_CPPFLAGS)
  AC_SUBST(HDF5_LDFLAGS)
  AC_SUBST(HDF5_LIBS)
  AC_SUBST(HDF5_FC)
  AC_SUBST(HDF5_FFLAGS)
  AC_SUBST(HDF5_FLIBS)
  AC_SUBST(HDF5_TYPE)
  AC_DEFINE(HAVE_HDF5)

and sets with_hdf5="yes". ...

This means that the macro will define a set of variables that can be manupulated within configure.ac and/or Makefile.in files. In this example, we choose the former and will append the necessary flags with the HDF5_ ones.

configure.ac:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

#initialize autoconf
AC_INIT([sherman-morrison], [0.0.1], [])

# do the tests with C++ flags
AC_LANG(C++)
# search for the C++ compiler
AC_PROG_CXX
# check if the C++ compiler accepts -c and -o simultaneously
AC_PROG_CXX_C_O

# tell Autoconf the name of directory with external M4 macros
AC_CONFIG_MACRO_DIR([m4])
# call the m4 macro to locate the HDF5 library
AX_LIB_HDF5()

# raise an error if HDF5 is missing
if test "x${with_hdf5}" = xno; then
  AC_MSG_ERROR([
    ------------------------------------------
    Configuring with the HDF5 library is
    required to build this project.
    ------------------------------------------])
fi

# prepend compiler and linking flags with that of HDF5
CXXFLAGS="${HDF5_CFLAGS} ${CXXFLAGS} -std=c++11"
CPPFLAGS="${HDF5_CPPFLAGS} ${CPPFLAGS} -I./include/"
LDFLAGS="${HDF5_LDFLAGS} ${LDFLAGS}"
LIBS="${HDF5_LIBS} ${LIBS} -lhdf5_cpp"

AC_CONFIG_FILES([Makefile])

AC_OUTPUT

echo \
"-------------------------------------------------

${PACKAGE_NAME} Version ${PACKAGE_VERSION}

CXX ...........:  ${CXX}
CXXFLAGS ......:  ${CXXFLAGS}
CPPFLAGS ......:  ${CPPFLAGS}
LDFLAGS .......:  ${LDFLAGS}
LIBS ..........:  ${LIBS}

Package features:
  Compilation with HDF5 ..:  ${with_hdf5}
  HDF5 version ...........:  ${HDF5_VERSION}

--------------------------------------------------"

Makefile.in (only the top part has to be changed):

CXX      = @CXX@
CXXFLAGS = @CXXFLAGS@
CPPFLAGS = @CPPFLAGS@
LIBS     = @LIBS@
LDFLAGS  = @LDFLAGS@

Now reconfigure and examine the output produced by the configure script. In the end, you will see a more verbose summary of the detected flags and newly added package features. The exact output may differ on your machine depending on the detected compiler and/or the HDF5 library.

-------------------------------------------------

sherman-morrison Version 0.0.1

CXX ...........:  /home/user/.conda/envs/autotools/bin/x86_64-conda-linux-gnu-c++
CXXFLAGS ......:   -fPIC -isystem ${CONDA_PREFIX}/include -std=c++11
CPPFLAGS ......:  -I/home/user/.conda/envs/autotools/include   -I./include/
LDFLAGS .......:   -L/home/user/.conda/envs/autotools/lib  
LIBS ..........:  -lhdf5_hl -lhdf5  -lcrypto -lcurl -lrt -lpthread -lz -ldl -lm   -lhdf5_cpp

Package features:
  Compilation with HDF5 ..:  yes
  HDF5 version ...........:  1.12.1

--------------------------------------------------

You can now see how all hard-coded HDF5 paths have been detected by the ax_lib_hdf5.m4 macro. Moreover, all flags are propagated to the resulting Makefile due to the use of substitution variables.

The pkg-config program is a powerful tool to get information about installed libraries. Autoconf incorporates some of the pkg-config functionality.

The PKG_CHECK_MODULES can be configured to detect and check the required version of the HDF5 library (see the Autotools Mythbuster website for more details). For example, PKG_CHECK_MODULES([HDF5], [hdf5 >= 1.8]) verifies that the HDF5 version is not older than 1.8.

configure.ac:

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

#initialize autoconf
AC_INIT([sherman-morrison], [0.0.1], [])

# do the tests with C++ flags
AC_LANG(C++)
# search for the C++ compiler
AC_PROG_CXX
# check if the C++ compiler accepts -c and -o simultaneously
AC_PROG_CXX_C_O

# tell Autoconf the name of directory with external M4 macros
AC_CONFIG_MACRO_DIR([m4])
# call the m4 macro to locate the HDF5 library
AX_LIB_HDF5()

# raise an error if HDF5 is missing
if test "x${with_hdf5}" = xno; then
  AC_MSG_ERROR([
    ------------------------------------------
    Configuring with the HDF5 library is
    required to build this project.
    ------------------------------------------])
fi

# prepend compiler and linking flags with that of HDF5
CXXFLAGS="${HDF5_CFLAGS} ${CXXFLAGS} -std=c++11"
CPPFLAGS="${HDF5_CPPFLAGS} ${CPPFLAGS} -I./include/"
LDFLAGS="${HDF5_LDFLAGS} ${LDFLAGS}"
LIBS="${HDF5_LIBS} ${LIBS} -lhdf5_cpp"

# clean the output of AX_LIB_HDF5 macro since the same variables are substitutted by PKG_CHECK_MODULES
HDF5_CFLAGS=""
HDF5_LIBS=""
# locate HDF5 with pkg-config
PKG_CHECK_MODULES([HDF5], [hdf5 >= 1.8])

AC_CONFIG_FILES([Makefile])

AC_OUTPUT

echo \
"-------------------------------------------------

${PACKAGE_NAME} Version ${PACKAGE_VERSION}

CXX ...........:  ${CXX}
CXXFLAGS ......:  ${CXXFLAGS}
CPPFLAGS ......:  ${CPPFLAGS}
LDFLAGS .......:  ${LDFLAGS}
LIBS ..........:  ${LIBS}

Package features:
  Compilation with HDF5 ..:  ${with_hdf5}
  HDF5 version ...........:  ${HDF5_VERSION}
  .........................
  pkg-config CFLAGS ......:  ${HDF5_CFLAGS}
  pkg-config LIBS ........:  ${HDF5_LIBS}

--------------------------------------------------"

Reconfiguration should fail with the following error message:

checking for hdf5 >= 1.8... no
configure: error: Package requirements (hdf5 >= 1.8) were not met:

No package 'hdf5' found

This might happen when package developers forgot to provide the configuration file for pkg-config or when it is installed in some non-standard location.

Many Autoconf macros (including PKG_CHECK_MODULES) allow to specify actions to be taken if a given file/program/library has been found or not. In the case of PKG_CHECK_MODULES, the syntax is the following:

PKG_CHECK_MODULES(prefix, list-of-modules, action-if-found, action-if-not-found)

Notably, the raised warning allows the configuration to pass (unlike AC_MSG_ERROR).

In this Tutorial you have learned how to make your in-house Makefile compatible with Autoconf. This is achieved by converting Makefile into Makefile.in and introducing substitution variables. You have also learned how to use Autoconf to create the configure script, which sets up and checks programs and libraries on the host machine. The configure script is also responsible for producing the final Makefile, which is used to build the project. Finally, you used the Autoconf macro from the GNU archive to configure the project, which has an external dependency (i.e. the HDF5 library).