Create highly portable ELF binaries using the build-anywhere toolchain

This post describes the basic requirements for compiling highly portable ELF binaries. Essentially using a newer linux distro like Ubuntu 18.10 to build complex projects that run on older distros like CentOS 6. The details are limited to C/C++ projects and to x86_64 architectures.

The low-level solution is to use a C++ runtime that requires only glibc 2.13+ runtime linkage and link all third-party libraries as well as the compiler runtime and C++ implementation statically. Do not make a "fully static" binary. You will most likely find a glibc newer than 2.13 on every Linux distribution released since 2011. The high-level solution is to use the build-anywhere scripts to build a easy-to-use toolchain and set compiler flags.

What is build-anywhere?

I often find I need to build a set of third-party librarys for an executable I'd like to run on a lot of heterogenious Linux installs. I want a "really really static binary" but know in practice this is hard to acheive.

Introducing ' build-anywhere ', a x86_64 compiler and glibc runtime designed to help you build ultra-portable binaries. The build-anywhere toolchain is new, but it has had intial successes and should not change much beyond stabilization.

There are two bits of magic inside:

Both a static GCC libstdc++ and static LLVM libc++ implementation with C linkage against a glibc 2.13 and zlib 1.2.11. Scripts you can source to pick the right CC/CXX/CFLAGS/LDFLAGS options.

Quick TL;DR

Download x86_64-anywhere-linux-gnu-v2.tar.xz from the GitHub releases page Untar and install where ever you want Run source ./x86_64-anywhere-linux-gnu/scripts/anywhere-setup-security.sh Build your third-party libraries and make install Build you project Check your work with willitrun

In practice your third-party dependencies will install shared libraries and your final project will want to link those dispite your efforts to configure the build. You can "cheat" this by removing the shared libraries installed into the build-anywhere prefix.

A few examples

I have installed my anywhere runtime into /opt/rt . After sourcing the above script, some of my environment variables change.

The goal is now to work with a few library build systems so they pick up these variables and install static libraries into $PREFIX .

$ source ./x86_64-anywhere-linux-gnu/scripts/anywhere-setup-security.sh $ env | grep anywhere PATH=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin:/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/sbin:... CC=clang --gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot CXX=clang++ --gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot CONFIG_SITE=/opt/rt/x86_64-anywhere-linux-gnu/config.site PKG_CONFIG_PATH=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib/pkgconfig LIBRARY_PATH=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib CFLAGS=--gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot -march=x86-64 -fPIC -fPIE -fstack-protector-all -fsanitize=safe-stack CXXFLAGS=--gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot -march=x86-64 -fPIC -fPIE -fstack-protector-all -fsanitize=safe-stack -stdlib=libc++ PREFIX=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr

What does all of this mean?

Auto tools, CMake, and other build configuration managers will usually respect the environment's CC , CXX , CFLAGS , CXXFLAGS , and LDFLAGS settings If pkg-config is needed, it should prefer configurations in your build-anywhere runtime. The "prefix" that packages should install into is available as $PREFIX .

A lot of things can go wrong here. Off the top of my head, CPPFLAGS is not set, AS and other assembler flags are not set, you will most likely get duplicate-flag or unused-flag warnings, nothing controls package preference for static or shared library usage or production.

Never-the-less, let's jump in and build ssdeep .

$ wget https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz $ tar xf ssdeep-2.14.1.tar.gz $ cd ssdeep-2.14.1 $ ./configure --disabled-shared $ make && make install

This installed an ssdeep binary and a libfuzzy.a static library into the $PREFIX . The only non-standard thing here is adding --disable-shared .

Let's inspect the resulting ssdeep binary's runtime linkage requirements:

$ ldd -v $PREFIX/bin/ssdeep linux-vdso.so.1 (0x00007ffc813e1000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0ecf0c6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0eceea7000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0ecec9f000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0ecea9b000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0ece6aa000) /lib64/ld-linux-x86-64.so.2 (0x00007f0ecf464000) Version information: ./ssdeep: libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2 ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2 ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2 libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0 libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6 libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6 libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6 libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6

This binary depends on glibc 2.7+. This should run anywhere.

Use the provided willitrun to double check. This is a quick-script to apply basic linkage dependency checks.

$ willitrun $PREFIX/bin/ssdeep /opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/ssdeep: yes

The next example is OpenSSL, the build system requires inspection and will not work out of the box. Here is a working example.

$ wget https://dl.bintray.com/homebrew/mirror/openssl-1.0.2o.tar.gz $ tar xf openssl-1.0.2o.tar.gz $ cd openssl-1.0.2o $ ./Configure --prefix=$PREFIX \ no-ssl3 no-asm no-weak-ssl-ciphers \ zlib-dynamic no-shared linux-x86_64 \ "$CFLAGS -Wno-unused-command-line-argument" \ "$LDFLAGS" $ make depend $ make && make install

Next try a more complicated build like cmake.

$ wget https://github.com/Kitware/CMake/releases/download/v3.13.2/cmake-3.13.2.tar.gz $ tar xf cmake-3.13.2.tar.gz $ cd cmake-3.13.2 $ ./bootstrap --prefix=$PR