Makefile issues when cross compiling to armv7-linux-musleabihf

[Matt3o12]

Hello, I have tried to cross compile this project to armv7-linux-musleabihf from x64-linux-gnu and had a lot of trouble trying to cross compile the project. I wanted to make this issue to show others what I have done to cross compile this project and possible ask the maintainers if there are any bugs in the build script that need to be fixed.

Specifically, in order to cross compile I had to build jsonc, util-linux, and openssl openssl first and then run:

git clone https://github.com/tpm2-software/tpm2-tss.git /tpm2-tss
cd /tpm2-tss

git checkout "$TSS_VERSION"

# we set the DESTDIR to prefix and prefix to "" (nothing). We have to do that because otherwise the generated
# pkg-config files contain the prefix twice and rust will complain that it cannot find
# `/usr/local/arm-linux-musleabihf/usr/local/arm-linux-musleabihf`. This seems to be a bug in combination with
# a cross compile toolchain that also includes the sysroot. This is not an issue for any other dependency though.
export DESTDIR=/usr/local/arm-linux-musleabihf
./bootstrap

# This is usually 4.1.0-commit-hash but rust tss-esapi does not support that notation, so we need to fake it
echo 4.1.0 > VERSION

# we disable all various tcti devices that we don't need because so we don't need to build all deps for them
# as well. Most of those devices are test or usb devices anyway
# we also have to specify all lib paths because the sysroot is included twice by default
./configure --host=arm-linux-musleabihf --enable-esys --disable-fapi --disable-fapi-async-tests \
  --disable-weakcrypto LIBCRYPTO_CFLAGS="-I/usr/local/arm-linux-musleabihf/include" --enable-nodl \
  LIBCRYPTO_LIBS="-L/usr/local/arm-linux-musleabihf/lib -lcrypto" \
  JSONC_LIBS="-L/usr/local/arm-linux-musleabihf/lib -ljson-c" \
  JSONC_CFLAGS="-I/usr/local/arm-linux-musleabihf/include -I/usr/local/arm-linux-musleabihf/include/json-c" \
  --prefix= --build=x86_64-pc-linux-gnu \
  --disable-tcti-mssim --disable-tcti-swtpm --disable-tcti-pcap --disable-tcti-libtpms --disable-tcti-cmd \
  --disable-tcti-spi-helper --disable-tcti-spi-ltt2go --disable-tcti-spidev --disable-tcti-spi-ftdi \
  --disable-tcti-i2c-helper --disable-tcti-i2c-ftdi

make -j4

/usr/bin/install -c src/tss2-tcti/libtss2*.la /usr/local/arm-linux-musleabihf/lib/
/usr/bin/install -c src/tss2-tcti/.libs/libtss2*so* /usr/local/arm-linux-musleabihf/lib/

make install
unset DESTDIR

Here are the issues I had with the build process:

• For some reason, when I used prefix, the .pc files contained the preifx twice which caused compile errors in rust. So I had to set the prefix= and instead misuse DESTDIR to point to the sysroot. I am not sure why this is happening, I don't see this problem with openssl. So maybe it's a bug in the build script.
• I also had the same issue with the _LIBS and _CFLAGS options. Because it also included the sysroot twice.
• I disabled all tcti-* expect for device to make the compilation easier and because I don't need them. After doing this, the lib tss2-tcti-device was not installed by make install anymore. So I had to install them manually. After doing this, make install worked without any errors (before it was complaining that it could not find tcti-devices in lib). This seems to be a bug to me.

Here is the full build script:

#!/usr/bin/env bash

set -xeuf -o pipefail

required_vars=(
  "TSS_VERSION"
  "JSONC_VERSION"
  "UTIL_LINUX_VERSION"
  "OPENSSL_VERSION"
  "CC"
  "CXX"
  "PKG_CONFIG_PATH"
  "PKG_CONFIG_SYSROOT_DIR"
  "OPENSSL_NO_VENDOR" # since we are already building openssl, we should not vendor it by accident
)

required_pkgs=(
  "libltdl-dev"
  "autoconf-archive"
)

for var_name in "${required_vars[@]}"; do
  if [[ -z "${!var_name:-}" ]]; then
    echo "Error: Environment variable $var_name is not set."
    exit 1
  fi
done

for pkg in "${required_pkgs[@]}"; do
  dpkg -s $pkg >/dev/null 2>&1 || { echo "Error: $pkg is not installed."; exit 1; }
done

J_COUNT=$(nproc)

# Install jsonc
mkdir /jsonc /jsonc/build
curl -L "https://s3.amazonaws.com/json-c_releases/releases/json-c-$JSONC_VERSION-nodoc.tar.gz" | tar xz -C /jsonc --strip-components=1
cd /jsonc/build
../cmake-configure --prefix=/usr/local/arm-linux-musleabihf
make -j"$J_COUNT"
make install

# Install linux util for uuid
mkdir /util-linux
curl -L "https://www.kernel.org/pub/linux/utils/util-linux/v$UTIL_LINUX_VERSION/util-linux-$UTIL_LINUX_VERSION.tar.gz" | tar xz -C /util-linux --strip-components=1
cd /util-linux
./configure --disable-all-programs --prefix=/usr/local/arm-linux-musleabihf \
  --enable-libuuid --host=arm-linux-musleabihf --disable-year2038
make -j"$J_COUNT"
make install

# Install openssl
mkdir /openssl
curl -L "https://github.com/openssl/openssl/releases/download/openssl-$OPENSSL_VERSION/openssl-$OPENSSL_VERSION.tar.gz" | tar xz -C /openssl --strip-components=1
cd /openssl
./Configure linux-generic32 shared no-async no-unit-test no-tests --prefix=/usr/local/arm-linux-musleabihf \
  openssldir=/usr/local/arm-linux-musleabihf/openssl
make -j"$J_COUNT"
make install_sw install_ssldirs

# Install tpm2-tss
git clone https://github.com/tpm2-software/tpm2-tss.git /tpm2-tss
cd /tpm2-tss

git checkout "$TSS_VERSION"

# we set the DESTDIR to prefix and prefix to "" (nothing). We have to do that because otherwise the generated
# pkg-config files contain the prefix twice and rust will complain that it cannot find
# `/usr/local/arm-linux-musleabihf/usr/local/arm-linux-musleabihf`. This seems to be a bug in combination with
# a cross compile toolchain that also includes the sysroot. This is not an issue for any other dependency though.
export DESTDIR=/usr/local/arm-linux-musleabihf
./bootstrap

# This is usually 4.1.0-commit-hash but rust tss-esapi does not support that notation, so we need to fake it
echo 4.1.0 > VERSION

# we disable all various tcti devices that we don't need them and building against them is very difficult
# all of those devices are insecure anyways
# we also have to specify all lib paths because the sysroot is included twice by default
./configure --host=arm-linux-musleabihf --enable-esys --disable-fapi --disable-fapi-async-tests \
  --disable-weakcrypto LIBCRYPTO_CFLAGS="-I/usr/local/arm-linux-musleabihf/include" --enable-nodl \
  LIBCRYPTO_LIBS="-L/usr/local/arm-linux-musleabihf/lib -lcrypto" \
  JSONC_LIBS="-L/usr/local/arm-linux-musleabihf/lib -ljson-c" \
  JSONC_CFLAGS="-I/usr/local/arm-linux-musleabihf/include -I/usr/local/arm-linux-musleabihf/include/json-c" \
  --prefix= --build=x86_64-pc-linux-gnu \
  --disable-tcti-mssim --disable-tcti-swtpm --disable-tcti-pcap --disable-tcti-libtpms --disable-tcti-cmd \
  --disable-tcti-spi-helper --disable-tcti-spi-ltt2go --disable-tcti-spidev --disable-tcti-spi-ftdi \
  --disable-tcti-i2c-helper --disable-tcti-i2c-ftdi

make -j"$J_COUNT"

/usr/bin/install -c src/tss2-tcti/libtss2*.la /usr/local/arm-linux-musleabihf/lib/
/usr/bin/install -c src/tss2-tcti/.libs/libtss2*so* /usr/local/arm-linux-musleabihf/lib/

make install
unset DESTDIR

#cargo build --target armv7-unknown-linux-musleabihf

# TODO: remove all build deps

The docker image, ghcr.io/cross-rs/armv7-unknown-linux-musleabihf, provides a complete arm-linux-musleabihf toolchain that was generated by musl-cross-make. I used this because I already used it for rust anyways and it already has some very useful env variables set.

FROM ghcr.io/cross-rs/armv7-unknown-linux-musleabihf:edge

ENV RUSTUP_VERSION=1.27.1
ENV RUST_VERSION=stable

# Install rustup and rust/cargo, change HOME as that would point to /tmp/home which is not persisted
RUN curl https://static.rust-lang.org/rustup/archive/$RUSTUP_VERSION/x86_64-unknown-linux-gnu/rustup-init -o rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init --default-toolchain $RUST_VERSION --target armv7-unknown-linux-musleabihf -y --profile minimal -c fmt -c clippy && \
    apt-get update && apt-get install -y autoconf-archive libltdl-dev && rm -rf /var/lib/apt/lists/* && \
    rm rustup-init

ENV PATH="/root/.cargo/bin:$PATH"
# master as of Feb 18, 2025. We need to use master because enable-nodl is broken on the latest release.
# we must use enable-nodl for static builds
# see: https://github.com/tpm2-software/tpm2-tss/issues/2936
ENV TSS_VERSION=e9b61e7764f680b924c80fed38f8fa65ceb00b1a
ENV JSONC_VERSION=0.18
ENV UTIL_LINUX_VERSION=2.40
ENV OPENSSL_VERSION=3.0.16

ENV CC=arm-linux-musleabihf-gcc
ENV CXX=arm-linux-musleabihf-g++
ENV PKG_CONFIG_PATH=/usr/local/arm-linux-musleabihf/lib/pkgconfig
ENV PKG_CONFIG_SYSROOT_DIR=/usr/local/arm-linux-musleabihf
ENV SYSROOT=/usr/local/arm-linux-musleabihf

# For some reason, cargo's tss-esapi cannot find libtss2 nor libcrypto, despite pkg_config, so we need to set the rust
# flags manually.
ENV RUSTFLAGS="-lcrypto -L/usr/local/arm-linux-musleabihf/lib -ltss2-esys -ltss2-tcti-device"

# Important: as we are already compiling openssl, we should probably not use the vendored openssl...
ENV OPENSSL_NO_VENDOR=yes
COPY cross-compile-tss.sh /tmp

RUN apt-get update && apt-get install -y vim less tree bash-completion
RUN /tmp/cross-compile-tss.sh

Please let me know if you could reproduce the issues. I am not an expert in C and cross compiling C code, so some of the options could have also caused my issues.

[AndreasFuchsTPM]

I assume that you are referring to tss2-esys.pc correct ?

This is weird because the .pc files are generated directly from configure itself, and the entry is just

tpm2-tss/lib/tss2-esys.pc.in

Line 1 in 23d9c3c

prefix=@prefix@

Could you maybe post your tss2-esys.pc file as well as your config.log file ?

[Matt3o12]

Sure, I got this file when I use prefix=/usr/local/arm-linux-musleabihf:

cat lib/tss2-esys.pc
prefix=/usr/local/arm-linux-musleabihf
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: tss2-esys
Description: TPM2 Enhanced System API library.
URL: https://github.com/tpm2-software/tpm2-tss
Version: 4.1.0-82-g23d9c3c1
Requires.private: tss2-mu tss2-sys
Cflags: -I${includedir} -I${includedir}/tss2
Libs: -ltss2-esys -L${libdir}
Libs.private:    -L/usr/local/arm-linux-musleabihf/usr/local/arm-linux-musleabihf/lib -lcrypto

Note that Libs.private is incorrect which is used for static compliation (and why I am using musl).

config.log