1=================================================================== 2How to Cross Compile Compiler-rt Builtins For Arm 3=================================================================== 4 5Introduction 6============ 7 8This document contains information about building and testing the builtins part 9of compiler-rt for an Arm target, from an x86_64 Linux machine. 10 11While this document concentrates on Arm and Linux the general principles should 12apply to other targets supported by compiler-rt. Further contributions for other 13targets are welcome. 14 15The instructions in this document depend on libraries and programs external to 16LLVM, there are many ways to install and configure these dependencies so you 17may need to adapt the instructions here to fit your own local situation. 18 19Prerequisites 20============= 21 22In this use case we'll be using cmake on a Debian-based Linux system, 23cross-compiling from an x86_64 host to a hard-float Armv7-A target. We'll be 24using as many of the LLVM tools as we can, but it is possible to use GNU 25equivalents. 26 27 * ``A build of LLVM/clang for the llvm-tools and llvm-config`` 28 * ``A clang executable with support for the ARM target`` 29 * ``compiler-rt sources`` 30 * ``The qemu-arm user mode emulator`` 31 * ``An arm-linux-gnueabihf sysroot`` 32 33In this example we will be using ninja. 34 35See https://compiler-rt.llvm.org/ for more information about the dependencies 36on clang and LLVM. 37 38See https://llvm.org/docs/GettingStarted.html for information about obtaining 39the source for LLVM and compiler-rt. Note that the getting started guide 40places compiler-rt in the projects subdirectory, but this is not essential and 41if you are using the BaremetalARM.cmake cache for v6-M, v7-M and v7-EM then 42compiler-rt must be placed in the runtimes directory. 43 44``qemu-arm`` should be available as a package for your Linux distribution. 45 46The most complicated of the prerequisites to satisfy is the arm-linux-gnueabihf 47sysroot. In theory it is possible to use the Linux distributions multiarch 48support to fulfill the dependencies for building but unfortunately due to 49/usr/local/include being added some host includes are selected. The easiest way 50to supply a sysroot is to download the arm-linux-gnueabihf toolchain. This can 51be found at: 52* https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads for gcc 8 and above 53* https://releases.linaro.org/components/toolchain/binaries/ for gcc 4.9 to 7.3 54 55Building compiler-rt builtins for Arm 56===================================== 57We will be doing a standalone build of compiler-rt using the following cmake 58options. 59 60* ``path/to/compiler-rt`` 61* ``-G Ninja`` 62* ``-DCMAKE_AR=/path/to/llvm-ar`` 63* ``-DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"`` 64* ``-DCMAKE_ASM_FLAGS="build-c-flags"`` 65* ``-DCMAKE_C_COMPILER=/path/to/clang`` 66* ``-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"`` 67* ``-DCMAKE_C_FLAGS="build-c-flags"`` 68* ``-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld"`` 69* ``-DCMAKE_NM=/path/to/llvm-nm`` 70* ``-DCMAKE_RANLIB=/path/to/llvm-ranlib`` 71* ``-DCOMPILER_RT_BUILD_BUILTINS=ON`` 72* ``-DCOMPILER_RT_BUILD_LIBFUZZER=OFF`` 73* ``-DCOMPILER_RT_BUILD_MEMPROF=OFF`` 74* ``-DCOMPILER_RT_BUILD_PROFILE=OFF`` 75* ``-DCOMPILER_RT_BUILD_SANITIZERS=OFF`` 76* ``-DCOMPILER_RT_BUILD_XRAY=OFF`` 77* ``-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON`` 78* ``-DLLVM_CONFIG_PATH=/path/to/llvm-config`` 79 80The ``build-c-flags`` need to be sufficient to pass the C-make compiler check, 81compile compiler-rt, and if you are running the tests, compile and link the 82tests. When cross-compiling with clang we will need to pass sufficient 83information to generate code for the Arm architecture we are targeting. We will 84need to select the Arm target, select the Armv7-A architecture and choose 85between using Arm or Thumb. 86instructions. For example: 87 88* ``--target=arm-linux-gnueabihf`` 89* ``-march=armv7a`` 90* ``-mthumb`` 91 92When using a GCC arm-linux-gnueabihf toolchain the following flags are 93needed to pick up the includes and libraries: 94 95* ``--gcc-toolchain=/path/to/dir/toolchain`` 96* ``--sysroot=/path/to/toolchain/arm-linux-gnueabihf/libc`` 97 98In this example we will be adding all of the command line options to both 99``CMAKE_C_FLAGS`` and ``CMAKE_ASM_FLAGS``. There are cmake flags to pass some of 100these options individually which can be used to simplify the ``build-c-flags``: 101 102* ``-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"`` 103* ``-DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"`` 104* ``-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=/path/to/dir/toolchain`` 105* ``-DCMAKE_SYSROOT=/path/to/dir/toolchain/arm-linux-gnueabihf/libc`` 106 107Once cmake has completed the builtins can be built with ``ninja builtins`` 108 109Testing compiler-rt builtins using qemu-arm 110=========================================== 111To test the builtins library we need to add a few more cmake flags to enable 112testing and set up the compiler and flags for test case. We must also tell 113cmake that we wish to run the tests on ``qemu-arm``. 114 115* ``-DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armhf/sysroot`` 116* ``-DCOMPILER_RT_INCLUDE_TESTS=ON`` 117* ``-DCOMPILER_RT_TEST_COMPILER="/path/to/clang"`` 118* ``-DCOMPILER_RT_TEST_COMPILER_CFLAGS="test-c-flags"`` 119 120The ``/path/to/armhf/sysroot`` should be the same as the one passed to 121``--sysroot`` in the "build-c-flags". 122 123The "test-c-flags" need to include the target, architecture, gcc-toolchain, 124sysroot and arm/thumb state. The additional cmake defines such as 125``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` do not apply when building the tests. If 126you have put all of these in "build-c-flags" then these can be repeated. If you 127wish to use lld to link the tests then add ``"-fuse-ld=lld``. 128 129Once cmake has completed the tests can be built and run using 130``ninja check-builtins`` 131 132Troubleshooting 133=============== 134 135The cmake try compile stage fails 136--------------------------------- 137At an early stage cmake will attempt to compile and link a simple C program to 138test if the toolchain is working. 139 140This stage can often fail at link time if the ``--sysroot=`` and 141``--gcc-toolchain=`` options are not passed to the compiler. Check the 142``CMAKE_C_FLAGS`` and ``CMAKE_C_COMPILER_TARGET`` flags. 143 144It can be useful to build a simple example outside of cmake with your toolchain 145to make sure it is working. For example: ``clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c`` 146 147Clang uses the host header files 148-------------------------------- 149On debian based systems it is possible to install multiarch support for 150arm-linux-gnueabi and arm-linux-gnueabihf. In many cases clang can successfully 151use this multiarch support when ``--gcc-toolchain=`` and ``--sysroot=`` are not supplied. 152Unfortunately clang adds ``/usr/local/include`` before 153``/usr/include/arm-linux-gnueabihf`` leading to errors when compiling the hosts 154header files. 155 156The multiarch support is not sufficient to build the builtins you will need to 157use a separate arm-linux-gnueabihf toolchain. 158 159No target passed to clang 160------------------------- 161If clang is not given a target it will typically use the host target, this will 162not understand the Arm assembly language files resulting in error messages such 163as ``error: unknown directive .syntax unified``. 164 165You can check the clang invocation in the error message to see if there is no 166``--target`` or if it is set incorrectly. The cause is usually 167``CMAKE_ASM_FLAGS`` not containing ``--target`` or ``CMAKE_ASM_COMPILER_TARGET`` not being present. 168 169Arm architecture not given 170-------------------------- 171The ``--target=arm-linux-gnueabihf`` will default to arm architecture v4t which 172cannot assemble the barrier instructions used in the synch_and_fetch source 173files. 174 175The cause is usually a missing ``-march=armv7a`` from the ``CMAKE_ASM_FLAGS``. 176 177Compiler-rt builds but the tests fail to build 178---------------------------------------------- 179The flags used to build the tests are not the same as those used to build the 180builtins. The c flags are provided by ``COMPILER_RT_TEST_COMPILE_CFLAGS`` and 181the ``CMAKE_C_COMPILER_TARGET``, ``CMAKE_ASM_COMPILER_TARGET``, 182``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` and ``CMAKE_SYSROOT`` flags are not 183applied. 184 185Make sure that ``COMPILER_RT_TEST_COMPILE_CFLAGS`` contains all the necessary 186information. 187 188 189Modifications for other Targets 190=============================== 191 192Arm Soft-Float Target 193--------------------- 194The instructions for the Arm hard-float target can be used for the soft-float 195target by substituting soft-float equivalents for the sysroot and target. The 196target to use is: 197 198* ``-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi`` 199 200Depending on whether you want to use floating point instructions or not you 201may need extra c-flags such as ``-mfloat-abi=softfp`` for use of floating-point 202instructions, and ``-mfloat-abi=soft -mfpu=none`` for software floating-point 203emulation. 204 205You will need to use an arm-linux-gnueabi GNU toolchain for soft-float. 206 207AArch64 Target 208-------------- 209The instructions for Arm can be used for AArch64 by substituting AArch64 210equivalents for the sysroot, emulator and target. 211 212* ``-DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu`` 213* ``-DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot`` 214 215The CMAKE_C_FLAGS and COMPILER_RT_TEST_COMPILER_CFLAGS may also need: 216``"--sysroot=/path/to/aarch64/sysroot --gcc-toolchain=/path/to/gcc-toolchain"`` 217 218Armv6-m, Armv7-m and Armv7E-M targets 219------------------------------------- 220To build and test the libraries using a similar method to Armv7-A is possible 221but more difficult. The main problems are: 222 223* There isn't a ``qemu-arm`` user-mode emulator for bare-metal systems. The ``qemu-system-arm`` can be used but this is significantly more difficult to setup. 224* The targets to compile compiler-rt have the suffix -none-eabi. This uses the BareMetal driver in clang and by default won't find the libraries needed to pass the cmake compiler check. 225 226As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions 227that are supported on Armv7-A we can still get most of the value of running the 228tests using the same ``qemu-arm`` that we used for Armv7-A by building and 229running the test cases for Armv7-A but using the builtins compiled for 230Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked 231into a binary and execute the tests correctly but it will not catch if the 232builtins use instructions that are supported on Armv7-A but not Armv6-M, 233Armv7-M and Armv7E-M. 234 235To get the cmake compile test to pass you will need to pass the libraries 236needed to successfully link the cmake test via ``CMAKE_CFLAGS``. It is 237strongly recommended that you use version 3.6 or above of cmake so you can use 238``CMAKE_TRY_COMPILE_TARGET=STATIC_LIBRARY`` to skip the link step. 239 240* ``-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY`` 241* ``-DCOMPILER_RT_OS_DIR="baremetal"`` 242* ``-DCOMPILER_RT_BUILD_BUILTINS=ON`` 243* ``-DCOMPILER_RT_BUILD_SANITIZERS=OFF`` 244* ``-DCOMPILER_RT_BUILD_XRAY=OFF`` 245* ``-DCOMPILER_RT_BUILD_LIBFUZZER=OFF`` 246* ``-DCOMPILER_RT_BUILD_PROFILE=OFF`` 247* ``-DCMAKE_C_COMPILER=${host_install_dir}/bin/clang`` 248* ``-DCMAKE_C_COMPILER_TARGET="your *-none-eabi target"`` 249* ``-DCMAKE_ASM_COMPILER_TARGET="your *-none-eabi target"`` 250* ``-DCMAKE_AR=/path/to/llvm-ar`` 251* ``-DCMAKE_NM=/path/to/llvm-nm`` 252* ``-DCMAKE_RANLIB=/path/to/llvm-ranlib`` 253* ``-DCOMPILER_RT_BAREMETAL_BUILD=ON`` 254* ``-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON`` 255* ``-DLLVM_CONFIG_PATH=/path/to/llvm-config`` 256* ``-DCMAKE_C_FLAGS="build-c-flags"`` 257* ``-DCMAKE_ASM_FLAGS="build-c-flags"`` 258* ``-DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armv7-A/sysroot"`` 259* ``-DCOMPILER_RT_INCLUDE_TESTS=ON`` 260* ``-DCOMPILER_RT_TEST_COMPILER="/path/to/clang"`` 261* ``-DCOMPILER_RT_TEST_COMPILER_CFLAGS="test-c-flags"`` 262 263The Armv6-M builtins will use the soft-float ABI. When compiling the tests for 264Armv7-A we must include ``"-mthumb -mfloat-abi=soft -mfpu=none"`` in the 265test-c-flags. We must use an Armv7-A soft-float abi sysroot for ``qemu-arm``. 266 267Depending on the linker used for the test cases you may encounter BuildAttribute 268mismatches between the M-profile objects from compiler-rt and the A-profile 269objects from the test. The lld linker does not check the profile 270BuildAttribute so it can be used to link the tests by adding -fuse-ld=lld to the 271``COMPILER_RT_TEST_COMPILER_CFLAGS``. 272 273Alternative using a cmake cache 274------------------------------- 275If you wish to build, but not test compiler-rt for Armv6-M, Armv7-M or Armv7E-M 276the easiest way is to use the BaremetalARM.cmake recipe in clang/cmake/caches. 277 278You will need a bare metal sysroot such as that provided by the GNU ARM 279Embedded toolchain. 280 281The libraries can be built with the cmake options: 282 283* ``-DBAREMETAL_ARMV6M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi`` 284* ``-DBAREMETAL_ARMV7M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi`` 285* ``-DBAREMETAL_ARMV7EM_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi`` 286* ``-C /path/to/llvm/source/tools/clang/cmake/caches/BaremetalARM.cmake`` 287* ``/path/to/llvm`` 288 289**Note** that for the recipe to work the compiler-rt source must be checked out 290into the directory llvm/runtimes. You will also need clang and lld checked out. 291 292