1.. _libc_gpu_building: 2 3====================== 4Building libs for GPUs 5====================== 6 7.. contents:: Table of Contents 8 :depth: 4 9 :local: 10 11Building the GPU C library 12========================== 13 14This document will present recipes to build the LLVM C library targeting a GPU 15architecture. The GPU build uses the same :ref:`cross build<full_cross_build>` 16support as the other targets. However, the GPU target has the restriction that 17it *must* be built with an up-to-date ``clang`` compiler. This is because the 18GPU target uses several compiler extensions to target GPU architectures. 19 20The LLVM C library currently supports two GPU targets. This is either 21``nvptx64-nvidia-cuda`` for NVIDIA GPUs or ``amdgcn-amd-amdhsa`` for AMD GPUs. 22Targeting these architectures is done through ``clang``'s cross-compiling 23support using the ``--target=<triple>`` flag. The following sections will 24describe how to build the GPU support specifically. 25 26Once you have finished building, refer to :ref:`libc_gpu_usage` to get started 27with the newly built C library. 28 29Standard runtimes build 30----------------------- 31 32The simplest way to build the GPU libc is to use the existing LLVM runtimes 33support. This will automatically handle bootstrapping an up-to-date ``clang`` 34compiler and using it to build the C library. The following CMake invocation 35will instruct it to build the ``libc`` runtime targeting both AMD and NVIDIA 36GPUs. The ``LIBC_GPU_BUILD`` option can also be enabled to add the relevant 37arguments automatically. 38 39.. code-block:: sh 40 41 $> cd llvm-project # The llvm-project checkout 42 $> mkdir build 43 $> cd build 44 $> cmake ../llvm -G Ninja \ 45 -DLLVM_ENABLE_PROJECTS="clang;lld" \ 46 -DLLVM_ENABLE_RUNTIMES="openmp" \ 47 -DCMAKE_BUILD_TYPE=<Debug|Release> \ # Select build type 48 -DCMAKE_INSTALL_PREFIX=<PATH> \ # Where the libraries will live 49 -DRUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES=libc \ 50 -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=libc \ 51 -DLLVM_RUNTIME_TARGETS="default;amdgcn-amd-amdhsa;nvptx64-nvidia-cuda" 52 $> ninja install 53 54We need ``clang`` to build the GPU C library and ``lld`` to link AMDGPU 55executables, so we enable them in ``LLVM_ENABLE_PROJECTS``. We add ``openmp`` to 56``LLVM_ENABLED_RUNTIMES`` so it is built for the default target and provides 57OpenMP support. We then set ``RUNTIMES_<triple>_LLVM_ENABLE_RUNTIMES`` to enable 58``libc`` for the GPU targets. The ``LLVM_RUNTIME_TARGETS`` sets the enabled 59targets to build, in this case we want the default target and the GPU targets. 60Note that if ``libc`` were included in ``LLVM_ENABLE_RUNTIMES`` it would build 61targeting the default host environment as well. Alternatively, you can point 62your build towards the ``libc/cmake/caches/gpu.cmake`` cache file with ``-C``. 63 64Runtimes cross build 65-------------------- 66 67For users wanting more direct control over the build process, the build steps 68can be done manually instead. This build closely follows the instructions in the 69:ref:`main documentation<full_cross_build>` but is specialized for the GPU 70build. We follow the same steps to first build the libc tools and a suitable 71compiler. These tools must all be up-to-date with the libc source. 72 73.. code-block:: sh 74 75 $> cd llvm-project # The llvm-project checkout 76 $> mkdir build-libc-tools # A different build directory for the build tools 77 $> cd build-libc-tools 78 $> HOST_C_COMPILER=<C compiler for the host> # For example "clang" 79 $> HOST_CXX_COMPILER=<C++ compiler for the host> # For example "clang++" 80 $> cmake ../llvm \ 81 -G Ninja \ 82 -DLLVM_ENABLE_PROJECTS="clang" \ 83 -DCMAKE_C_COMPILER=$HOST_C_COMPILER \ 84 -DCMAKE_CXX_COMPILER=$HOST_CXX_COMPILER \ 85 -DLLVM_LIBC_FULL_BUILD=ON \ 86 -DCMAKE_BUILD_TYPE=Release # Release suggested to make "clang" fast 87 $> ninja # Build the 'clang' compiler 88 89Once this has finished the build directory should contain the ``clang`` 90compiler executable. We will use the ``clang`` compiler to build the GPU code. 91We use these tools to bootstrap the build out of the runtimes directory 92targeting a GPU architecture. 93 94.. code-block:: sh 95 96 $> cd llvm-project # The llvm-project checkout 97 $> mkdir build # A different build directory for the build tools 98 $> cd build 99 $> TARGET_TRIPLE=<amdgcn-amd-amdhsa or nvptx64-nvidia-cuda> 100 $> TARGET_C_COMPILER=</path/to/clang> 101 $> TARGET_CXX_COMPILER=</path/to/clang++> 102 $> cmake ../runtimes \ # Point to the runtimes build 103 -G Ninja \ 104 -DLLVM_ENABLE_RUNTIMES=libc \ 105 -DCMAKE_C_COMPILER=$TARGET_C_COMPILER \ 106 -DCMAKE_CXX_COMPILER=$TARGET_CXX_COMPILER \ 107 -DLLVM_LIBC_FULL_BUILD=ON \ 108 -DLLVM_RUNTIMES_TARGET=$TARGET_TRIPLE \ 109 -DCMAKE_BUILD_TYPE=Release 110 $> ninja install 111 112The above steps will result in a build targeting one of the supported GPU 113architectures. Building for multiple targets requires separate CMake 114invocations. 115 116Standalone cross build 117---------------------- 118 119The GPU build can also be targeted directly as long as the compiler used is a 120supported ``clang`` compiler. This method is generally not recommended as it can 121only target a single GPU architecture. 122 123.. code-block:: sh 124 125 $> cd llvm-project # The llvm-project checkout 126 $> mkdir build # A different build directory for the build tools 127 $> cd build 128 $> CLANG_C_COMPILER=</path/to/clang> # Must be a trunk build 129 $> CLANG_CXX_COMPILER=</path/to/clang++> # Must be a trunk build 130 $> TARGET_TRIPLE=<amdgcn-amd-amdhsa or nvptx64-nvidia-cuda> 131 $> cmake ../llvm \ # Point to the llvm directory 132 -G Ninja \ 133 -DLLVM_ENABLE_PROJECTS=libc \ 134 -DCMAKE_C_COMPILER=$CLANG_C_COMPILER \ 135 -DCMAKE_CXX_COMPILER=$CLANG_CXX_COMPILER \ 136 -DLLVM_LIBC_FULL_BUILD=ON \ 137 -DLIBC_TARGET_TRIPLE=$TARGET_TRIPLE \ 138 -DCMAKE_BUILD_TYPE=Release 139 $> ninja install 140 141This will build and install the GPU C library along with all the other LLVM 142libraries. 143 144Build overview 145============== 146 147Once installed, the GPU build will create several files used for different 148targets. This section will briefly describe their purpose. 149 150**include/<target-triple>** 151 The include directory where all of the generated headers for the target will 152 go. These definitions are strictly for the GPU when being targeted directly. 153 154**lib/clang/<llvm-major-version>/include/llvm-libc-wrappers/llvm-libc-decls** 155 These are wrapper headers created for offloading languages like CUDA, HIP, or 156 OpenMP. They contain functions supported in the GPU libc along with attributes 157 and metadata that declare them on the target device and make them compatible 158 with the host headers. 159 160**lib/<target-triple>/libc.a** 161 The main C library static archive containing LLVM-IR targeting the given GPU. 162 It can be linked directly or inspected depending on the target support. 163 164**lib/<target-triple>/libm.a** 165 The C library static archive providing implementations of the standard math 166 functions. 167 168**lib/<target-triple>/libc.bc** 169 An alternate form of the library provided as a single LLVM-IR bitcode blob. 170 This can be used similarly to NVIDIA's or AMD's device libraries. 171 172**lib/<target-triple>/libm.bc** 173 An alternate form of the library provided as a single LLVM-IR bitcode blob 174 containing the standard math functions. 175 176**lib/<target-triple>/crt1.o** 177 An LLVM-IR file containing startup code to call the ``main`` function on the 178 GPU. This is used similarly to the standard C library startup object. 179 180**bin/amdhsa-loader** 181 A binary utility used to launch executables compiled targeting the AMD GPU. 182 This will be included if the build system found the ``hsa-runtime64`` library 183 either in ``/opt/rocm`` or the current CMake installation directory. This is 184 required to build the GPU tests .See the :ref:`libc GPU usage<libc_gpu_usage>` 185 for more information. 186 187**bin/nvptx-loader** 188 A binary utility used to launch executables compiled targeting the NVIDIA GPU. 189 This will be included if the build system found the CUDA driver API. This is 190 required for building tests. 191 192**include/llvm-libc-rpc-server.h** 193 A header file containing definitions that can be used to interface with the 194 :ref:`RPC server<libc_gpu_rpc>`. 195 196**lib/libllvmlibc_rpc_server.a** 197 The static library containing the implementation of the RPC server. This can 198 be used to enable host services for anyone looking to interface with the 199 :ref:`RPC client<libc_gpu_rpc>`. 200 201.. _gpu_cmake_options: 202 203CMake options 204============= 205 206This section briefly lists a few of the CMake variables that specifically 207control the GPU build of the C library. These options can be passed individually 208to each target using ``-DRUNTIMES_<target>_<variable>=<value>`` when using a 209standard runtime build. 210 211**LLVM_LIBC_FULL_BUILD**:BOOL 212 This flag controls whether or not the libc build will generate its own 213 headers. This must always be on when targeting the GPU. 214 215**LIBC_GPU_BUILD**:BOOL 216 Shorthand for enabling GPU support. Equivalent to enabling support for both 217 AMDGPU and NVPTX builds for ``libc``. 218 219**LIBC_GPU_TEST_ARCHITECTURE**:STRING 220 Sets the architecture used to build the GPU tests for, such as ``gfx90a`` or 221 ``sm_80`` for AMD and NVIDIA GPUs respectively. The default behavior is to 222 detect the system's GPU architecture using the ``native`` option. If this 223 option is not set and a GPU was not detected the tests will not be built. 224 225**LIBC_GPU_TEST_JOBS**:STRING 226 Sets the number of threads used to run GPU tests. The GPU test suite will 227 commonly run out of resources if this is not constrained so it is recommended 228 to keep it low. The default value is a single thread. 229 230**LIBC_GPU_LOADER_EXECUTABLE**:STRING 231 Overrides the default loader used for running GPU tests. If this is not 232 provided the standard one will be built. 233