1============== 2Testing libc++ 3============== 4 5.. contents:: 6 :local: 7 8Getting Started 9=============== 10 11libc++ uses LIT to configure and run its tests. 12 13The primary way to run the libc++ tests is by using ``make check-cxx``. 14 15However since libc++ can be used in any number of possible 16configurations it is important to customize the way LIT builds and runs 17the tests. This guide provides information on how to use LIT directly to 18test libc++. 19 20Please see the `Lit Command Guide`_ for more information about LIT. 21 22.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html 23 24Usage 25----- 26 27After building libc++, you can run parts of the libc++ test suite by simply 28running ``llvm-lit`` on a specified test or directory. If you're unsure 29whether the required libraries have been built, you can use the 30``cxx-test-depends`` target. For example: 31 32.. code-block:: bash 33 34 $ cd <monorepo-root> 35 $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt 36 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests 37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic 39 40.. note:: 41 If you used the Bootstrapping build instead of the default runtimes build, the 42 ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and 43 you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the 44 paths of all tests. 45 46In the default configuration, the tests are built against headers that form a 47fake installation root of libc++. This installation root has to be updated when 48changes are made to the headers, so you should re-run the ``cxx-test-depends`` 49target before running the tests manually with ``lit`` when you make any sort of 50change, including to the headers. 51 52Sometimes you'll want to change the way LIT is running the tests. Custom options 53can be specified using the ``--param <name>=<val>`` flag. The most common option 54you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the 55test suite will select the newest C++ dialect supported by the compiler and use 56that. However, you can manually specify the option like so if you want: 57 58.. code-block:: bash 59 60 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std 61 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03 62 63Other parameters are supported by the test suite. Those are defined in ``libcxx/utils/libcxx/test/params.py``. 64If you want to customize how to run the libc++ test suite beyond what is available 65in ``params.py``, you most likely want to use a custom site configuration instead. 66 67The libc++ test suite works by loading a site configuration that defines various 68"base" parameters (via Lit substitutions). These base parameters represent things 69like the compiler to use for running the tests, which default compiler and linker 70flags to use, and how to run an executable. This system is meant to be easily 71extended for custom needs, in particular when porting the libc++ test suite to 72new platforms. 73 74Using a Custom Site Configuration 75--------------------------------- 76 77By default, the libc++ test suite will use a site configuration that matches 78the current CMake configuration. It does so by generating a ``lit.site.cfg`` 79file in the build directory from one of the configuration file templates in 80``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around 81``llvm/utils/lit/lit.py``) to that file. So when you're running 82``<build>/bin/llvm-lit``, the generated ``lit.site.cfg`` file is always loaded 83instead of ``libcxx/test/lit.cfg.py``. If you want to use a custom site 84configuration, simply point the CMake build to it using 85``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration 86will be used instead. That file can use CMake variables inside it to make 87configuration easier. 88 89 .. code-block:: bash 90 91 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config> 92 $ make -C <build> cxx-test-depends 93 $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file 94 95Additional tools 96---------------- 97 98The libc++ test suite uses a few optional tools to improve the code quality. 99 100These tools are: 101- clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks) 102 103Writing Tests 104------------- 105 106When writing tests for the libc++ test suite, you should follow a few guidelines. 107This will ensure that your tests can run on a wide variety of hardware and under 108a wide variety of configurations. We have several unusual configurations such as 109building the tests on one host but running them on a different host, which add a 110few requirements to the test suite. Here's some stuff you should know: 111 112- All tests are run in a temporary directory that is unique to that test and 113 cleaned up after the test is done. 114- When a test needs data files as inputs, these data files can be saved in the 115 repository (when reasonable) and referenced by the test as 116 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or 117 directories will be made available to the test in the temporary directory 118 where it is run. 119- You should never hardcode a path from the build-host in a test, because that 120 path will not necessarily be available on the host where the tests are run. 121- You should try to reduce the runtime dependencies of each test to the minimum. 122 For example, requiring Python to run a test is bad, since Python is not 123 necessarily available on all devices we may want to run the tests on (even 124 though supporting Python is probably trivial for the build-host). 125 126Benchmarks 127========== 128 129Libc++ contains benchmark tests separately from the test of the test suite. 130The benchmarks are written using the `Google Benchmark`_ library, a copy of which 131is stored in the libc++ repository. 132 133For more information about using the Google Benchmark library see the 134`official documentation <https://github.com/google/benchmark>`_. 135 136.. _`Google Benchmark`: https://github.com/google/benchmark 137 138Building Benchmarks 139------------------- 140 141The benchmark tests are not built by default. The benchmarks can be built using 142the ``cxx-benchmarks`` target. 143 144An example build would look like: 145 146.. code-block:: bash 147 148 $ cd build 149 $ ninja cxx-benchmarks 150 151This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 152built against the just-built libc++. The compiled tests are output into 153``build/projects/libcxx/benchmarks``. 154 155The benchmarks can also be built against the platforms native standard library 156using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 157is useful for comparing the performance of libc++ to other standard libraries. 158The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 159``<test>.native.out`` otherwise. 160 161Also See: 162 163 * :ref:`Building Libc++ <build instructions>` 164 * :ref:`CMake Options` 165 166Running Benchmarks 167------------------ 168 169The benchmarks must be run manually by the user. Currently there is no way 170to run them as part of the build. 171 172For example: 173 174.. code-block:: bash 175 176 $ cd build/projects/libcxx/benchmarks 177 $ ./algorithms.libcxx.out # Runs all the benchmarks 178 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 179 180For more information about running benchmarks see `Google Benchmark`_. 181