Lines Matching +full:libc +full:- +full:build
4 Testing libc++
13 libc++ uses LIT to configure and run its tests.
15 The primary way to run the libc++ tests is by using ``make check-cxx``.
17 However since libc++ can be used in any number of possible
20 test libc++.
27 -----
29 After :ref:`building libc++ <VendorDocumentation>`, you can run parts of the libc++ test suite by simply
30 running ``llvm-lit`` on a specified test or directory. If you're unsure
32 ``cxx-test-depends`` target. For example:
34 .. code-block:: bash
36 $ cd <monorepo-root>
37 $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt
38 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
39 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
40 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
42 If you used **ninja** as your build system, running ``ninja -C <build> check-cxx`` will run
43 all the tests in the libc++ testsuite.
46 If you used the Bootstrapping build instead of the default runtimes build, the
47 ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and
48 you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the
50 ``<build>/bin/llvm-lit -sv <build>/runtimes/runtimes-bins/libcxx/test``.
53 fake installation root of libc++. This installation root has to be updated when
54 changes are made to the headers, so you should re-run the ``cxx-test-depends``
56 change, including to the headers. We recommend using the provided ``libcxx/utils/libcxx-lit``
60 .. code-block:: bash
62 $ cd <monorepo-root>
63 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/re # Build testing dependencies and run all of the std::regex tests
66 can be specified using the ``--param <name>=<val>`` flag. The most common option
67 you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the
71 .. code-block:: bash
73 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers # Run the tests with the newest -std
74 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03
77 If you want to customize how to run the libc++ test suite beyond what is available
80 The libc++ test suite works by loading a site configuration that defines various
84 extended for custom needs, in particular when porting the libc++ test suite to
94 ---------------------------------
96 By default, the libc++ test suite will use a site configuration that matches
98 file in the build directory from one of the configuration file templates in
99 ``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around
101 ``<build>/bin/llvm-lit`` either directly or indirectly, the generated ``lit.site.cfg``
103 custom site configuration, simply point the CMake build to it using
104 ``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration
108 .. code-block:: bash
110 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>
111 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test # will use your custom config file
114 ----------------
116 The libc++ test suite uses a few optional tools to improve the code quality.
119 - clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks)
122 -----------------------------
124 Libc++ has extensive CI that tests various configurations of the library. The testing for
125 all these configurations is located in ``libcxx/utils/ci/run-buildbot``. Most of our
129 matches our CI images by running ``libcxx/utils/ci/run-buildbot-container``, and then run
130 the specific CI job that you're interested in (from within the container) using the ``run-buildbot``
132 ``CXX`` environment variables before calling ``run-buildbot`` to select the right compiler.
144 When writing tests for the libc++ test suite, you should follow a few guidelines.
150 - All tests are run in a temporary directory that is unique to that test and
152 - When a test needs data files as inputs, these data files can be saved in the
154 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
157 - You should never hardcode a path from the build-host in a test, because that
159 - You should try to reduce the runtime dependencies of each test to the minimum.
162 though supporting Python is probably trivial for the build-host).
165 --------------------------------------------
167 The tests of libc++ are stored in libc++'s testing related subdirectories:
169 - ``libcxx/test/support`` This directory contains several helper headers with
172 This is similar to the ``__config`` file in libc++'s ``include`` directory.
173 Since libc++'s tests are used by other Standard libraries, tests should use
175 specific to libc++.
176 - ``libcxx/test/std`` This directory contains the tests that validate the library under
181 between keeping things at up-to-date locations and unnecessary churn.
182 - ``libcxx/test/libcxx`` This directory contains the tests that validate libc++
183 specific behavior and implementation details. For example, libc++ has
185 libc++ and not mandated by the Standard, tests for those are located under
190 -------------------
192 Some platforms where libc++ is tested have requirement on the signature of
196 .. code-block:: cpp
208 .. code-block:: cpp
222 Tests in libc++ mainly use ``assert`` and ``static_assert`` for testing. There
241 - if it is a ``const char*`` or ``std::string`` its contents are written to
243 - otherwise it must be a callable that is invoked without any additional
247 either by supplying a hard-coded string or generate it at runtime.
253 use-case is to fail when code is reached that should be unreachable.
266 If the library under test is libc++ it behaves like ``TEST_REQUIRE``, else it
267 is a no-op. This makes it possible to test libc++ specific behaviour. For
268 example testing whether the ``what()`` of an exception thrown matches libc++'s
292 .. code-block:: cpp
309 .. code-block:: cpp
334 was added at a time where most of libc++'s C++17 support was complete.
340 ----------
342 The names of test files have meaning for the libc++-specific configuration of
344 the code contained therein in different ways. Refer to the `Lit Meaning of libc++
347 .. _Lit Meaning of libc++ Test Filenames:
348 .. list-table:: Lit Meaning of libc++ Test Filenames
350 :header-rows: 1
352 * - Name Pattern
353 - Meaning
354 * - ``FOO.pass.cpp``
355 - Checks whether the C++ code in the file compiles, links and runs successfully.
356 * - ``FOO.pass.mm``
357 - Same as ``FOO.pass.cpp``, but for Objective-C++.
359 * - ``FOO.compile.pass.cpp``
360 - Checks whether the C++ code in the file compiles successfully. In general, prefer ``compile`` tests over ``verify`` tests,
362 * - ``FOO.compile.pass.mm``
363 - Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
364 * - ``FOO.compile.fail.cpp``
365 - Checks that the code in the file does *not* compile successfully.
367 * - ``FOO.verify.cpp``
368 - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
369 For additional information about how to write ``verify`` tests, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
378 * - ``FOO.link.pass.cpp``
379 - Checks that the C++ code in the file compiles and links successfully -- no run attempted.
380 * - ``FOO.link.pass.mm``
381 - Same as ``FOO.link.pass.cpp``, but for Objective-C++.
382 * - ``FOO.link.fail.cpp``
383 - Checks whether the C++ code in the file fails to link after successful compilation.
384 * - ``FOO.link.fail.mm``
385 - Same as ``FOO.link.fail.cpp``, but for Objective-C++.
387 * - ``FOO.sh.<anything>``
388 - A *builtin Lit Shell* test.
389 * - ``FOO.gen.<anything>``
390 - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
391 by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
395 * - ``FOO.bench.cpp``
396 - A benchmark test. These tests are linked against the GoogleBenchmark library and generally consist of micro-benchmarks of individual
400 libc++-Specific Lit Features
401 ----------------------------
406 Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes
407 writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES``, ``ADDITIONAL_COMPILE_FLAGS``, and ``MODULE_DEPENDENCIES`` libc++-specific directives.
409 .. _libc++-specific Lit Directives:
410 .. list-table:: libc++-specific Lit Directives
412 :header-rows: 1
414 * - Directive
415 - Parameters
416 - Usage
417 * - ``FILE_DEPENDENCIES``
418 - ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``
419 - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test
420 input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy
425 * - ``ADDITIONAL_COMPILE_FLAGS``
426 - ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...``
427 - The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}``
428 substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see
429 `Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill.
430 * - ``MODULE_DEPENDENCIES``
431 - ``// MODULE_DEPENDENCIES: std std.compat``
432 - This directive will build the required C++23 standard library
434 %{compile_flags}. (Libc++ offers these modules in C++20 as an
441 Historically libc++ tests used to filter the tests for C++ Standard versions
444 .. code-block:: cpp
451 .. code-block:: cpp
453 // UNSUPPORTED: std-at-least-c++26
455 There is no corresponding ``std-at-most-c++23``. This could be useful when
460 .. code-block:: cpp
473 Libc++ contains benchmark tests separately from the test of the test suite.
475 is stored in the libc++ repository.
485 .. code-block:: bash
487 $ libcxx/utils/libcxx-lit <build> libcxx/test/benchmarks/string.bench.cpp --show-all --param optimization=speed
489 Note that benchmarks are only dry-run when run via the ``check-cxx`` target since
491 run through ``check-cxx`` for anything, instead run the benchmarks manually using
495 ``libcxx-compare-benchmarks`` helper tool. First, configure CMake in a build directory
498 .. code-block:: bash
500 $ cmake -S runtimes -B <build1> [...]
501 $ libcxx/utils/libcxx-lit <build1> libcxx/test/benchmarks/string.bench.cpp --param optimization=speed
503 Then, do the same for the second configuration you want to test. Use a different build
506 .. code-block:: bash
508 $ cmake -S runtimes -B <build2> [...]
509 $ libcxx/utils/libcxx-lit <build2> libcxx/test/benchmarks/string.bench.cpp --param optimization=speed
511 Finally, use ``libcxx-compare-benchmarks`` to compare both:
513 .. code-block:: bash
515 $ libcxx/utils/libcxx-compare-benchmarks <build1> <build2> libcxx/test/benchmarks/string.bench.cpp
519 .. _testing-hardening-assertions:
525 ``TEST_LIBCPP_ASSERT_FAILURE`` macro). Use the ``libcpp-hardening-mode`` Lit
531 .. code-block:: cpp
535 // REQUIRES: has-unix-headers
537 // XFAIL: libcpp-hardening-mode=debug && availability-verbose_abort-missing
540 // UNSUPPORTED: libcpp-hardening-mode=none
542 // REQUIRES: libcpp-hardening-mode=debug
544 // REQUIRES: libcpp-hardening-mode={{extensive|debug}}
552 int bad_input = -1;