Lines Matching +full:lit +full:- +full:tests
13 libc++ uses LIT to configure and run its tests.
15 The primary way to run the libc++ tests is by using ``make check-cxx``.
18 configurations it is important to customize the way LIT builds and runs
19 the tests. This guide provides information on how to use LIT directly to
22 Please see the `Lit Command Guide`_ for more information about LIT.
24 .. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
27 -----
30 running ``llvm-lit`` on a specified test or directory. If you're unsure
32 ``cxx-test-depends`` target. For example:
34 .. code-block:: bash
36 $ cd <monorepo-root>
37 $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt
38 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
39 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
40 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
42 If you used **ninja** as your build system, running ``ninja -C <build> check-cxx`` will run
43 all the tests in the libc++ testsuite.
47 ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and
48 you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the
49 paths of all tests. For example, to run all the libcxx tests you can do
50 ``<build>/bin/llvm-lit -sv <build>/runtimes/runtimes-bins/libcxx/test``.
52 In the default configuration, the tests are built against headers that form a
54 changes are made to the headers, so you should re-run the ``cxx-test-depends``
55 target before running the tests manually with ``lit`` when you make any sort of
56 change, including to the headers. We recommend using the provided ``libcxx/utils/libcxx-lit``
60 .. code-block:: bash
62 $ cd <monorepo-root>
63 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/re # Build testing dependencies and run all of the std::regex tests
65 Sometimes you'll want to change the way LIT is running the tests. Custom options
66 can be specified using the ``--param <name>=<val>`` flag. The most common option
67 you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the
71 .. code-block:: bash
73 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers # Run the tests with the newest -std
74 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03
81 "base" parameters (via Lit substitutions). These base parameters represent things
82 like the compiler to use for running the tests, which default compiler and linker
94 ---------------------------------
97 the current CMake configuration. It does so by generating a ``lit.site.cfg``
99 ``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around
100 ``llvm/utils/lit/lit.py``) to that file. So when you're running
101 ``<build>/bin/llvm-lit`` either directly or indirectly, the generated ``lit.site.cfg``
102 file is always loaded instead of ``libcxx/test/lit.cfg.py``. If you want to use a
104 ``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration
108 .. code-block:: bash
110 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>
111 $ libcxx/utils/libcxx-lit <build> -sv libcxx/test # will use your custom config file
114 ----------------
119 - clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks)
122 -----------------------------
124 Libc++ has extensive CI that tests various configurations of the library. The testing for
125 all these configurations is located in ``libcxx/utils/ci/run-buildbot``. Most of our
129 matches our CI images by running ``libcxx/utils/ci/run-buildbot-container``, and then run
130 the specific CI job that you're interested in (from within the container) using the ``run-buildbot``
132 ``CXX`` environment variables before calling ``run-buildbot`` to select the right compiler.
141 Writing Tests
144 When writing tests for the libc++ test suite, you should follow a few guidelines.
145 This will ensure that your tests can run on a wide variety of hardware and under
147 building the tests on one host but running them on a different host, which add a
150 - All tests are run in a temporary directory that is unique to that test and
152 - When a test needs data files as inputs, these data files can be saved in the
154 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
157 - You should never hardcode a path from the build-host in a test, because that
158 path will not necessarily be available on the host where the tests are run.
159 - You should try to reduce the runtime dependencies of each test to the minimum.
161 necessarily available on all devices we may want to run the tests on (even
162 though supporting Python is probably trivial for the build-host).
165 --------------------------------------------
167 The tests of libc++ are stored in libc++'s testing related subdirectories:
169 - ``libcxx/test/support`` This directory contains several helper headers with
170 generic parts for the tests. The most important header is ``test_macros.h``.
173 Since libc++'s tests are used by other Standard libraries, tests should use
176 - ``libcxx/test/std`` This directory contains the tests that validate the library under
179 reorganises its structure, therefore some tests are at a location based on
181 between keeping things at up-to-date locations and unnecessary churn.
182 - ``libcxx/test/libcxx`` This directory contains the tests that validate libc++
185 libc++ and not mandated by the Standard, tests for those are located under
190 -------------------
196 .. code-block:: cpp
208 .. code-block:: cpp
222 Tests in libc++ mainly use ``assert`` and ``static_assert`` for testing. There
224 write common tests.
232 helper function. For example the ``std::format`` tests use a helper function
241 - if it is a ``const char*`` or ``std::string`` its contents are written to
243 - otherwise it must be a callable that is invoked without any additional
247 either by supplying a hard-coded string or generate it at runtime.
253 use-case is to fail when code is reached that should be unreachable.
267 is a no-op. This makes it possible to test libc++ specific behaviour. For
289 tests using exceptions. The code to write a test manually would be:
292 .. code-block:: cpp
295 #ifndef TEST_HAS_NO_EXCEPTIONS // do nothing when tests are disabled
309 .. code-block:: cpp
326 ``stderr``. This is useful for tests where the arguments use different
335 Since it is not expected to add this to existing tests no effort was
340 ----------
342 The names of test files have meaning for the libc++-specific configuration of
343 Lit. Based on the pattern that matches the name of a test file, Lit will test
344 the code contained therein in different ways. Refer to the `Lit Meaning of libc++
348 .. list-table:: Lit Meaning of libc++ Test Filenames
350 :header-rows: 1
352 * - Name Pattern
353 - Meaning
354 * - ``FOO.pass.cpp``
355 - Checks whether the C++ code in the file compiles, links and runs successfully.
356 * - ``FOO.pass.mm``
357 - Same as ``FOO.pass.cpp``, but for Objective-C++.
359 * - ``FOO.compile.pass.cpp``
360 - Checks whether the C++ code in the file compiles successfully. In general, prefer ``compile`` tests over ``verify`` tests,
361 subject to the specific recommendations, below, for when to write ``verify`` tests.
362 * - ``FOO.compile.pass.mm``
363 - Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
364 * - ``FOO.compile.fail.cpp``
365 - Checks that the code in the file does *not* compile successfully.
367 * - ``FOO.verify.cpp``
368 - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
369 For additional information about how to write ``verify`` tests, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
370 Prefer `verify` tests over ``compile`` tests to test that compilation fails for a particular reason. For example, use a ``verify`` test
378 * - ``FOO.link.pass.cpp``
379 - Checks that the C++ code in the file compiles and links successfully -- no run attempted.
380 * - ``FOO.link.pass.mm``
381 - Same as ``FOO.link.pass.cpp``, but for Objective-C++.
382 * - ``FOO.link.fail.cpp``
383 - Checks whether the C++ code in the file fails to link after successful compilation.
384 * - ``FOO.link.fail.mm``
385 - Same as ``FOO.link.fail.cpp``, but for Objective-C++.
387 * - ``FOO.sh.<anything>``
388 - A *builtin Lit Shell* test.
389 * - ``FOO.gen.<anything>``
390 - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
391 by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
392 of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
395 * - ``FOO.bench.cpp``
396 - A benchmark test. These tests are linked against the GoogleBenchmark library and generally consist of micro-benchmarks of individual
400 libc++-Specific Lit Features
401 ----------------------------
406 Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes
407 writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES``, ``ADDITIONAL_COMPILE_FLAGS``, and ``MODULE_DEPENDENCIES`` libc++-specific directives.
409 .. _libc++-specific Lit Directives:
410 .. list-table:: libc++-specific Lit Directives
412 :header-rows: 1
414 * - Directive
415 - Parameters
416 - Usage
417 * - ``FILE_DEPENDENCIES``
418 - ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``
419 - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test
420 input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy
422 (i.e. the source directory of the test being executed) which makes it possible to use relative paths to specify the location of dependency files. After Lit copies
425 * - ``ADDITIONAL_COMPILE_FLAGS``
426 - ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...``
427 - The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}``
428 substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see
429 `Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill.
430 * - ``MODULE_DEPENDENCIES``
431 - ``// MODULE_DEPENDENCIES: std std.compat``
432 - This directive will build the required C++23 standard library
438 C++ Standard version tests
441 Historically libc++ tests used to filter the tests for C++ Standard versions
442 with lit directives like:
444 .. code-block:: cpp
451 .. code-block:: cpp
453 // UNSUPPORTED: std-at-least-c++26
455 There is no corresponding ``std-at-most-c++23``. This could be useful when
456 tests are only valid for a small set of standard versions. For example, a
458 removed from the Standard. These tests should be written like:
460 .. code-block:: cpp
466 There are a lot of tests with the first style, these can remain as they are.
467 The new style is only intended to be used for new tests.
473 Libc++ contains benchmark tests separately from the test of the test suite.
481 works in the same way as running a test. Both the benchmarks and the tests share
485 .. code-block:: bash
487 $ libcxx/utils/libcxx-lit <build> libcxx/test/benchmarks/string.bench.cpp --show-all --param optimization=speed
489 Note that benchmarks are only dry-run when run via the ``check-cxx`` target since
491 run through ``check-cxx`` for anything, instead run the benchmarks manually using
492 the instructions for running individual tests.
495 ``libcxx-compare-benchmarks`` helper tool. First, configure CMake in a build directory
498 .. code-block:: bash
500 $ cmake -S runtimes -B <build1> [...]
501 $ libcxx/utils/libcxx-lit <build1> libcxx/test/benchmarks/string.bench.cpp --param optimization=speed
506 .. code-block:: bash
508 $ cmake -S runtimes -B <build2> [...]
509 $ libcxx/utils/libcxx-lit <build2> libcxx/test/benchmarks/string.bench.cpp --param optimization=speed
511 Finally, use ``libcxx-compare-benchmarks`` to compare both:
513 .. code-block:: bash
515 $ libcxx/utils/libcxx-compare-benchmarks <build1> <build2> libcxx/test/benchmarks/string.bench.cpp
519 .. _testing-hardening-assertions:
524 Each hardening assertion should be tested using death tests (via the
525 ``TEST_LIBCPP_ASSERT_FAILURE`` macro). Use the ``libcpp-hardening-mode`` Lit
531 .. code-block:: cpp
535 // REQUIRES: has-unix-headers
537 // XFAIL: libcpp-hardening-mode=debug && availability-verbose_abort-missing
540 // UNSUPPORTED: libcpp-hardening-mode=none
542 // REQUIRES: libcpp-hardening-mode=debug
544 // REQUIRES: libcpp-hardening-mode={{extensive|debug}}
552 int bad_input = -1;