Lines Matching +full:lit +full:- +full:tests
20 tests.
32 The LLVM testing infrastructure contains three major categories of tests:
33 unit tests, regression tests and whole programs. The unit tests and regression
34 tests are contained inside the LLVM repository itself under ``llvm/unittests``
35 and ``llvm/test`` respectively and are expected to always pass -- they should be
38 The whole programs tests are referred to as the "LLVM test suite" (or
39 "test-suite") and are in the ``test-suite``
40 `repository on GitHub <https://github.com/llvm/llvm-test-suite.git>`_.
41 For historical reasons, these tests are also referred to as the "nightly
42 tests" in places, which is less ambiguous than "test-suite" and remains
45 Unit tests
46 ----------
48 Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/docs/primer.md>`_
51 In general unit tests are reserved for targeting the support library and other
52 generic data structure, we prefer relying on regression tests for testing
55 Regression tests
56 ----------------
58 The regression tests are small pieces of code that test a specific
60 written in depends on the part of LLVM being tested. These tests are driven by
61 the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
70 ----------------
74 regression tests, by creating a separate "Printer" pass to consume the analysis
77 See `llvm/test/Analysis/BranchProbabilityInfo/loop.ll <https://github.com/llvm/llvm-project/blob/main/llvm/test/Analysis/BranchProbabilityInfo/loop.ll>`_
80 ``test-suite``
81 --------------
84 can be compiled and linked into a stand-alone program that can be
93 In addition to compiling and executing programs, whole program tests
98 The test-suite is located in the ``test-suite``
99 `repository on GitHub <https://github.com/llvm/llvm-test-suite.git>`_.
103 Debugging Information tests
104 ---------------------------
106 The test suite contains tests to check quality of debugging information.
109 These tests are compiled and run under a debugger. The debugger output
112 ``cross-project-tests/debuginfo-tests`` directory.
117 The tests are located in two separate repositories. The unit and
118 regression tests are in the main "llvm"/ directory under the directories
119 ``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the
120 main LLVM tree). Use ``make check-all`` to run the unit and regression tests
123 The ``test-suite`` module contains more comprehensive tests including whole C
126 Unit and Regression tests
127 -------------------------
129 To run all of the LLVM unit tests use the check-llvm-unit target:
131 .. code-block:: bash
133 % make check-llvm-unit
135 To run all of the LLVM regression tests use the check-llvm target:
137 .. code-block:: bash
139 % make check-llvm
144 .. code-block:: bash
146 % cmake -DCMAKE_BUILD_TYPE="Release" -DLLVM_ENABLE_ASSERTIONS=On
149 can run the LLVM and Clang tests simultaneously using:
151 .. code-block:: bash
153 % make check-all
155 To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make
156 variable to pass the required options to lit. For example, you can use:
158 .. code-block:: bash
160 % make check LIT_ARGS="-v --vg --vg-leak"
164 To run individual tests or subsets of tests, you can use the ``llvm-lit``
168 .. code-block:: bash
170 % llvm-lit <path to llvm-project>/llvm/test/Integer/BitPacked.ll
173 The test files are in the ``llvm-project`` directory, not the directory you
176 Or you can run a whole folder of tests. To run all of the ARM CodeGen tests:
178 .. code-block:: bash
180 % llvm-lit <path to llvm-project>/llvm/test/CodeGen/ARM
182 The regression tests will use the Python psutil module only if installed in a
183 **non-user** location. Under Linux, install with sudo or within a virtual
187 For more information on using the :program:`lit` tool, see ``llvm-lit --help``
188 or the :doc:`lit man page <CommandGuide/lit>`.
190 Debugging Information tests
191 ---------------------------
193 To run debugging information tests simply add the ``cross-project-tests``
195 command-line.
200 The LLVM regression tests are driven by :program:`lit` and are located in the
203 This directory contains a large array of small tests that exercise
205 The directory is broken into several sub-directories, each focused on a
208 Writing new regression tests
209 ----------------------------
213 and is written to a file, ``test/lit.site.cfg.py`` in the build directory.
216 In order for the regression tests to work, each directory of tests must
217 have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
218 how to run the tests. This file is just Python code and thus is very
219 flexible, but we've standardized it for the LLVM regression tests. If
220 you're adding a directory of tests, just copy ``lit.local.cfg`` from
221 another directory to get running. The standard ``lit.local.cfg`` simply
222 specifies which files to look in for tests. Any directory that contains
223 only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
224 documentation <CommandGuide/lit>` for more information.
226 Each test file must contain lines starting with "RUN:" that tell :program:`lit`
227 how to run it. If there are no RUN lines, :program:`lit` will issue an error
232 to execute. Together, these lines form the "script" that :program:`lit`
236 script, they are not. RUN lines are interpreted by :program:`lit`.
240 :program:`lit` performs substitution on each RUN line to replace LLVM tool names
242 ``$(LLVM_OBJ_ROOT)/bin``). This ensures that :program:`lit` does
251 execution. :program:`lit` will substitute variables and arrange for the pipeline
257 .. code-block:: llvm
259 ; RUN: llvm-as < %s | llvm-dis > %t1
260 ; RUN: llvm-dis < %s.bc-13 > %t2
267 your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
269 To avoid this use curly braces to tell :program:`lit` that it should treat
276 lines is deprecated - please do not send or commit patches that use it.]*
278 Put related tests into a single file rather than having a separate file per
282 Generating assertions in regression tests
283 -----------------------------------------
289 For example to generate assertions in an :program:`llc`-based test, after
292 .. code-block:: bash
294 % llvm/utils/update_llc_test_checks.py --llc-binary build/bin/llc test.ll
299 If you want to update assertions in an existing test case, pass the `-u` option
302 Sometimes a test absolutely depends on hand-written assertions and should not
311 .. code-block:: none
314 opt -passes='print<cost-model>'
323 llvm-mca
331 Precommit workflow for tests
332 ----------------------------
335 baseline check-lines first. That is, the test will show a miscompile or
339 A follow-up patch with code changes to the compiler will then show check-line
340 differences to the tests, so it is easier to see the effect of the patch.
343 Baseline tests (no-functional-change or NFC patch) may be pushed to main
344 without pre-commit review if you have commit access.
346 Best practices for regression tests
347 -----------------------------------
349 - Use auto-generated check lines (produced by the scripts mentioned above)
351 - Include comments about what is tested/expected in a particular test. If there
354 - Avoid undefined behavior and poison/undef values unless necessary. For
357 - Minimize tests by removing unnecessary instructions, metadata, attributes,
358 etc. Tools like ``llvm-reduce`` can help automate this.
359 - Outside PhaseOrdering tests, only run a minimal set of passes. For example,
360 prefer ``opt -S -passes=instcombine`` over ``opt -S -O3``.
361 - Avoid unnamed instructions/blocks (such as ``%0`` or ``1:``), because they may
363 running the test through ``opt -S -passes=instnamer``.
364 - Try to give values (including variables, blocks and functions) meaningful
369 -----------
373 using ``split-file`` to extract them. For example,
375 .. code-block:: llvm
377 ; RUN: split-file %s %t
378 ; RUN: llvm-link -S %t/a.ll %t/b.ll | FileCheck %s
382 ;--- a.ll
384 ;--- b.ll
387 The parts are separated by the regex ``^(.|//)--- <part>``.
390 ``--leading-lines`` to add leading empty lines to preserve line numbers.
407 .. code-block:: llvm
411 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
415 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
416 ; CHECK-DAG: "Compiler V1"
417 ; CHECK-DAG: "Compiler V2"
418 ; CHECK-DAG: "Compiler V3"
436 Some existing tests use ``RUN: true`` in extra files instead of just
440 Elaborated tests
441 ----------------
444 unnecessary details. However, for tests requiring elaborate IR or assembly
447 ``split-file`` part called ``gen``. Then, run
451 .. code-block:: none
453 ; RUN: rm -rf %t && split-file %s %t && cd %t
454 ; RUN: opt -S a.ll ... | FileCheck %s
458 ;--- a.cc
460 ;--- gen
461 clang --target=x86_64-linux -S -emit-llvm -g a.cc -o -
463 ;--- a.ll
466 .. code-block:: bash
470 The script will prepare extra files with ``split-file``, invoke ``gen``, and
475 skip ``split-file`` in RUN lines.
477 .. code-block:: none
479 # RUN: llvm-mc -filetype=obj -triple=x86_64 %s -o a.o
485 #--- a.cc
487 #--- gen
488 clang --target=x86_64-linux -S -g a.cc -o -
498 don't need ``-fdebug-compilation-dir=`` since its default value is ``PWD``.
503 If the test body contains multiple files, you can print ``---`` separators and
504 utilize ``split-file`` in ``RUN`` lines.
506 .. code-block:: none
508 # RUN: rm -rf %t && split-file %s %t && cd %t
511 #--- a.cc
513 #--- b.cc
515 #--- gen
516 clang --target=x86_64-linux -S -O1 -g a.cc -o -
517 echo '#--- b.s'
518 clang --target=x86_64-linux -S -O1 -g b.cc -o -
519 #--- a.s
521 Fragile tests
522 -------------
528 .. code-block:: console
535 $ opt -S /path/to/example.ll
544 .. code-block:: llvm
546 ; RUN: opt -S %s | FileCheck
549 ; CHECK-NOT: load
555 To make your tests robust, always use ``opt ... < %s`` in the RUN line.
558 Platform-Specific Tests
559 -----------------------
561 Whenever adding tests that require the knowledge of a specific platform,
562 either related to code generated, specific output or back-end features,
564 run on different architectures (and don't even compile all back-ends),
567 The first problem is to check for target-specific output, for example sizes
570 * Tests containing Windows paths will fail on Linux and vice-versa.
571 * Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
572 * Tests where the debug information calculates the size of types and structures.
574 Also, if the test rely on any behaviour that is coded in any back-end, it must
575 go in its own directory. So, for instance, code generator tests for ARM go
577 ``lit`` configuration file that ensure all tests in that directory will
578 only run if a specific back-end is compiled and available.
580 For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
582 .. code-block:: python
588 Other platform-specific tests are those that depend on a specific feature
589 of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
591 For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
594 .. code-block:: llvm
596 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
597 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
598 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
602 .. code-block:: llvm
611 So, if you're testing for a behaviour that you know is platform-specific or
612 depends on special features of sub-architectures, you must add the specific
618 ---------------------------
620 Some tests can be run only in specific configurations, such as
624 Some tests are expected to fail. For example, there may be a known bug
629 .. code-block:: llvm
634 ; UNSUPPORTED: system-linux
638 ``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
641 - Features added to ``config.available_features`` by configuration files such as ``lit.cfg``.
642 String comparison of features is case-sensitive. Furthermore, a boolean expression can
647 - The default target triple, preceded by the string ``target=`` (for example,
648 ``target=x86_64-pc-windows-msvc``). Typically regular expressions are used
649 to match parts of the triple (for example, ``target={{.*}}-windows{{.*}}``
659 .. code-block:: llvm
663 ; UNSUPPORTED: system-windows, target={{.*linux.*}} && !target={{.*android.*}}
665 ; XFAIL: target=powerpc{{.*}}, system-darwin
669 ----------------------------
692 Checking the target triple can be tricky; it's easy to mis-specify. For
694 mips64, and mips64el. ``target={{.*}}-linux-gnu`` will match
695 x86_64-unknown-linux-gnu, but not armv8l-unknown-linux-gnueabihf.
696 Prefer to use hyphens to delimit triple components (``target=mips-{{.*}}``)
705 ``target={{.+-freebsd.*}} || target={{.+-netbsd.*}}``
709 -------------
738 misused and cause race conditions between tests.
740 Use ``rm -rf %t && mkdir %t`` instead if a temporary directory is necessary.
748 ``%{fs-src-root}``
752 ``%{fs-tmp-root}``
757 ``%{fs-sep}``
808 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
814 For example, this can be used in tests with multiple RUN lines,
817 **LLVM-specific substitutions:**
831 **Clang-specific substitutions:**
840 Invokes the CL-compatible Clang driver.
843 Invokes the G++-compatible Clang driver.
851 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
852 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
855 **FileCheck-specific substitutions:**
862 .. _Test-specific substitutions:
864 **Test-specific substitutions:**
868 - Lit configuration files (e.g., ``lit.cfg`` or ``lit.local.cfg``) can define
869 substitutions for all tests in a test directory. They do so by extending the
871 consisting of a pattern and its replacement, which lit applies as plain text
874 - To define substitutions within a single test file, lit supports the
877 substitution list that is produced by lit configuration files.
883 .. code-block:: llvm
889 ; DEFINE: %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags} \
890 ; DEFINE: -emit-llvm -o - %s | \
893 Alternatively, the above substitutions can be defined in a lit configuration
898 .. code-block:: llvm
900 ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0 -fopenmp-simd
901 ; REDEFINE: %{fcflags} = -check-prefix=SIMD
904 ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu -fopenmp-simd
905 ; REDEFINE: %{fcflags} = -check-prefix=SIMD
908 ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0
909 ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
912 ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu
913 ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
925 - Upon arriving at each ``RUN:`` line, lit expands all substitutions in that
929 ``%(line-<number>)``.
930 - When expanding substitutions in a ``RUN:`` line, lit makes only one pass
935 setting `recursiveExpansionLimit`_ in a lit configuration file.)
936 - While lit configuration files can insert anywhere in the substitution list,
940 - Defining a substitution in terms of itself, whether directly or via other
949 - ``DEFINE: %{name} = value``
954 confusing expansions (e.g., a lit configuration file might define a
958 the same test file or in a lit configuration file, and both will expand.
960 - ``REDEFINE: %{name} = value``
972 - **Substitution name**: In the directive, whitespace immediately before or
978 - It is impossible for ``%{name}`` to contain sequences that are special in
980 ``%{name}`` as a substitution pattern in a lit configuration file could
982 - The braces help avoid the possibility that another substitution's pattern
983 will match part of ``%{name}`` or vice-versa, producing confusing
984 expansions. However, the patterns of substitutions defined by lit
985 configuration files and by lit itself are not restricted to this form, so
988 - **Substitution value**: The value includes all text from the first
989 non-whitespace character after ``=`` to the last non-whitespace character. If
990 there is no non-whitespace character after ``=``, the value is the empty
993 - **Line continuations**: If the last non-whitespace character on the line after
997 above to parse the text after ``:`` in either directive, lit joins that text
1008 line, lit makes only one pass through the substitution list by default. Thus,
1014 .. code-block:: llvm
1028 .. code-block:: shell
1037 To address such use cases, lit configuration files support
1038 ``config.recursiveExpansionLimit``, which can be set to a non-negative integer
1040 the above example, setting the limit to 2 would cause lit to make a second pass
1044 .. code-block:: shell
1048 To improve performance, lit will stop making passes when it notices the ``RUN:``
1052 To facilitate debugging, after reaching the limit, lit will make one extra pass
1054 setting the limit to 1 will thus cause lit to report an error instead of
1058 -------
1060 The llvm lit configuration allows to customize some things with user options:
1067 % llvm-lit "-Dllc=llc -verify-machineinstrs"
1070 Enable the execution of long running tests.
1073 Load the specified lit configuration instead of the default one.
1077 --------------
1080 helpers are in the PATH when running tests, so you can just call them using
1085 Zero result codes become 1. Non-zero result codes become 0.
1087 To make the output more useful, :program:`lit` will scan
1089 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number