Lines Matching +full:llvm +full:- +full:test
2 LLVM Testing Infrastructure Guide
17 This document is the reference manual for the LLVM testing
18 infrastructure. It documents the structure of the LLVM testing
25 In order to use the LLVM testing infrastructure, you will need all of the
26 software required to build LLVM, as well as `Python <http://python.org>`_ 3.8 or
29 LLVM Testing Infrastructure Organization
32 The LLVM testing infrastructure contains three major categories of tests:
34 tests are contained inside the LLVM repository itself under ``llvm/unittests``
35 and ``llvm/test`` respectively and are expected to always pass -- they should be
38 The whole programs tests are referred to as the "LLVM test suite" (or
39 "test-suite") and are in the ``test-suite``
40 `repository on GitHub <https://github.com/llvm/llvm-test-suite.git>`_.
42 tests" in places, which is less ambiguous than "test-suite" and remains
46 ----------
48 Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/docs/primer.md>`_
50 and are located in the ``llvm/unittests`` directory.
56 ----------------
58 The regression tests are small pieces of code that test a specific
59 feature of LLVM or trigger a specific bug in LLVM. The language they are
60 written in depends on the part of LLVM being tested. These tests are driven by
61 the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
62 are located in the ``llvm/test`` directory.
64 Typically when a bug is found in LLVM, a regression test containing just
67 piece of LLVM IR distilled from an actual application or benchmark.
70 ----------------
77 See `llvm/test/Analysis/BranchProbabilityInfo/loop.ll <https://github.com/llvm/llvm-project/blob/main/llvm/test/Analysis/BranchProbabilityInfo/loop.ll>`_
78 for an example of such test.
80 ``test-suite``
81 --------------
83 The test suite contains whole programs, which are pieces of code which
84 can be compiled and linked into a stand-alone program that can be
94 serve as a way of benchmarking LLVM performance, both in terms of the
96 LLVM compiles, optimizes, and generates code.
98 The test-suite is located in the ``test-suite``
99 `repository on GitHub <https://github.com/llvm/llvm-test-suite.git>`_.
104 ---------------------------
106 The test suite contains tests to check quality of debugging information.
107 The test are written in C based languages or in LLVM assembly language.
111 test suite for more information. This test suite is located in the
112 ``cross-project-tests/debuginfo-tests`` directory.
118 regression tests are in the main "llvm"/ directory under the directories
119 ``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the
120 main LLVM tree). Use ``make check-all`` to run the unit and regression tests
121 after building LLVM.
123 The ``test-suite`` module contains more comprehensive tests including whole C
127 -------------------------
129 To run all of the LLVM unit tests use the check-llvm-unit target:
131 .. code-block:: bash
133 % make check-llvm-unit
135 To run all of the LLVM regression tests use the check-llvm target:
137 .. code-block:: bash
139 % make check-llvm
141 In order to get reasonable testing performance, build LLVM and subprojects
144 .. code-block:: bash
146 % cmake -DCMAKE_BUILD_TYPE="Release" -DLLVM_ENABLE_ASSERTIONS=On
148 If you have `Clang <https://clang.llvm.org/>`_ checked out and built, you
149 can run the LLVM and Clang tests simultaneously using:
151 .. code-block:: bash
153 % make check-all
158 .. code-block:: bash
160 % make check LIT_ARGS="-v --vg --vg-leak"
164 To run individual tests or subsets of tests, you can use the ``llvm-lit``
165 script which is built as part of LLVM. For example, to run the
166 ``Integer/BitPacked.ll`` test by itself you can run:
168 .. code-block:: bash
170 % llvm-lit <path to llvm-project>/llvm/test/Integer/BitPacked.ll
173 The test files are in the ``llvm-project`` directory, not the directory you
174 are building LLVM in.
178 .. code-block:: bash
180 % llvm-lit <path to llvm-project>/llvm/test/CodeGen/ARM
183 **non-user** location. Under Linux, install with sudo or within a virtual
187 For more information on using the :program:`lit` tool, see ``llvm-lit --help``
191 ---------------------------
193 To run debugging information tests simply add the ``cross-project-tests``
195 command-line.
197 Regression test structure
200 The LLVM regression tests are driven by :program:`lit` and are located in the
201 ``llvm/test`` directory.
204 various features of LLVM and to ensure that regressions do not occur.
205 The directory is broken into several sub-directories, each focused on a
206 particular area of LLVM.
209 ----------------------------
211 The regression test structure is very simple, but does require some
213 and is written to a file, ``test/lit.site.cfg.py`` in the build directory.
214 The ``llvm/test`` Makefile does this work for you.
219 flexible, but we've standardized it for the LLVM regression tests. If
226 Each test file must contain lines starting with "RUN:" that tell :program:`lit`
228 while running a test.
230 RUN lines are specified in the comments of the test program using the
233 executes to run the test case. The syntax of the RUN lines is similar to a
240 :program:`lit` performs substitution on each RUN line to replace LLVM tool names
243 not invoke any stray LLVM tools in the user's path during testing.
253 test case) fails too.
257 .. code-block:: llvm
259 ; RUN: llvm-as < %s | llvm-dis > %t1
260 ; RUN: llvm-dis < %s.bc-13 > %t2
274 The recommended way to examine output to figure out if the test passes is using
276 lines is deprecated - please do not send or commit patches that use it.]*
279 test. Check if there are files already covering your feature and consider
283 -----------------------------------------
285 Some regression test cases are very large and complex to write/update by hand.
287 llvm/utils/ to generate the assertions.
289 For example to generate assertions in an :program:`llc`-based test, after
292 .. code-block:: bash
294 % llvm/utils/update_llc_test_checks.py --llc-binary build/bin/llc test.ll
299 If you want to update assertions in an existing test case, pass the `-u` option
302 Sometimes a test absolutely depends on hand-written assertions and should not
304 not autogenerate`` to the first line, and the scripts will skip that test. It
305 is a good idea to explain why generated assertions will not work for the test
311 .. code-block:: none
314 opt -passes='print<cost-model>'
323 llvm-mca
332 ----------------------------
334 If the test does not crash, assert, or infinite loop, commit the test with
335 baseline check-lines first. That is, the test will show a miscompile or
337 something is expected to change in a test.
339 A follow-up patch with code changes to the compiler will then show check-line
343 Baseline tests (no-functional-change or NFC patch) may be pushed to main
344 without pre-commit review if you have commit access.
347 -----------------------------------
349 - Use auto-generated check lines (produced by the scripts mentioned above)
351 - Include comments about what is tested/expected in a particular test. If there
354 - Avoid undefined behavior and poison/undef values unless necessary. For
357 - Minimize tests by removing unnecessary instructions, metadata, attributes,
358 etc. Tools like ``llvm-reduce`` can help automate this.
359 - Outside PhaseOrdering tests, only run a minimal set of passes. For example,
360 prefer ``opt -S -passes=instcombine`` over ``opt -S -O3``.
361 - Avoid unnamed instructions/blocks (such as ``%0`` or ``1:``), because they may
362 require renumbering on future test modifications. These can be removed by
363 running the test through ``opt -S -passes=instnamer``.
364 - Try to give values (including variables, blocks and functions) meaningful
369 -----------
371 If your test requires extra files besides the file containing the ``RUN:`` lines
373 using ``split-file`` to extract them. For example,
375 .. code-block:: llvm
377 ; RUN: split-file %s %t
378 ; RUN: llvm-link -S %t/a.ll %t/b.ll | FileCheck %s
382 ;--- a.ll
384 ;--- b.ll
387 The parts are separated by the regex ``^(.|//)--- <part>``.
389 If you want to test relative line numbers like ``[[#@LINE+1]]``, specify
390 ``--leading-lines`` to add leading empty lines to preserve line numbers.
395 For example, consider ``test/Linker/ident.ll``. The directory structure is
398 test/
407 .. code-block:: llvm
411 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
413 ; Verify that multiple input llvm.ident metadata are linked together.
415 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
416 ; CHECK-DAG: "Compiler V1"
417 ; CHECK-DAG: "Compiler V2"
418 ; CHECK-DAG: "Compiler V3"
422 !llvm.ident = !{!0, !1}
428 !llvm.ident = !{!0}
432 actually participate in the test besides holding the ``RUN:`` lines.
441 ----------------
443 Generally, IR and assembly test files benefit from being cleaned to remove
447 ``split-file`` part called ``gen``. Then, run
448 ``llvm/utils/update_test_body.py`` on the test file to generate the needed
451 .. code-block:: none
453 ; RUN: rm -rf %t && split-file %s %t && cd %t
454 ; RUN: opt -S a.ll ... | FileCheck %s
458 ;--- a.cc
460 ;--- gen
461 clang --target=x86_64-linux -S -emit-llvm -g a.cc -o -
463 ;--- a.ll
466 .. code-block:: bash
468 PATH=/path/to/clang_build/bin:$PATH llvm/utils/update_test_body.py path/to/test.ll
470 The script will prepare extra files with ``split-file``, invoke ``gen``, and
473 For convenience, if the test needs one single assembly file, you can also wrap
475 skip ``split-file`` in RUN lines.
477 .. code-block:: none
479 # RUN: llvm-mc -filetype=obj -triple=x86_64 %s -o a.o
485 #--- a.cc
487 #--- gen
488 clang --target=x86_64-linux -S -g a.cc -o -
498 don't need ``-fdebug-compilation-dir=`` since its default value is ``PWD``.
503 If the test body contains multiple files, you can print ``---`` separators and
504 utilize ``split-file`` in ``RUN`` lines.
506 .. code-block:: none
508 # RUN: rm -rf %t && split-file %s %t && cd %t
511 #--- a.cc
513 #--- b.cc
515 #--- gen
516 clang --target=x86_64-linux -S -O1 -g a.cc -o -
517 echo '#--- b.s'
518 clang --target=x86_64-linux -S -O1 -g b.cc -o -
519 #--- a.s
522 -------------
524 It is easy to write a fragile test that would fail spuriously if the tool being
528 .. code-block:: console
535 $ opt -S /path/to/example.ll
544 .. code-block:: llvm
546 ; RUN: opt -S %s | FileCheck
549 ; CHECK-NOT: load
553 This test will fail if placed into a ``download`` directory.
558 Platform-Specific Tests
559 -----------------------
562 either related to code generated, specific output or back-end features,
564 run on different architectures (and don't even compile all back-ends),
567 The first problem is to check for target-specific output, for example sizes
570 * Tests containing Windows paths will fail on Linux and vice-versa.
574 Also, if the test rely on any behaviour that is coded in any back-end, it must
576 into ``test/CodeGen/ARM`` and so on. Those directories contain a special
578 only run if a specific back-end is compiled and available.
580 For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
582 .. code-block:: python
584 config.suffixes = ['.ll', '.c', '.cpp', '.test']
588 Other platform-specific tests are those that depend on a specific feature
589 of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
591 For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
594 .. code-block:: llvm
596 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
597 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
598 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
602 .. code-block:: llvm
611 So, if you're testing for a behaviour that you know is platform-specific or
612 depends on special features of sub-architectures, you must add the specific
613 triple, test with the specific FileCheck and put it into the specific
617 Constraining test execution
618 ---------------------------
622 and ``UNSUPPORTED`` to control when the test is enabled.
625 that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
626 An ``XFAIL`` test will be successful if its execution fails, and
629 .. code-block:: llvm
631 ; This test will be only enabled in the build with asserts.
633 ; This test is disabled when running on Linux.
634 ; UNSUPPORTED: system-linux
635 ; This test is expected to fail when targeting PowerPC.
638 ``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
641 - Features added to ``config.available_features`` by configuration files such as ``lit.cfg``.
642 String comparison of features is case-sensitive. Furthermore, a boolean expression can
647 - The default target triple, preceded by the string ``target=`` (for example,
648 ``target=x86_64-pc-windows-msvc``). Typically regular expressions are used
649 to match parts of the triple (for example, ``target={{.*}}-windows{{.*}}``
652 | ``REQUIRES`` enables the test if all expressions are true.
653 | ``UNSUPPORTED`` disables the test if any expression is true.
654 | ``XFAIL`` expects the test to fail if any expression is true.
656 Use, ``XFAIL: *`` if the test is expected to fail everywhere. Similarly, use
657 ``UNSUPPORTED: target={{.*}}`` to disable the test everywhere.
659 .. code-block:: llvm
661 ; This test is disabled when running on Windows,
663 ; UNSUPPORTED: system-windows, target={{.*linux.*}} && !target={{.*android.*}}
664 ; This test is expected to fail when targeting PowerPC or running on Darwin.
665 ; XFAIL: target=powerpc{{.*}}, system-darwin
669 ----------------------------
676 understand. Generally, people use ``REQUIRES`` to state things that the test
678 the test is expected never to work.
682 Both of these indicate that the test isn't expected to work; however, they
683 have different effects. ``UNSUPPORTED`` causes the test to be skipped;
684 this saves execution time, but then you'll never know whether the test
685 actually would start working. Conversely, ``XFAIL`` actually runs the test
687 if/when the test begins to behave correctly (an XPASS test result). You
692 Checking the target triple can be tricky; it's easy to mis-specify. For
694 mips64, and mips64el. ``target={{.*}}-linux-gnu`` will match
695 x86_64-unknown-linux-gnu, but not armv8l-unknown-linux-gnueabihf.
696 Prefer to use hyphens to delimit triple components (``target=mips-{{.*}}``)
704 prevent a ``grep freebsd`` from finding this test. Better to use:
705 ``target={{.+-freebsd.*}} || target={{.+-netbsd.*}}``
709 -------------
711 Besides replacing LLVM tool names the following substitutions are performed in
718 File path to the test case's source. This is suitable for passing on the
719 command line as the input to an LLVM tool.
721 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``
724 Directory path to the test case's source.
726 Example: ``/home/user/llvm/test/MC/ELF``
729 File path to a temporary file name that could be used for this test case.
730 The file name won't conflict with other test cases. You can append to it
734 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
740 Use ``rm -rf %t && mkdir %t`` instead if a temporary directory is necessary.
742 Example: ``/home/user/llvm.build/test/MC/ELF/Output``
748 ``%{fs-src-root}``
752 ``%{fs-tmp-root}``
753 Expands to the root component of file system paths for the test's temporary
757 ``%{fs-sep}``
808 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
815 which reference the test file's line numbers.
817 **LLVM-specific substitutions:**
831 **Clang-specific substitutions:**
840 Invokes the CL-compatible Clang driver.
843 Invokes the G++-compatible Clang driver.
850 the desired ABI. For example, if the test suite is running with the
851 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
852 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
855 **FileCheck-specific substitutions:**
859 output affects test results. It's usually easy to tell: just look for
862 .. _Test-specific substitutions:
864 **Test-specific substitutions:**
868 - Lit configuration files (e.g., ``lit.cfg`` or ``lit.local.cfg``) can define
869 substitutions for all tests in a test directory. They do so by extending the
874 - To define substitutions within a single test file, lit supports the
876 they have no effect on other test files, these directives modify a copy of the
879 For example, the following directives can be inserted into a test file to define
883 .. code-block:: llvm
889 ; DEFINE: %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags} \
890 ; DEFINE: -emit-llvm -o - %s | \
894 file to be shared with other test files. Either way, the test file can then
898 .. code-block:: llvm
900 ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0 -fopenmp-simd
901 ; REDEFINE: %{fcflags} = -check-prefix=SIMD
904 ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu -fopenmp-simd
905 ; REDEFINE: %{fcflags} = -check-prefix=SIMD
908 ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0
909 ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
912 ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu
913 ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
920 order in a test file: define a substitution before any substitution that might
925 - Upon arriving at each ``RUN:`` line, lit expands all substitutions in that
929 ``%(line-<number>)``.
930 - When expanding substitutions in a ``RUN:`` line, lit makes only one pass
936 - While lit configuration files can insert anywhere in the substitution list,
940 - Defining a substitution in terms of itself, whether directly or via other
949 - ``DEFINE: %{name} = value``
958 the same test file or in a lit configuration file, and both will expand.
960 - ``REDEFINE: %{name} = value``
972 - **Substitution name**: In the directive, whitespace immediately before or
978 - It is impossible for ``%{name}`` to contain sequences that are special in
982 - The braces help avoid the possibility that another substitution's pattern
983 will match part of ``%{name}`` or vice-versa, producing confusing
988 - **Substitution value**: The value includes all text from the first
989 non-whitespace character after ``=`` to the last non-whitespace character. If
990 there is no non-whitespace character after ``=``, the value is the empty
993 - **Line continuations**: If the last non-whitespace character on the line after
1014 .. code-block:: llvm
1028 .. code-block:: shell
1033 ``%{outer}`` and ``%{inner}``. However, if a test has a complex set of
1038 ``config.recursiveExpansionLimit``, which can be set to a non-negative integer
1044 .. code-block:: shell
1058 -------
1060 The llvm lit configuration allows to customize some things with user options:
1063 Substitute the respective llvm tool name with a custom command line. This
1067 % llvm-lit "-Dllc=llc -verify-machineinstrs"
1077 --------------
1085 Zero result codes become 1. Non-zero result codes become 0.
1088 the lines of the test case for ones that contain a pattern that matches
1089 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
1090 that is related to the test case. The number after "PR" specifies the
1091 LLVM Bugzilla number. When a PR number is specified, it will be used in
1093 a test fails.
1099 (a) it prevents special interpretation of lines that are part of the test
1100 program, not the instructions to the test case, and
1102 (b) it speeds things up for really big test cases by avoiding