Lines Matching +full:lit +full:- +full:tests
1 lit - LLVM Integrated Tester
4 .. program:: lit
7 --------
9 :program:`lit` [*options*] [*tests*]
12 -----------
14 :program:`lit` is a portable tool for executing LLVM and Clang style test
16 :program:`lit` is designed to be a lightweight testing tool with as simple a
19 :program:`lit` should be run with one or more *tests* to run specified on the
20 command line. Tests can be either individual test files or directories to
21 search for tests (see :ref:`test-discovery`).
24 tests have been run :program:`lit` will print summary information on the number
25 of tests which passed or failed (see :ref:`test-status-results`). The
26 :program:`lit` program will execute with a non-zero exit code if any tests
29 By default :program:`lit` will use a succinct progress display and will only
30 print summary information for test failures. See :ref:`output-options` for
31 options controlling the :program:`lit` progress display and output.
33 :program:`lit` also includes a number of options for controlling how tests are
35 :ref:`execution-options` for more information.
37 Finally, :program:`lit` also supports additional options for only running a
39 :ref:`selection-options` for more information.
41 :program:`lit` parses options from the environment variable ``LIT_OPTS`` after
43 supplementing or overriding the command-line options supplied to :program:`lit`
46 :program:`lit` can also read options from response files which are specified as
52 Users interested in the :program:`lit` architecture or designing a
53 :program:`lit` testing implementation should see :ref:`lit-infrastructure`.
56 ---------------
58 .. option:: -h, --help
60 Show the :program:`lit` help message and exit.
62 .. option:: --version
64 Show :program:`lit`'s version number and exit.
66 .. option:: -j N, --workers=N
68 Run ``N`` tests in parallel. By default, this is automatically chosen to
71 .. option:: --config-prefix=NAME
74 test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`.
76 .. option:: -D NAME[=VALUE], --param NAME[=VALUE]
82 .. _output-options:
85 --------------
87 .. option:: -q, --quiet
91 .. option:: -s, --succinct
93 Show less output, for example don't show information on tests that pass.
94 Also show a progress bar, unless ``--no-progress-bar`` is specified.
96 .. option:: -v, --verbose
103 Moreover, :program:`lit` inserts ``'RUN: at line N'`` before each
107 .. option:: -vv, --echo-all-commands
109 Deprecated alias for -v.
111 .. option:: -a, --show-all
113 Enable -v, but for all tests not just failed tests.
115 .. option:: -o PATH, --output PATH
119 .. option:: --no-progress-bar
123 .. option:: --show-excluded
125 Show excluded tests.
127 .. option:: --show-skipped
129 Show skipped tests.
131 .. option:: --show-unsupported
133 Show unsupported tests.
135 .. option:: --show-pass
137 Show passed tests.
139 .. option:: --show-flakypass
141 Show passed with retry tests.
143 .. option:: --show-xfail
145 Show expectedly failed tests.
147 .. _execution-options:
150 -----------------
152 .. option:: --gtest-sharding
156 .. option:: --no-gtest-sharding
160 .. option:: --path=PATH
162 Specify an additional ``PATH`` to use when searching for executables in tests.
164 .. option:: --vg
166 Run individual tests under valgrind (using the memcheck tool). The
167 ``--error-exitcode`` argument for valgrind is used so that valgrind failures
168 will cause the program to exit with a non-zero status.
170 When this option is enabled, :program:`lit` will also automatically provide a
172 failure in) certain tests.
174 .. option:: --vg-leak
176 When :option:`--vg` is used, enable memory leak checks. When this option is
177 enabled, :program:`lit` will also automatically provide a "``vg_leak``"
179 certain tests.
181 .. option:: --vg-arg=ARG
183 When :option:`--vg` is used, specify an additional argument to pass to
186 .. option:: --no-execute
188 Don't execute any tests (assume that they pass).
190 .. option:: --xunit-xml-output XUNIT_XML_OUTPUT
192 Write XUnit-compatible XML test reports to the specified file.
194 .. option:: --report-failures-only
196 Only include unresolved, timed out, failed and unexpectedly passed tests in the report.
198 .. option:: --resultdb-output RESULTDB_OUTPUT
202 .. option:: --time-trace-output TIME_TRACE_OUTPUT
206 .. option:: --timeout MAXINDIVIDUALTESTTIME
211 .. option:: --timeout=N
215 alias for :option:`--max-time`; the two are different kinds of maximums.
217 .. option:: --max-failures MAX_FAILURES
221 .. option:: --allow-empty-runs
223 Do not fail the run if all tests are filtered out.
225 .. option:: --per-test-coverage
231 .. option:: --ignore-fail
233 Exit with status zero even if some tests fail.
235 .. option:: --skip-test-time-recording
239 .. option:: --time-tests
241 Track the wall time individual tests take to execute and includes the results
242 in the summary output. This is useful for determining which tests in a test
245 .. _selection-options:
248 -----------------
250 By default, `lit` will run failing tests first, then run tests in descending
252 changed using the :option:`--order` option.
255 `.lit_test_times.txt`. If this file does not exist, then `lit` checks the
258 .. option:: --max-tests=N
260 Run at most ``N`` tests and then terminate.
262 .. option:: --max-time=N
264 Spend at most ``N`` seconds (approximately) running tests and then terminate.
265 Note that this is not an alias for :option:`--timeout`; the two are
268 .. option:: --order={lexical,random,smart}
270 Define the order in which tests are run. The supported values are:
272 - lexical - tests will be run in lexical order according to the test file
275 - random - tests will be run in random order.
277 - smart - tests that failed previously will be run first, then the remaining
278 tests, all in descending execution time order. This is the default as it
281 .. option:: --shuffle
283 Run the tests in a random order, not failing/slowest first. Deprecated,
284 use :option:`--order` instead.
286 .. option:: -i, --incremental
288 Run failed tests first (DEPRECATED: use ``--order=smart``).
290 .. option:: --filter=REGEXP
292 Run only those tests whose name matches the regular expression specified in
295 to ``lit`` is issued indirectly.
297 .. option:: --filter-out=REGEXP
299 Filter out those tests whose name matches the regular expression specified in
302 call to ``lit`` is issued indirectly.
304 .. option:: --xfail=LIST
306 Treat those tests whose name is in the semicolon separated list ``LIST`` as
310 ``lit`` is issued indirectly.
315 .. code-block:: none
317 LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
319 In this case, all of the following tests are treated as ``XFAIL``:
321 .. code-block:: none
323 libomp :: affinity/kmp-hw-subset.c
324 libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
325 libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
328 reported in LIT output. For example, we can adjust the previous
329 example not to treat the ``nvptx64-nvidia-cuda`` version of
332 .. code-block:: none
334 LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
336 .. option:: --xfail-not=LIST
338 Do not treat the specified tests as ``XFAIL``. The environment variable
340 same as for :option:`--xfail` and ``LIT_XFAIL``. :option:`--xfail-not` and
342 including an :option:`--xfail` appearing later on the command line. The
346 .. option:: --num-shards=M
348 Divide the set of selected tests into ``M`` equal-sized subsets or
350 ``--run-shard=N`` option, which selects the shard to run. The environment
356 .. option:: --run-shard=N
358 Select which shard to run, assuming the ``--num-shards=M`` option was
364 ------------------
366 .. option:: --debug
368 Run :program:`lit` in debug mode, for debugging configuration issues and
369 :program:`lit` itself.
371 .. option:: --show-suites
375 .. option:: --show-tests
377 List all of the discovered tests and exit.
379 .. option:: --show-used-features
385 -----------
387 :program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS
389 for non-test related failures (for example a user error or an internal program
392 .. _test-discovery:
395 --------------
397 The inputs passed to :program:`lit` can be either individual tests, or entire
398 directories or hierarchies of tests to run. When :program:`lit` starts up, the
399 first thing it does is convert the inputs into a complete list of tests to run
402 In the :program:`lit` model, every test must exist inside some *test suite*.
403 :program:`lit` resolves the inputs specified on the command line to test suites
404 by searching upwards from the input path until it finds a :file:`lit.cfg` or
405 :file:`lit.site.cfg` file. These files serve as both a marker of test suites
406 and as configuration files which :program:`lit` loads in order to understand
407 how to find and run the tests inside the test suite.
409 Once :program:`lit` has mapped the inputs into test suites it traverses the
410 list of inputs adding tests for individual files and recursively searching for
411 tests in directories.
413 This behavior makes it easy to specify a subset of tests to run, while still
414 allowing the test suite configuration to control exactly how tests are
415 interpreted. In addition, :program:`lit` always identifies tests by the test
417 appropriately configured projects, this allows :program:`lit` to provide
418 convenient and flexible support for out-of-tree builds.
420 .. _test-status-results:
423 -------------------
433 The test succeeded after being re-run more than once. This only applies to
434 tests containing an ``ALLOW_RETRIES:`` annotation.
444 The test succeeded, but it was expected to fail. This is used for tests which
460 which can report unsupported tests.
467 Depending on the test format tests may produce additional information about
468 their status (generally only for failures). See the :ref:`output-options`
471 .. _lit-infrastructure:
473 LIT INFRASTRUCTURE
474 ------------------
476 This section describes the :program:`lit` testing architecture for users interested in
477 creating a new :program:`lit` testing implementation, or extending an existing one.
479 :program:`lit` proper is primarily an infrastructure for discovering and running
480 arbitrary tests, and to expose a single convenient interface to these
481 tests. :program:`lit` itself doesn't know how to run tests, rather this logic is
487 As described in :ref:`test-discovery`, tests are always located inside a *test
488 suite*. Test suites serve to define the format of the tests they contain, the
489 logic for finding those tests, and any additional information to run the tests.
491 :program:`lit` identifies test suites as directories containing ``lit.cfg`` or
492 ``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are
495 :option:`--show-suites` to display the discovered test suites at startup.
503 The global **lit** configuration object (a *LitConfig* instance), which defines
518 discover and run tests in the test suite. Generally this will be a builtin test
519 format available from the *lit.formats* module.
521 **test_source_root** The filesystem path to the test suite root. For out-of-dir
522 builds this is the directory that will be scanned for tests.
524 **test_exec_root** For out-of-dir builds, the path to the test suite root inside
525 the object directory. This is where tests will be run and temporary output files
529 tests in the suite.
531 **standalone_tests** When true, mark a directory with tests expected to be run
532 standalone. Test discovery is disabled for that directory. *lit.suffixes* and
533 *lit.excludes* must be empty when this variable is true.
535 **suffixes** For **lit** test formats which scan directories for tests, this
538 **substitutions** For **lit** test formats which substitute variables into a test
541 **unsupported** Mark an unsupported directory, all tests within it will be
547 **root** The root configuration. This is the top-most :program:`lit` configuration in
560 Once test suites are located, :program:`lit` recursively traverses the source
561 directory (following *test_source_root*) looking for tests. When :program:`lit`
562 enters a sub-directory, it first checks to see if a nested test suite is
565 :ref:`local-configuration-files`).
567 Tests are identified by the test suite they are contained within, and the
570 "virtual tests" which have a path that contains both the path to the actual
573 .. _local-configuration-files:
578 When :program:`lit` loads a subdirectory in a test suite, it instantiates a
580 --- the root of this configuration chain will always be a test suite. Once the
581 test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file
584 be used to define subdirectories of optional tests, or to change other
585 configuration parameters --- for example, to change the test format, or the
591 :program:`lit` allows patterns to be substituted inside RUN commands. It also
602 %{fs-src-root} root component of file system paths pointing to the LLVM checkout
603 %{fs-tmp-root} root component of file system paths pointing to the test's temporary directory
604 %{fs-sep} file system path separator
645 modules :ref:`local-configuration-files`.
653 The :program:`lit` output for a test run conforms to the following schema, in
661 .. code-block:: none
665 where ``<result-code>`` is a standard test result such as PASS, FAIL, XFAIL,
678 .. code-block:: none
689 The following is an example of a test run output which consists of four tests A,
692 .. code-block:: none
705 For convenience :program:`lit` automatically adds **available_features** for
708 :program:`lit` adds a feature based on the operating system being built on, for
709 example: `system-darwin`, `system-linux`, etc. :program:`lit` also
711 `target-x86_64`, `target-aarch64`, etc.
713 When building with sanitizers enabled, :program:`lit` automatically adds the
717 *llvm/utils/lit/lit/llvm/config.py*.
719 LIT EXAMPLE TESTS
722 The :program:`lit` distribution contains several example implementations of
726 --------