1lit - LLVM Integrated Tester 2============================ 3 4.. program:: lit 5 6SYNOPSIS 7-------- 8 9:program:`lit` [*options*] [*tests*] 10 11DESCRIPTION 12----------- 13 14:program:`lit` is a portable tool for executing LLVM and Clang style test 15suites, summarizing their results, and providing indication of failures. 16:program:`lit` is designed to be a lightweight testing tool with as simple a 17user interface as possible. 18 19:program:`lit` should be run with one or more *tests* to run specified on the 20command line. Tests can be either individual test files or directories to 21search for tests (see :ref:`test-discovery`). 22 23Each specified test will be executed (potentially concurrently) and once all 24tests have been run :program:`lit` will print summary information on the number 25of tests which passed or failed (see :ref:`test-status-results`). The 26:program:`lit` program will execute with a non-zero exit code if any tests 27fail. 28 29By default :program:`lit` will use a succinct progress display and will only 30print summary information for test failures. See :ref:`output-options` for 31options controlling the :program:`lit` progress display and output. 32 33:program:`lit` also includes a number of options for controlling how tests are 34executed (specific features may depend on the particular test format). See 35:ref:`execution-options` for more information. 36 37Finally, :program:`lit` also supports additional options for only running a 38subset of the options specified on the command line, see 39:ref:`selection-options` for more information. 40 41:program:`lit` parses options from the environment variable ``LIT_OPTS`` after 42parsing options from the command line. ``LIT_OPTS`` is primarily useful for 43supplementing or overriding the command-line options supplied to :program:`lit` 44by ``check`` targets defined by a project's build system. 45 46Users interested in the :program:`lit` architecture or designing a 47:program:`lit` testing implementation should see :ref:`lit-infrastructure`. 48 49GENERAL OPTIONS 50--------------- 51 52.. option:: -h, --help 53 54 Show the :program:`lit` help message. 55 56.. option:: -j N, --workers=N 57 58 Run ``N`` tests in parallel. By default, this is automatically chosen to 59 match the number of detected available CPUs. 60 61.. option:: --config-prefix=NAME 62 63 Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for 64 test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`. 65 66.. option:: -D NAME[=VALUE], --param NAME[=VALUE] 67 68 Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty 69 string if not given). The meaning and use of these parameters is test suite 70 dependent. 71 72.. _output-options: 73 74OUTPUT OPTIONS 75-------------- 76 77.. option:: -q, --quiet 78 79 Suppress any output except for test failures. 80 81.. option:: -s, --succinct 82 83 Show less output, for example don't show information on tests that pass. 84 Also show a progress bar, unless ``--no-progress-bar`` is specified. 85 86.. option:: -v, --verbose 87 88 Show more information on test failures, for example the entire test output 89 instead of just the test result. 90 91.. option:: -vv, --echo-all-commands 92 93 Echo all commands to stdout, as they are being executed. 94 This can be valuable for debugging test failures, as the last echoed command 95 will be the one which has failed. 96 :program:`lit` normally inserts a no-op command (``:`` in the case of bash) 97 with argument ``'RUN: at line N'`` before each command pipeline, and this 98 option also causes those no-op commands to be echoed to stdout to help you 99 locate the source line of the failed command. 100 This option implies ``--verbose``. 101 102.. option:: -a, --show-all 103 104 Show more information about all tests, for example the entire test 105 commandline and output. 106 107.. option:: --no-progress-bar 108 109 Do not use curses based progress bar. 110 111.. option:: --show-unsupported 112 113 Show the names of unsupported tests. 114 115.. option:: --show-xfail 116 117 Show the names of tests that were expected to fail. 118 119.. _execution-options: 120 121EXECUTION OPTIONS 122----------------- 123 124.. option:: --path=PATH 125 126 Specify an additional ``PATH`` to use when searching for executables in tests. 127 128.. option:: --vg 129 130 Run individual tests under valgrind (using the memcheck tool). The 131 ``--error-exitcode`` argument for valgrind is used so that valgrind failures 132 will cause the program to exit with a non-zero status. 133 134 When this option is enabled, :program:`lit` will also automatically provide a 135 "``valgrind``" feature that can be used to conditionally disable (or expect 136 failure in) certain tests. 137 138.. option:: --vg-arg=ARG 139 140 When :option:`--vg` is used, specify an additional argument to pass to 141 :program:`valgrind` itself. 142 143.. option:: --vg-leak 144 145 When :option:`--vg` is used, enable memory leak checks. When this option is 146 enabled, :program:`lit` will also automatically provide a "``vg_leak``" 147 feature that can be used to conditionally disable (or expect failure in) 148 certain tests. 149 150.. option:: --time-tests 151 152 Track the wall time individual tests take to execute and includes the results 153 in the summary output. This is useful for determining which tests in a test 154 suite take the most time to execute. 155 156.. option:: --ignore-fail 157 158 Exit with status zero even if some tests fail. 159 160.. option:: --no-indirectly-run-check 161 162 Do not error if a test would not be run if the user had specified the 163 containing directory instead of naming the test directly. 164 165.. _selection-options: 166 167SELECTION OPTIONS 168----------------- 169 170By default, `lit` will run failing tests first, then run tests in descending 171execution time order to optimize concurrency. 172 173The timing data is stored in the `test_exec_root` in a file named 174`.lit_test_times.txt`. If this file does not exist, then `lit` checks the 175`test_source_root` for the file to optionally accelerate clean builds. 176 177.. option:: --shuffle 178 179 Run the tests in a random order, not failing/slowest first. 180 181.. option:: --max-failures N 182 183 Stop execution after the given number ``N`` of failures. 184 An integer argument should be passed on the command line 185 prior to execution. 186 187.. option:: --max-tests=N 188 189 Run at most ``N`` tests and then terminate. 190 191.. option:: --max-time=N 192 193 Spend at most ``N`` seconds (approximately) running tests and then terminate. 194 Note that this is not an alias for :option:`--timeout`; the two are 195 different kinds of maximums. 196 197.. option:: --num-shards=M 198 199 Divide the set of selected tests into ``M`` equal-sized subsets or 200 "shards", and run only one of them. Must be used with the 201 ``--run-shard=N`` option, which selects the shard to run. The environment 202 variable ``LIT_NUM_SHARDS`` can also be used in place of this 203 option. These two options provide a coarse mechanism for partitioning large 204 testsuites, for parallel execution on separate machines (say in a large 205 testing farm). 206 207.. option:: --run-shard=N 208 209 Select which shard to run, assuming the ``--num-shards=M`` option was 210 provided. The two options must be used together, and the value of ``N`` 211 must be in the range ``1..M``. The environment variable 212 ``LIT_RUN_SHARD`` can also be used in place of this option. 213 214.. option:: --timeout=N 215 216 Spend at most ``N`` seconds (approximately) running each individual test. 217 ``0`` means no time limit, and ``0`` is the default. Note that this is not an 218 alias for :option:`--max-time`; the two are different kinds of maximums. 219 220.. option:: --filter=REGEXP 221 222 Run only those tests whose name matches the regular expression specified in 223 ``REGEXP``. The environment variable ``LIT_FILTER`` can be also used in place 224 of this option, which is especially useful in environments where the call 225 to ``lit`` is issued indirectly. 226 227.. option:: --filter-out=REGEXP 228 229 Filter out those tests whose name matches the regular expression specified in 230 ``REGEXP``. The environment variable ``LIT_FILTER_OUT`` can be also used in 231 place of this option, which is especially useful in environments where the 232 call to ``lit`` is issued indirectly. 233 234.. option:: --xfail=LIST 235 236 Treat those tests whose name is in the semicolon separated list ``LIST`` as 237 ``XFAIL``. This can be helpful when one does not want to modify the test 238 suite. The environment variable ``LIT_XFAIL`` can be also used in place of 239 this option, which is especially useful in environments where the call to 240 ``lit`` is issued indirectly. 241 242ADDITIONAL OPTIONS 243------------------ 244 245.. option:: --debug 246 247 Run :program:`lit` in debug mode, for debugging configuration issues and 248 :program:`lit` itself. 249 250.. option:: --show-suites 251 252 List the discovered test suites and exit. 253 254.. option:: --show-tests 255 256 List all of the discovered tests and exit. 257 258EXIT STATUS 259----------- 260 261:program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS 262results. Otherwise, it will exit with the status 0. Other exit codes are used 263for non-test related failures (for example a user error or an internal program 264error). 265 266.. _test-discovery: 267 268TEST DISCOVERY 269-------------- 270 271The inputs passed to :program:`lit` can be either individual tests, or entire 272directories or hierarchies of tests to run. When :program:`lit` starts up, the 273first thing it does is convert the inputs into a complete list of tests to run 274as part of *test discovery*. 275 276In the :program:`lit` model, every test must exist inside some *test suite*. 277:program:`lit` resolves the inputs specified on the command line to test suites 278by searching upwards from the input path until it finds a :file:`lit.cfg` or 279:file:`lit.site.cfg` file. These files serve as both a marker of test suites 280and as configuration files which :program:`lit` loads in order to understand 281how to find and run the tests inside the test suite. 282 283Once :program:`lit` has mapped the inputs into test suites it traverses the 284list of inputs adding tests for individual files and recursively searching for 285tests in directories. 286 287This behavior makes it easy to specify a subset of tests to run, while still 288allowing the test suite configuration to control exactly how tests are 289interpreted. In addition, :program:`lit` always identifies tests by the test 290suite they are in, and their relative path inside the test suite. For 291appropriately configured projects, this allows :program:`lit` to provide 292convenient and flexible support for out-of-tree builds. 293 294.. _test-status-results: 295 296TEST STATUS RESULTS 297------------------- 298 299Each test ultimately produces one of the following eight results: 300 301**PASS** 302 303 The test succeeded. 304 305**FLAKYPASS** 306 307 The test succeeded after being re-run more than once. This only applies to 308 tests containing an ``ALLOW_RETRIES:`` annotation. 309 310**XFAIL** 311 312 The test failed, but that is expected. This is used for test formats which allow 313 specifying that a test does not currently work, but wish to leave it in the test 314 suite. 315 316**XPASS** 317 318 The test succeeded, but it was expected to fail. This is used for tests which 319 were specified as expected to fail, but are now succeeding (generally because 320 the feature they test was broken and has been fixed). 321 322**FAIL** 323 324 The test failed. 325 326**UNRESOLVED** 327 328 The test result could not be determined. For example, this occurs when the test 329 could not be run, the test itself is invalid, or the test was interrupted. 330 331**UNSUPPORTED** 332 333 The test is not supported in this environment. This is used by test formats 334 which can report unsupported tests. 335 336**TIMEOUT** 337 338 The test was run, but it timed out before it was able to complete. This is 339 considered a failure. 340 341Depending on the test format tests may produce additional information about 342their status (generally only for failures). See the :ref:`output-options` 343section for more information. 344 345.. _lit-infrastructure: 346 347LIT INFRASTRUCTURE 348------------------ 349 350This section describes the :program:`lit` testing architecture for users interested in 351creating a new :program:`lit` testing implementation, or extending an existing one. 352 353:program:`lit` proper is primarily an infrastructure for discovering and running 354arbitrary tests, and to expose a single convenient interface to these 355tests. :program:`lit` itself doesn't know how to run tests, rather this logic is 356defined by *test suites*. 357 358TEST SUITES 359~~~~~~~~~~~ 360 361As described in :ref:`test-discovery`, tests are always located inside a *test 362suite*. Test suites serve to define the format of the tests they contain, the 363logic for finding those tests, and any additional information to run the tests. 364 365:program:`lit` identifies test suites as directories containing ``lit.cfg`` or 366``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are 367initially discovered by recursively searching up the directory hierarchy for 368all the input files passed on the command line. You can use 369:option:`--show-suites` to display the discovered test suites at startup. 370 371Once a test suite is discovered, its config file is loaded. Config files 372themselves are Python modules which will be executed. When the config file is 373executed, two important global variables are predefined: 374 375**lit_config** 376 377 The global **lit** configuration object (a *LitConfig* instance), which defines 378 the builtin test formats, global configuration parameters, and other helper 379 routines for implementing test configurations. 380 381**config** 382 383 This is the config object (a *TestingConfig* instance) for the test suite, 384 which the config file is expected to populate. The following variables are also 385 available on the *config* object, some of which must be set by the config and 386 others are optional or predefined: 387 388 **name** *[required]* The name of the test suite, for use in reports and 389 diagnostics. 390 391 **test_format** *[required]* The test format object which will be used to 392 discover and run tests in the test suite. Generally this will be a builtin test 393 format available from the *lit.formats* module. 394 395 **test_source_root** The filesystem path to the test suite root. For out-of-dir 396 builds this is the directory that will be scanned for tests. 397 398 **test_exec_root** For out-of-dir builds, the path to the test suite root inside 399 the object directory. This is where tests will be run and temporary output files 400 placed. 401 402 **environment** A dictionary representing the environment to use when executing 403 tests in the suite. 404 405 **standalone_tests** When true, mark a directory with tests expected to be run 406 standalone. Test discovery is disabled for that directory and 407 *--no-indirectly-run-check* is in effect. *lit.suffixes* and *lit.excludes* 408 must be empty when this variable is true. 409 410 **suffixes** For **lit** test formats which scan directories for tests, this 411 variable is a list of suffixes to identify test files. Used by: *ShTest*. 412 413 **substitutions** For **lit** test formats which substitute variables into a test 414 script, the list of substitutions to perform. Used by: *ShTest*. 415 416 **unsupported** Mark an unsupported directory, all tests within it will be 417 reported as unsupported. Used by: *ShTest*. 418 419 **parent** The parent configuration, this is the config object for the directory 420 containing the test suite, or None. 421 422 **root** The root configuration. This is the top-most :program:`lit` configuration in 423 the project. 424 425 **pipefail** Normally a test using a shell pipe fails if any of the commands 426 on the pipe fail. If this is not desired, setting this variable to false 427 makes the test fail only if the last command in the pipe fails. 428 429 **available_features** A set of features that can be used in `XFAIL`, 430 `REQUIRES`, and `UNSUPPORTED` directives. 431 432TEST DISCOVERY 433~~~~~~~~~~~~~~ 434 435Once test suites are located, :program:`lit` recursively traverses the source 436directory (following *test_source_root*) looking for tests. When :program:`lit` 437enters a sub-directory, it first checks to see if a nested test suite is 438defined in that directory. If so, it loads that test suite recursively, 439otherwise it instantiates a local test config for the directory (see 440:ref:`local-configuration-files`). 441 442Tests are identified by the test suite they are contained within, and the 443relative path inside that suite. Note that the relative path may not refer to 444an actual file on disk; some test formats (such as *GoogleTest*) define 445"virtual tests" which have a path that contains both the path to the actual 446test file and a subpath to identify the virtual test. 447 448.. _local-configuration-files: 449 450LOCAL CONFIGURATION FILES 451~~~~~~~~~~~~~~~~~~~~~~~~~ 452 453When :program:`lit` loads a subdirectory in a test suite, it instantiates a 454local test configuration by cloning the configuration for the parent directory 455--- the root of this configuration chain will always be a test suite. Once the 456test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file 457in the subdirectory. If present, this file will be loaded and can be used to 458specialize the configuration for each individual directory. This facility can 459be used to define subdirectories of optional tests, or to change other 460configuration parameters --- for example, to change the test format, or the 461suffixes which identify test files. 462 463SUBSTITUTIONS 464~~~~~~~~~~~~~ 465 466:program:`lit` allows patterns to be substituted inside RUN commands. It also 467provides the following base set of substitutions, which are defined in 468TestRunner.py: 469 470 ======================= ============== 471 Macro Substitution 472 ======================= ============== 473 %s source path (path to the file currently being run) 474 %S source dir (directory of the file currently being run) 475 %p same as %S 476 %{pathsep} path separator 477 %t temporary file name unique to the test 478 %basename_t The last path component of %t but without the ``.tmp`` extension 479 %T parent directory of %t (not unique, deprecated, do not use) 480 %% % 481 %/s %s but ``\`` is replaced by ``/`` 482 %/S %S but ``\`` is replaced by ``/`` 483 %/p %p but ``\`` is replaced by ``/`` 484 %/t %t but ``\`` is replaced by ``/`` 485 %/T %T but ``\`` is replaced by ``/`` 486 %{/s:regex_replacement} %/s but escaped for use in the replacement of a ``s@@@`` command in sed 487 %{/S:regex_replacement} %/S but escaped for use in the replacement of a ``s@@@`` command in sed 488 %{/p:regex_replacement} %/p but escaped for use in the replacement of a ``s@@@`` command in sed 489 %{/t:regex_replacement} %/t but escaped for use in the replacement of a ``s@@@`` command in sed 490 %{/T:regex_replacement} %/T but escaped for use in the replacement of a ``s@@@`` command in sed 491 %:s On Windows, %/s but a ``:`` is removed if its the second character. 492 Otherwise, %s but with a single leading ``/`` removed. 493 %:S On Windows, %/S but a ``:`` is removed if its the second character. 494 Otherwise, %S but with a single leading ``/`` removed. 495 %:p On Windows, %/p but a ``:`` is removed if its the second character. 496 Otherwise, %p but with a single leading ``/`` removed. 497 %:t On Windows, %/t but a ``:`` is removed if its the second character. 498 Otherwise, %t but with a single leading ``/`` removed. 499 %:T On Windows, %/T but a ``:`` is removed if its the second character. 500 Otherwise, %T but with a single leading ``/`` removed. 501 ======================= ============== 502 503Other substitutions are provided that are variations on this base set and 504further substitution patterns can be defined by each test module. See the 505modules :ref:`local-configuration-files`. 506 507By default, substitutions are expanded exactly once, so that if e.g. a 508substitution ``%build`` is defined in top of another substitution ``%cxx``, 509``%build`` will expand to ``%cxx`` textually, not to what ``%cxx`` expands to. 510However, if the ``recursiveExpansionLimit`` property of the ``TestingConfig`` 511is set to a non-negative integer, substitutions will be expanded recursively 512until that limit is reached. It is an error if the limit is reached and 513expanding substitutions again would yield a different result. 514 515More detailed information on substitutions can be found in the 516:doc:`../TestingGuide`. 517 518TEST RUN OUTPUT FORMAT 519~~~~~~~~~~~~~~~~~~~~~~ 520 521The :program:`lit` output for a test run conforms to the following schema, in 522both short and verbose modes (although in short mode no PASS lines will be 523shown). This schema has been chosen to be relatively easy to reliably parse by 524a machine (for example in buildbot log scraping), and for other tools to 525generate. 526 527Each test result is expected to appear on a line that matches: 528 529.. code-block:: none 530 531 <result code>: <test name> (<progress info>) 532 533where ``<result-code>`` is a standard test result such as PASS, FAIL, XFAIL, 534XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and 535REGRESSED are also allowed. 536 537The ``<test name>`` field can consist of an arbitrary string containing no 538newline. 539 540The ``<progress info>`` field can be used to report progress information such 541as (1/300) or can be empty, but even when empty the parentheses are required. 542 543Each test result may include additional (multiline) log information in the 544following format: 545 546.. code-block:: none 547 548 <log delineator> TEST '(<test name>)' <trailing delineator> 549 ... log message ... 550 <log delineator> 551 552where ``<test name>`` should be the name of a preceding reported test, ``<log 553delineator>`` is a string of "*" characters *at least* four characters long 554(the recommended length is 20), and ``<trailing delineator>`` is an arbitrary 555(unparsed) string. 556 557The following is an example of a test run output which consists of four tests A, 558B, C, and D, and a log message for the failing test C: 559 560.. code-block:: none 561 562 PASS: A (1 of 4) 563 PASS: B (2 of 4) 564 FAIL: C (3 of 4) 565 ******************** TEST 'C' FAILED ******************** 566 Test 'C' failed as a result of exit code 1. 567 ******************** 568 PASS: D (4 of 4) 569 570LIT EXAMPLE TESTS 571~~~~~~~~~~~~~~~~~ 572 573The :program:`lit` distribution contains several example implementations of 574test suites in the *ExampleTests* directory. 575 576SEE ALSO 577-------- 578 579valgrind(1) 580