Lines Matching +full:llvm +full:- +full:test
4 Test Suite Structure
5 --------------------
7 The LLDB test suite consists of three different kinds of test:
10 * **Shell tests**: Integration tests that test the debugger through the command
12 driver or through ``lldb-test`` which is a tool that exposes the internal
13 data structures in an easy-to-parse way for testing. Most people will know
14 these as *lit tests* in LLVM, although lit is the test driver and ShellTest
15 is the test format that uses ``RUN:`` lines. `FileCheck
16 <https://llvm.org/docs/CommandGuide/FileCheck.html>`_ is used to verify
23 All three test suites use ``lit`` (`LLVM Integrated Tester
24 <https://llvm.org/docs/CommandGuide/lit.html>`_ ) as the test driver. The test
31 Unit tests are located under ``lldb/unittests``. If it's possible to test
32 something in isolation or as a single unit, you should make it a unit test.
35 process, in order to test something meaningful. We already have a handful of
42 Shell tests are located under ``lldb/test/Shell``. These tests are generally
44 ``lldb-test`` using ``FileCheck``. Shell tests are generally small and fast to
47 ``lldb-test`` is a relatively new addition to the test suite. It was the first
49 continuously extended with new subcommands, improving our test coverage. Among
54 the subcomponents already exposed by lldb-test. But when it comes to LLDB's
56 as the Python API. For example, to test setting a breakpoint, you could do it
62 means that you have to have a well-defined test scenario that you can easily
68 ``dotest.py`` test framework has extensive support for complex build scenarios
76 of (some) aspects of handling of binaries with non-native architectures or
80 and the test verifies the output. The debugger can be sensitive to its
82 that the same test might behave slightly differently on macOS and Linux.
83 Additionally, the debugger is an interactive tool, and the shell test provide
90 API tests are located under ``lldb/test/API``. They are run with the
91 ``dotest.py``. Tests are written in Python and test binaries (inferiors) are
92 compiled with Make. The majority of API tests are end-to-end tests that compile
97 several extensions and custom test primitives on top of what's offered by
100 `lldbtest.py <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/lldbtest.py>`_.
102 Below is the directory layout of the `example API test
103 <https://github.com/llvm/llvm-project/tree/main/lldb/test/API/sample_test>`_.
104 The test directory will always contain a python file, starting with ``Test``.
115 Let's start with the Python test file. Every test is its own class and can have
116 one or more test methods, that start with ``test_``. Many tests define
117 multiple test methods and share a bunch of common code. For example, for a
118 fictive test that makes sure we can set breakpoints we might have one test
128 <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/lldbutil.py>`_
130 Because we can't always audit every existing test, this is doubly true when
131 looking at an existing test for inspiration.
134 <https://ftp.gnu.org/old-gnu/Manuals/dejagnu-1.3/html_node/dejagnu_6.html>`_
145 it's even possible to define a function in a test case that determines whether
146 the test should be run or not.
153 test, the API test also allow for much more complex scenarios when it comes to
154 building inferiors. Every test has its own ``Makefile``, most of them only a
159 Here's an example of a simple ``Makefile`` used by the example test.
164 CFLAGS_EXTRAS := -std=c99
169 `Makefile.rules <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/make/Makefile.rules>`_
173 Another thing this enables is having different variants for the same test
174 case. By default, we run every test for two debug info formats, once with
176 DWARF (DWO) on Linux. But there are many more things we can test
177 that are orthogonal to the test itself. On GreenDragon we have a matrix bot
178 that runs the test suite under different configurations, with older host
183 to increase test coverage. It doesn't scale. It's easy to set up, but increases
186 The test variants are most useful when developing a larger feature (e.g. support
187 for a new DWARF version). The test suite contains a large number of fairly
188 generic tests, so running the test suite with the feature enabled is a good way
191 specific test covers, a random modification to the test case can make it start
196 For this reason, we recommend using test variants only while developing a new
197 feature. This can often be done by running the test suite with different
198 arguments -- without any modifications to the code. You can create a focused
199 test for any bug found that way. Often, there will be many tests failing, but a
203 In conclusion, you'll want to opt for an API test to test the API itself or
204 when you need the expressivity, either for the test case itself or for the
216 **Don't unnecessarily launch the test executable.**
218 expensive part of a test and should be avoided if possible. A large part
220 of the test executable.
225 `SBFunction`, `SBInstruction`, `SBCompileUnit`, etc.). For test executables
227 time (e.g., C and C++) there is also usually no process necessary to test
228 the `SBType`-related parts of the API. With those languages it's also
229 possible to test `SBValue` by running expressions with
235 Languages such as Objective-C that have a dependency on a runtime
238 **Don't unnecessarily include system headers in test sources.**
239 Including external headers slows down the compilation of the test executable
240 and it makes reproducing test failures on other operating systems or
243 **Avoid specifying test-specific compiler flags when including system headers.**
244 If a test requires including a system header (e.g., a test for a libc++
249 in your test (e.g., adding ``-DFOO`` to the ``Makefile`` or ``self.build``
250 arguments), then the test will not use the shared precompiled header cache
252 a specific compiler flag for the test, you can avoid this issue by either
253 removing all system header includes or decorating the test function with
257 **Test programs should be kept simple.**
258 Test executables should do the minimum amount of work to bring the process
259 into the state that is required for the test. Simulating a 'real' program
261 and makes the test much harder to debug and maintain. The test programs
263 random test values).
266 Often test programs need to declare functions and classes which require
270 Never choose identifiers that are already used anywhere else in LLVM or
273 people ``grep``'ing the LLVM repository for those strings.
277 of utility functions that can do common test setup tasks (e.g., starting a
278 test executable and running the process to a breakpoint). Using these
279 functions not only keeps the test shorter and free of duplicated code, but
280 they also follow best test suite practices and usually give much clearer
281 error messages if something goes wrong. The test utilities also contain
290 incorrect test results. Especially improved error messages that contain
293 check will unexpectedly pass if it's ran as the first expression in a test:
308 A better way to write the test above would be using LLDB's testing function
317 last option as they give non-descriptive error messages. The test class has
324 +-----------------------------------------------+-----------------------------------------------------------------+
326 +-----------------------------------------------+-----------------------------------------------------------------+
328 +-----------------------------------------------+-----------------------------------------------------------------+
330 +-----------------------------------------------+-----------------------------------------------------------------+
332 +-----------------------------------------------+-----------------------------------------------------------------+
345 **Do not use hard-coded line numbers in your test case.**
350 As an example, take a look at test/API/functionalities/breakpoint/breakpoint_conditions/main.c which has these
353 .. code-block:: c
359 .. code-block:: c
363 The Python test case TestBreakpointConditions.py uses the comment strings to find the line numbers during setUp(self) and use them
369 These features can be use to properly mark your test class or method for platform-specific tests, compiler specific, version specific.
371 As an example, take a look at test/API/lang/c/forward/TestForwardDeclaration.py which has these lines:
373 .. code-block:: python
381 """Test that we are able to find complete types when using DWARF v5
383 self.do_test(dict(CFLAGS_EXTRAS="-gdwarf-5 -gpubnames"))
385 This tells the test harness that unless we are running "linux" and clang version equal & above 8.0, the test should be skipped.
387 **Class-wise cleanup after yourself.**
389 TestBase.tearDownClass(cls) provides a mechanism to invoke the platform-specific cleanup after finishing with a test class. A test
390 class can have more than one test methods, so the tearDownClass(cls) method gets run after all the test methods have been executed by
391 the test harness.
393 The default cleanup action performed by the packages/Python/lldbsuite/test/lldbtest.py module invokes the "make clean" os command.
396 for example, in test/API/terminal/TestSTTYBeforeAndAfter.py:
398 .. code-block:: python
402 """Cleanup the test byproducts."""
406 The 'child_send1.txt' file gets generated during the test run, so it makes sense to explicitly spell out the action in the same
411 --
413 LLVM Buildbot is the place where volunteers provide machines for building and
414 testing. Everyone can `add a buildbot for LLDB <https://llvm.org/docs/HowToAddABuilder.html>`_.
418 `https://lab.llvm.org/buildbot/#/builders?tags=lldb <https://lab.llvm.org/buildbot/#/builders?tags=lldb>`_
421 has a dedicated tab for LLDB: `https://green.lab.llvm.org/job/llvm.org/view/LLDB/
422 <https://green.lab.llvm.org/job/llvm.org/view/LLDB/>`_
426 -----------------
431 debug interpreter, when running the test suite against a debug version of
439 Running the Full Test Suite
442 The easiest way to run the LLDB test suite is to use the ``check-lldb`` build
447 $ ninja check-lldb
449 Changing Test Suite Options
452 By default, the ``check-lldb`` target builds the test programs with the same
456 You can also add to the test runner options by setting the
460 It is possible to customize the architecture of the test binaries and compiler
461 used by appending ``-A`` and ``-C`` options respectively. For example, to test
462 LLDB against 32-bit binaries built with a custom version of clang, do:
466 $ cmake -DLLDB_TEST_USER_ARGS="-A;i386;-C;/path/to/custom/clang" -G Ninja
467 $ ninja check-lldb
469 Note that multiple ``-A`` and ``-C`` flags can be specified to
473 the ``--setting`` option of the test runner via this same variable. For example
474 ``--setting;target.disable-aslr=true``.
476 For a full list of test runner options, see
477 ``<build-dir>/bin/lldb-dotest --help``.
479 Running a Single Test Suite
482 Each test suite can be run separately, similar to running the whole test suite
483 with ``check-lldb``.
485 * Use ``check-lldb-unit`` to run just the unit tests.
486 * Use ``check-lldb-api`` to run just the SB API tests.
487 * Use ``check-lldb-shell`` to run just the shell tests.
491 target ``check-lldb-shell-objectfile``. However, because the unit tests and API
492 tests don't actually live under ``lldb/test``, this convenience is only
495 Running a Single Test
498 The recommended way to run a single test is by invoking the lit driver with a
499 filter. This ensures that the test is run with the same configuration as when
500 run as part of a test suite.
504 $ ./bin/llvm-lit -sv <llvm-project-root>/lldb/test --filter <test>
512 $ ./bin/llvm-lit -sv <llvm-project-root>/lldb/test/Shell/Commands/CommandScriptImmediateOutput
516 passing ``--param`` to lit and setting a value for ``dotest-args``.
520 $ ./bin/llvm-lit -sv <llvm-project-root>/lldb/test --param dotest-args='-C gcc'
523 Below is an overview of running individual test in the unit and API test suites
526 Running a Specific Test or Set of Tests: API Tests
529 In addition to running all the LLDB test suites with the ``check-lldb`` CMake
531 build you can use the ``lldb-dotest`` binary, which is a wrapper around
534 Alternatively, you can use ``dotest.py`` directly, if you want to run a test
535 one-off with a different configuration.
537 For example, to run the test cases defined in TestInferiorCrashing.py, run:
541 $ ./bin/lldb-dotest -p TestInferiorCrashing.py
545 $ cd $lldb/test
546 $ python dotest.py --executable <path-to-lldb> -p TestInferiorCrashing.py ../packages/Python/lldbsuite/test
548 If the test is not specified by name (e.g. if you leave the ``-p`` argument
554 $ ./bin/lldb-dotest functionalities/data-formatter
558 $ python dotest.py --executable <path-to-lldb> functionalities/data-formatter
564 $ python dotest.py -h
567 Running a Specific Test or Set of Tests: Unit Tests
572 To run them, just run the test binary, for example, to run all the Host tests:
579 To run a specific test, pass a filter, for example:
583 $ ./tools/lldb/unittests/Host/HostTests --gtest_filter=SocketTest.DomainListenConnectAccept
586 Running the Test Suite Remotely
589 Running the test-suite remotely is similar to the process of running a local
590 test suite, but there are two things to have in mind:
592 1. You must have the lldb-server running on the remote system, ready to accept
595 2. You must tell the test-suite how to connect to the remote system. This is
597 flags to cmake, and ``--platform-name`` parameter to ``dotest.py``.
603 test sources. It can be disabled by setting ``LLDB_TEST_SHELL_DISABLE_REMOTE=On``.
611 QEMU can be used to test LLDB in an emulation environment in the absence of
612 actual hardware. :doc:`/resources/qemu-testing` describes how to setup an
614 ``llvm-project/lldb/scripts/lldb-test-qemu``. These scripts currently
617 Debugging Test Failures
618 -----------------------
620 On non-Windows platforms, you can use the ``-d`` option to ``dotest.py`` which
621 will cause the script to print out the pid of the test and wait for a while
622 until a debugger is attached. Then run ``lldb -p <pid>`` to attach.
624 To instead debug a test's python source, edit the test and insert ``import pdb; pdb.set_trace()`` or ``breakpoint()`` (Python 3 only) at the point you want to start debugging. The ``breakpoint()`` command can be used for any LLDB Python script, not just for API tests.
634 Debugging Test Failures on Windows
638 for debugging test failures. It can seamlessly step between native and managed
639 code, which is very helpful when you need to step through the test itself, and
640 then into the LLDB code that backs the operations the test is performing.
646 #. Go to File -> New -> Project -> Python -> From Existing Python Code.
647 #. Choose llvm/tools/lldb as the directory containing the Python code.
648 #. When asked where to save the .pyproj file, choose the folder ``llvm/tools/lldb/pyproj``. This is a special folder that is ignored by the ``.gitignore`` file, since it is not checked in.
649 #. Set test/dotest.py as the startup file
650 #. Make sure there is a Python Environment installed for your distribution. For example, if you installed Python to ``C:\Python35``, PTVS needs to know that this is the interpreter you want to use for running the test suite.
651 #. Go to Tools -> Options -> Python Tools -> Environment Options
656 #. In Debug/Search Paths, enter the path to your ninja/lib/site-packages directory.
658 #. If you want to enabled mixed mode debugging, check Enable native code debugging (this slows down debugging, so enable it only on an as-needed basis.)
659 #. Set the command line for the test suite to run.
666 --arch=i686
668 --executable D:/src/llvmbuild/ninja/bin/lldb.exe
670 -s D:/src/llvmbuild/ninja/lldb-test-traces
671 -u CXXFLAGS -u CFLAGS
672 # If a test crashes, show JIT debugging dialog.
673 --enable-crash-dialog
675 -C d:\src\llvmbuild\ninja_release\bin\clang.exe
676 # Path to the particular test you want to debug.
677 -p TestPaths.py
678 # Root of test tree
679 D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test
683 --arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe -p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess
685 .. [#] `https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName <https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName>`_