Lines Matching +full:llvm +full:- +full:test
2 test-suite Makefile Guide (deprecated)
11 First, all tests are executed within the LLVM object directory tree.
12 They *are not* executed inside of the LLVM source tree. This is because
13 the test suite creates temporary files during execution.
15 To run the test suite, you need to use the following steps:
17 #. Check out the ``test-suite`` module with:
19 .. code-block:: bash
21 % git clone https://github.com/llvm/llvm-test-suite.git test-suite
26 #. Configure and build ``llvm``.
28 #. Configure and build ``llvm-gcc``.
30 #. Install ``llvm-gcc`` somewhere.
32 #. *Re-configure* ``llvm`` from the top level of each build tree (LLVM
33 object directory tree) in which you want to run the test suite, just
34 as you do before building LLVM.
36 During the *re-configuration*, you must either: (1) have ``llvm-gcc``
38 just-built ``llvm-gcc`` is installed using
39 ``--with-llvmgccdir=$LLVM_GCC_DIR``.
41 You must also tell the configure machinery that the test suite is
44 .. code-block:: bash
46 % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]
49 *installed* llvm-gcc, not its src or obj directory.]
51 #. You can now run the test suite from your build tree as follows:
53 .. code-block:: bash
55 % cd $LLVM_OBJ_ROOT/projects/test-suite
60 again (unless the test code or configure script changes).
65 In order to run the External tests in the ``test-suite`` module, you
66 must specify *--with-externals*. This must be done during the
67 *re-configuration* step (see above), and the ``llvm`` re-configuration
68 must recognize the previously-built ``llvm-gcc``. If any of these is
71 * *--with-externals*
73 * *--with-externals=<directory>*
75 This tells LLVM where to find any external tests. They are expected to
79 names known to LLVM include:
95 In addition to the regular "whole program" tests, the ``test-suite``
97 ways. If the variable TEST is defined on the ``gmake`` command line, the
98 test system will include a Makefile named
99 ``TEST.<value of TEST variable>.Makefile``. This Makefile can modify
102 For example, the LLVM nightly tester uses ``TEST.nightly.Makefile`` to
103 create the nightly test reports. To run the nightly tests, run
104 ``gmake TEST=nightly``.
106 There are several TEST Makefiles available in the tree. Some of them are
107 designed for internal LLVM research and will not work outside of the
108 LLVM research group. They may still be valuable, however, as a guide to
109 writing your own TEST Makefile for any optimization or analysis passes
110 that you develop with LLVM.
112 Generating Test Output
122 Somewhat better is running ``gmake TEST=sometest test``, which runs the
123 specified test and usually adds per-program summaries to the output
124 (depending on which sometest you use). For example, the ``nightly`` test
125 explicitly outputs TEST-PASS or TEST-FAIL for every test after each
131 exact contents of the report are dependent on which ``TEST`` you are
134 running with ``TEST=<type>``). The ``report`` also generate a file
136 test run.
138 Writing Custom Tests for the test-suite
141 Assuming you can run the test suite, (e.g.
142 "``gmake TEST=nightly report``" should work), it is really easy to run
148 Lets say that you have an LLVM optimization pass, and you want to see
149 how many times it triggers. First thing you should do is add an LLVM
153 Following this, you can set up a test and a report that collects these
155 "``test-suite/TEST.XXX.Makefile``" fragment (where XXX is the name of
156 your test) and a "``test-suite/TEST.XXX.report``" file that indicates
158 various levels of sophistication included with the test suite, and the
162 "libcalls" test as an example. It can be run like this:
164 .. code-block:: bash
166 % cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level
167 % make TEST=libcalls report
182 Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
183 Prolangs-C/agrep/agrep | 33 | 12 |
184 Prolangs-C/allroots/allroots | * | * |
185 Prolangs-C/assembler/assembler | 47 | * |
186 Prolangs-C/bison/mybison | 74 | * |
189 This basically is grepping the -stats output and displaying it in a
190 table. You can also use the "TEST=libcalls report.html" target to get
193 The source for this is in ``test-suite/TEST.libcalls.*``. The format is
194 pretty simple: the Makefile indicates how to run the test (in this case,
195 "``opt -simplify-libcalls -stats``"), and the report contains one line