Lines Matching +full:build +full:- +full:docs
1 test-suite Guide
5 ----------
8 from an LLVM build:
11 % <path to llvm build>/bin/llvm-lit --version
19 % python3 -m venv .venv
21 % pip install git+https://github.com/llvm/llvm-project.git#subdirectory=llvm/utils/lit
22 % lit --version
31 % python3 -m venv .venv
34 % lit --version
41 2. Check out the `test-suite` module with:
44 % git clone https://github.com/llvm/llvm-test-suite.git test-suite
47 3. Create a build directory and use CMake to configure the suite. Use the
49 to choose a typical build configuration:
52 % mkdir test-suite-build
53 % cd test-suite-build
54 % cmake -DCMAKE_C_COMPILER=<path to llvm build>/bin/clang \
55 -C../test-suite/cmake/caches/O3.cmake \
56 ../test-suite
59 **NOTE!** if you are using your built clang, and you want to build and run the
60 MicroBenchmarks/XRay microbenchmarks, you need to add `compiler-rt` to your
63 4. Build the benchmarks:
67 Scanning dependencies of target timeit-target
68 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
69 [ 0%] Linking C executable timeit-target
76 % llvm-lit -v -j 1 -o results.json .
77 -- Testing: 474 tests, 1 threads --
78 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
79 ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
85 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
88 **NOTE!** even in the case you only want to get the compile-time results(code size, llvm stats etc),
89 you need to run the test with the above `llvm-lit` command. In that case, the *results.json* file will
90 contain compile-time metrics.
98 % test-suite/utils/compare.py results.json
100 % test-suite/utils/compare.py results_a.json results_b.json
105 ---------
107 The test-suite contains benchmark and test programs. The programs come with
112 The test-suite is divided into several directories:
114 - `SingleSource/`
119 - `MultiSource/`
124 - `MicroBenchmarks/`
126 Programs using the [google-benchmark](https://github.com/google/benchmark)
130 - `External/`
133 distributed with the test-suite. The most prominent members of this
135 See [External Suites](#external-suites).
137 - `Bitcode/`
141 - `CTMark/`
155 |----------------------|-----------|-------------------------------|----------------------|
161 | DOE-ProxyApps-C++ | C++ | HPC/scientific apps | Small applications, representative of our larger DOE workloads |
162 | DOE-ProxyApps-C | C | HPC/scientific apps | " |
163 | Fhourstones | C | Game/solver | Integer benchmark that efficiently solves positions in the game of Connect-4 |
164 | Fhourstones-3.1 | C | Game/solver | " |
166 | llubenchmark | C | Linked-list micro-benchmark | |
173 | NPB-serial | C | Parallel computing | Serial version of the NPB IS code |
177 | Prolangs-C++ | C++ | Benchmark suite | city, employ, life, NP, ocean, primes, simul, vcirc |
178 | Prolangs-C | C | Benchmark suite | agrep, archie-client, bison, gnugo, unix-smail |
179 | Ptrdist | C | Pointer-Intensive Benchmark Suite | |
181 | SciMark2-C | C | Scientific apps | FFT, LU, Montecarlo, sparse matmul |
182 | sim | C | Dynamic programming | A Time-Efficient, Linear-Space Local Similarity Algorithm |
183 | tramp3d-v4 | C++ | Numerical analysis | Template-intensive numerical program based on FreePOOMA |
192 -------------
194 The test-suite has configuration options to customize building and running the
198 % cd test-suite-build
200 % cmake -LH
202 % cmake -LAH
207 - `CMAKE_C_FLAGS`
213 - `CMAKE_C_COMPILER`
220 - `CMAKE_Fortran_COMPILER`
225 - `CMAKE_BUILD_TYPE`
227 Select a build type like `OPTIMIZE` or `DEBUG` selecting a set of predefined
232 - `TEST_SUITE_FORTRAN`
235 found in the [Flang documentation](https://flang.llvm.org/docs/FortranLLVMTestSuite.html)
237 - `TEST_SUITE_RUN_UNDER`
240 cross-compiled tests within a simulator tool.
242 - `TEST_SUITE_BENCHMARKING_ONLY`
248 - `TEST_SUITE_SUBDIRS`
250 Semicolon-separated list of directories to include. This can be used to only
251 build parts of the test-suite or to include external suites. This option
255 - `TEST_SUITE_COLLECT_STATS`
257 Collect internal LLVM statistics. Appends `-save-stats=obj` when invoking the
260 - `TEST_SUITE_RUN_BENCHMARKS`
263 collect build statistics like compile time and code size.
265 - `TEST_SUITE_USE_PERF`
268 comes with the test-suite. The `perf` is usually available on linux systems.
270 - `TEST_SUITE_SPEC2000_ROOT`, `TEST_SUITE_SPEC2006_ROOT`, `TEST_SUITE_SPEC2017_ROOT`, ...
278 - `-GNinja`
280 Generate build files for the ninja build tool.
282 - `-Ctest-suite/cmake/caches/<cachefile.cmake>`
284 Use a CMake cache. The test-suite comes with several CMake caches which
285 predefine common or tricky build configurations.
289 --------------------------------
292 produced when invoking lit with the `-o filename.json` flag.
296 - Basic Usage:
299 % test-suite/utils/compare.py baseline.json
300 Warning: 'test-suite :: External/SPEC/CINT2006/403.gcc/403.gcc.test' has No metrics!
320 - Show compile_time or text segment size metrics:
323 % test-suite/utils/compare.py -m compile_time baseline.json
324 % test-suite/utils/compare.py -m size.__text baseline.json
327 - Compare two result files and filter short running tests:
330 % test-suite/utils/compare.py --filter-short baseline.json experiment.json
334 SingleSour.../Benchmarks/Linpack/linpack-pc 5.16 4.30 -16.5%
335 MultiSourc...erolling-dbl/LoopRerolling-dbl 7.01 7.86 12.2%
336 SingleSour...UnitTests/Vectorizer/gcc-loops 3.89 3.54 -9.0%
340 - Merge multiple baseline and experiment result files by taking the minimum
344 % test-suite/utils/compare.py base0.json base1.json base2.json vs exp0.json exp1.json exp2.json
351 [https://llvm.org/docs/lnt](https://llvm.org/docs/lnt). The official LNT instance
356 ---------------
360 - placing (or linking) them into the `test-suite/test-suite-externals/xxx` directory (example: `test-suite/test-suite-externals/speccpu2000`)
361 - using a configuration option such as `-D TEST_SUITE_SPEC2000_ROOT=path/to/speccpu2000`
364 `test-suite/External/SPEC/README`.
372 llvm-test-suite, a lot more of potentially vectorizable code gets compiled
373 - which can catch compiler bugs merely by triggering code generation asserts.
377 available.) The integration into llvm-test-suite doesn't run the projects'
381 `llvm-test-suite/External` for further details.
385 -------------
387 You can build custom suites using the test-suite infrastructure. A custom suite
389 picked up automatically if placed into a subdirectory of the test-suite or when
393 % cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite
398 ---------------------------
403 using `llvm-profdata` so they can be used by the second compilation run.
408 % cmake -DTEST_SUITE_PROFILE_GENERATE=ON \
409 -DTEST_SUITE_USE_IR_PGO=ON \
410 -DTEST_SUITE_RUN_TYPE=train \
411 ../test-suite
413 % llvm-lit .
415 % cmake -DTEST_SUITE_PROFILE_GENERATE=OFF \
416 -DTEST_SUITE_PROFILE_USE=ON \
417 -DTEST_SUITE_RUN_TYPE=ref \
420 % llvm-lit -o result.json .
423 To use Clang frontend's PGO instead of LLVM IR PGO, set `-DTEST_SUITE_USE_IR_PGO=OFF`.
429 --------------------------------------
436 - [https://llvm.org/docs/lnt/tests.html#cross-compiling](https://llvm.org/docs/lnt/tests.html#cross-compiling)
438 - [https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html](https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html)
441 `test-suite/cmake/caches/target-target-*-iphoneos-internal.cmake` CMake cache
448 - Via SSH connection to an external device: The `TEST_SUITE_REMOTE_HOST` option
458 % cmake -G Ninja -D CMAKE_C_COMPILER=path/to/clang \
459 -C ../test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake \
460 -D CMAKE_BUILD_TYPE=Release \
461 -D TEST_SUITE_REMOTE_HOST=mydevice \
462 ../test-suite
465 % llvm-lit -j1 -o result.json .
468 - You can specify a simulator for the target machine with the
473 Running the test-suite via LNT
474 ------------------------------
476 The LNT tool can run the test-suite. Use this when submitting test results to
478 [https://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite](https://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite)
481 Running the test-suite via Makefiles (deprecated)
482 -------------------------------------------------
484 **Note**: The test-suite comes with a set of Makefiles that are considered
489 [test-suite Makefile Guide](TestSuiteMakefileGuide).