1<html lang="en"> 2<head> 3<title>Installing GCC: Testing</title> 4<meta http-equiv="Content-Type" content="text/html"> 5<meta name="description" content="Installing GCC: Testing"> 6<meta name="generator" content="makeinfo 4.13"> 7<link title="Top" rel="top" href="#Top"> 8<link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage"> 9<!-- 10Copyright (C) 1988-2017 Free Software Foundation, Inc. 11 12 Permission is granted to copy, distribute and/or modify this document 13under the terms of the GNU Free Documentation License, Version 1.3 or 14any later version published by the Free Software Foundation; with no 15Invariant Sections, the Front-Cover texts being (a) (see below), and 16with the Back-Cover Texts being (b) (see below). A copy of the 17license is included in the section entitled "GNU Free Documentation License". 18 19(a) The FSF's Front-Cover Text is: 20 21 A GNU Manual 22 23(b) The FSF's Back-Cover Text is: 24 25 You have freedom to copy and modify this GNU Manual, like GNU 26 software. Copies published by the Free Software Foundation raise 27 funds for GNU development.--> 28<meta http-equiv="Content-Style-Type" content="text/css"> 29<style type="text/css"><!-- 30 pre.display { font-family:inherit } 31 pre.format { font-family:inherit } 32 pre.smalldisplay { font-family:inherit; font-size:smaller } 33 pre.smallformat { font-family:inherit; font-size:smaller } 34 pre.smallexample { font-size:smaller } 35 pre.smalllisp { font-size:smaller } 36 span.sc { font-variant:small-caps } 37 span.roman { font-family:serif; font-weight:normal; } 38 span.sansserif { font-family:sans-serif; font-weight:normal; } 39--></style> 40</head> 41<body> 42<h1 class="settitle">Installing GCC: Testing</h1> 43<a name="index-Testing-1"></a><a name="index-Installing-GCC_003a-Testing-2"></a><a name="index-Testsuite-3"></a> 44Before you install GCC, we encourage you to run the testsuites and to 45compare your results with results from a similar configuration that have 46been submitted to the 47<a href="http://gcc.gnu.org/ml/gcc-testresults/">gcc-testresults mailing list</a>. 48Some of these archived results are linked from the build status lists 49at <a href="http://gcc.gnu.org/buildstat.html">http://gcc.gnu.org/buildstat.html</a>, although not everyone who 50reports a successful build runs the testsuites and submits the results. 51This step is optional and may require you to download additional software, 52but it can give you confidence in your new GCC installation or point out 53problems before you install and start using your new GCC. 54 55 <p>First, you must have <a href="download.html">downloaded the testsuites</a>. 56These are part of the full distribution, but if you downloaded the 57“core” compiler plus any front ends, you must download the testsuites 58separately. 59 60 <p>Second, you must have the testing tools installed. This includes 61<a href="http://www.gnu.org/software/dejagnu/">DejaGnu</a>, Tcl, and Expect; 62the DejaGnu site has links to these. For running the BRIG frontend 63tests, a tool to assemble the binary BRIGs from HSAIL text, 64<a href="https://github.com/HSAFoundation/HSAIL-Tools/">HSAILasm</a> must 65be installed. 66 67 <p>If the directories where <samp><span class="command">runtest</span></samp> and <samp><span class="command">expect</span></samp> were 68installed are not in the <samp><span class="env">PATH</span></samp>, you may need to set the following 69environment variables appropriately, as in the following example (which 70assumes that DejaGnu has been installed under <samp><span class="file">/usr/local</span></samp>): 71 72<pre class="smallexample"> TCL_LIBRARY = /usr/local/share/tcl8.0 73 DEJAGNULIBS = /usr/local/share/dejagnu 74</pre> 75 <p>(On systems such as Cygwin, these paths are required to be actual 76paths, not mounts or links; presumably this is due to some lack of 77portability in the DejaGnu code.) 78 79 <p>Finally, you can run the testsuite (which may take a long time): 80<pre class="smallexample"> cd <var>objdir</var>; make -k check 81</pre> 82 <p>This will test various components of GCC, such as compiler 83front ends and runtime libraries. While running the testsuite, DejaGnu 84might emit some harmless messages resembling 85‘<samp><span class="samp">WARNING: Couldn't find the global config file.</span></samp>’ or 86‘<samp><span class="samp">WARNING: Couldn't find tool init file</span></samp>’ that can be ignored. 87 88 <p>If you are testing a cross-compiler, you may want to run the testsuite 89on a simulator as described at <a href="http://gcc.gnu.org/simtest-howto.html">http://gcc.gnu.org/simtest-howto.html</a>. 90 91<h3 class="section"><a name="TOC0"></a>How can you run the testsuite on selected tests?</h3> 92 93<p>In order to run sets of tests selectively, there are targets 94‘<samp><span class="samp">make check-gcc</span></samp>’ and language specific ‘<samp><span class="samp">make check-c</span></samp>’, 95‘<samp><span class="samp">make check-c++</span></samp>’, ‘<samp><span class="samp">make check-fortran</span></samp>’, 96‘<samp><span class="samp">make check-ada</span></samp>’, ‘<samp><span class="samp">make check-objc</span></samp>’, ‘<samp><span class="samp">make check-obj-c++</span></samp>’, 97‘<samp><span class="samp">make check-lto</span></samp>’ 98in the <samp><span class="file">gcc</span></samp> subdirectory of the object directory. You can also 99just run ‘<samp><span class="samp">make check</span></samp>’ in a subdirectory of the object directory. 100 101 <p>A more selective way to just run all <samp><span class="command">gcc</span></samp> execute tests in the 102testsuite is to use 103 104<pre class="smallexample"> make check-gcc RUNTESTFLAGS="execute.exp <var>other-options</var>" 105</pre> 106 <p>Likewise, in order to run only the <samp><span class="command">g++</span></samp> “old-deja” tests in 107the testsuite with filenames matching ‘<samp><span class="samp">9805*</span></samp>’, you would use 108 109<pre class="smallexample"> make check-g++ RUNTESTFLAGS="old-deja.exp=9805* <var>other-options</var>" 110</pre> 111 <p>The <samp><span class="file">*.exp</span></samp> files are located in the testsuite directories of the GCC 112source, the most important ones being <samp><span class="file">compile.exp</span></samp>, 113<samp><span class="file">execute.exp</span></samp>, <samp><span class="file">dg.exp</span></samp> and <samp><span class="file">old-deja.exp</span></samp>. 114To get a list of the possible <samp><span class="file">*.exp</span></samp> files, pipe the 115output of ‘<samp><span class="samp">make check</span></samp>’ into a file and look at the 116‘<samp><span class="samp">Running ... .exp</span></samp>’ lines. 117 118<h3 class="section"><a name="TOC1"></a>Passing options and running multiple testsuites</h3> 119 120<p>You can pass multiple options to the testsuite using the 121‘<samp><span class="samp">--target_board</span></samp>’ option of DejaGNU, either passed as part of 122‘<samp><span class="samp">RUNTESTFLAGS</span></samp>’, or directly to <samp><span class="command">runtest</span></samp> if you prefer to 123work outside the makefiles. For example, 124 125<pre class="smallexample"> make check-g++ RUNTESTFLAGS="--target_board=unix/-O3/-fmerge-constants" 126</pre> 127 <p>will run the standard <samp><span class="command">g++</span></samp> testsuites (“unix” is the target name 128for a standard native testsuite situation), passing 129‘<samp><span class="samp">-O3 -fmerge-constants</span></samp>’ to the compiler on every test, i.e., 130slashes separate options. 131 132 <p>You can run the testsuites multiple times using combinations of options 133with a syntax similar to the brace expansion of popular shells: 134 135<pre class="smallexample"> ..."--target_board=arm-sim\{-mhard-float,-msoft-float\}\{-O1,-O2,-O3,\}" 136</pre> 137 <p>(Note the empty option caused by the trailing comma in the final group.) 138The following will run each testsuite eight times using the ‘<samp><span class="samp">arm-sim</span></samp>’ 139target, as if you had specified all possible combinations yourself: 140 141<pre class="smallexample"> --target_board='arm-sim/-mhard-float/-O1 \ 142 arm-sim/-mhard-float/-O2 \ 143 arm-sim/-mhard-float/-O3 \ 144 arm-sim/-mhard-float \ 145 arm-sim/-msoft-float/-O1 \ 146 arm-sim/-msoft-float/-O2 \ 147 arm-sim/-msoft-float/-O3 \ 148 arm-sim/-msoft-float' 149</pre> 150 <p>They can be combined as many times as you wish, in arbitrary ways. This 151list: 152 153<pre class="smallexample"> ..."--target_board=unix/-Wextra\{-O3,-fno-strength\}\{-fomit-frame,\}" 154</pre> 155 <p>will generate four combinations, all involving ‘<samp><span class="samp">-Wextra</span></samp>’. 156 157 <p>The disadvantage to this method is that the testsuites are run in serial, 158which is a waste on multiprocessor systems. For users with GNU Make and 159a shell which performs brace expansion, you can run the testsuites in 160parallel by having the shell perform the combinations and <samp><span class="command">make</span></samp> 161do the parallel runs. Instead of using ‘<samp><span class="samp">--target_board</span></samp>’, use a 162special makefile target: 163 164<pre class="smallexample"> make -j<var>N</var> check-<var>testsuite</var>//<var>test-target</var>/<var>option1</var>/<var>option2</var>/... 165</pre> 166 <p>For example, 167 168<pre class="smallexample"> make -j3 check-gcc//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu} 169</pre> 170 <p>will run three concurrent “make-gcc” testsuites, eventually testing all 171ten combinations as described above. Note that this is currently only 172supported in the <samp><span class="file">gcc</span></samp> subdirectory. (To see how this works, try 173typing <samp><span class="command">echo</span></samp> before the example given here.) 174 175<h3 class="section"><a name="TOC2"></a>How to interpret test results</h3> 176 177<p>The result of running the testsuite are various <samp><span class="file">*.sum</span></samp> and <samp><span class="file">*.log</span></samp> 178files in the testsuite subdirectories. The <samp><span class="file">*.log</span></samp> files contain a 179detailed log of the compiler invocations and the corresponding 180results, the <samp><span class="file">*.sum</span></samp> files summarize the results. These summaries 181contain status codes for all tests: 182 183 <ul> 184<li>PASS: the test passed as expected 185<li>XPASS: the test unexpectedly passed 186<li>FAIL: the test unexpectedly failed 187<li>XFAIL: the test failed as expected 188<li>UNSUPPORTED: the test is not supported on this platform 189<li>ERROR: the testsuite detected an error 190<li>WARNING: the testsuite detected a possible problem 191</ul> 192 193 <p>It is normal for some tests to report unexpected failures. At the 194current time the testing harness does not allow fine grained control 195over whether or not a test is expected to fail. This problem should 196be fixed in future releases. 197 198<h3 class="section"><a name="TOC3"></a>Submitting test results</h3> 199 200<p>If you want to report the results to the GCC project, use the 201<samp><span class="file">contrib/test_summary</span></samp> shell script. Start it in the <var>objdir</var> with 202 203<pre class="smallexample"> <var>srcdir</var>/contrib/test_summary -p your_commentary.txt \ 204 -m gcc-testresults@gcc.gnu.org |sh 205</pre> 206 <p>This script uses the <samp><span class="command">Mail</span></samp> program to send the results, so 207make sure it is in your <samp><span class="env">PATH</span></samp>. The file <samp><span class="file">your_commentary.txt</span></samp> is 208prepended to the testsuite summary and should contain any special 209remarks you have on your results or your build environment. Please 210do not edit the testsuite result block or the subject line, as these 211messages may be automatically processed. 212 213 <p><hr /> 214<p><a href="./index.html">Return to the GCC Installation page</a> 215 216<!-- ***Final install*********************************************************** --> 217<!-- ***Binaries**************************************************************** --> 218<!-- ***Specific**************************************************************** --> 219<!-- ***Old documentation****************************************************** --> 220<!-- ***GFDL******************************************************************** --> 221<!-- *************************************************************************** --> 222<!-- Part 6 The End of the Document --> 223</body></html> 224 225