1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> 2<html> 3<!-- Created by GNU Texinfo 6.8, https://www.gnu.org/software/texinfo/ --> 4<head> 5<meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 6<!-- Copyright (C) 1988-2020 Free Software Foundation, Inc. 7 8Permission is granted to copy, distribute and/or modify this document 9under the terms of the GNU Free Documentation License, Version 1.3 or 10any later version published by the Free Software Foundation; with no 11Invariant Sections, the Front-Cover texts being (a) (see below), and 12with the Back-Cover Texts being (b) (see below). A copy of the 13license is included in the section entitled "GNU 14Free Documentation License". 15 16(a) The FSF's Front-Cover Text is: 17 18A GNU Manual 19 20(b) The FSF's Back-Cover Text is: 21 22You have freedom to copy and modify this GNU Manual, like GNU 23 software. Copies published by the Free Software Foundation raise 24 funds for GNU development. --> 25<title>Installing GCC: Testing</title> 26 27<meta name="description" content="Installing GCC: Testing"> 28<meta name="keywords" content="Installing GCC: Testing"> 29<meta name="resource-type" content="document"> 30<meta name="distribution" content="global"> 31<meta name="Generator" content="makeinfo"> 32<meta name="viewport" content="width=device-width,initial-scale=1"> 33 34<style type="text/css"> 35<!-- 36a.copiable-anchor {visibility: hidden; text-decoration: none; line-height: 0em} 37a.summary-letter {text-decoration: none} 38blockquote.indentedblock {margin-right: 0em} 39div.display {margin-left: 3.2em} 40div.example {margin-left: 3.2em} 41kbd {font-style: oblique} 42pre.display {font-family: inherit} 43pre.format {font-family: inherit} 44pre.menu-comment {font-family: serif} 45pre.menu-preformatted {font-family: serif} 46span.nolinebreak {white-space: nowrap} 47span.roman {font-family: initial; font-weight: normal} 48span.sansserif {font-family: sans-serif; font-weight: normal} 49span:hover a.copiable-anchor {visibility: visible} 50ul.no-bullet {list-style: none} 51--> 52</style> 53 54 55</head> 56 57<body lang="en"> 58<h1 class="settitle" align="center">Installing GCC: Testing</h1> 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80<span id="index-Testing"></span> 81<span id="index-Installing-GCC_003a-Testing"></span> 82<span id="index-Testsuite"></span> 83 84<p>Before you install GCC, we encourage you to run the testsuites and to 85compare your results with results from a similar configuration that have 86been submitted to the 87<a href="http://gcc.gnu.org/ml/gcc-testresults/">gcc-testresults mailing list</a>. 88Some of these archived results are linked from the build status lists 89at <a href="http://gcc.gnu.org/buildstat.html">http://gcc.gnu.org/buildstat.html</a>, although not everyone who 90reports a successful build runs the testsuites and submits the results. 91This step is optional and may require you to download additional software, 92but it can give you confidence in your new GCC installation or point out 93problems before you install and start using your new GCC. 94</p> 95<p>First, you must have <a href="download.html">downloaded the testsuites</a>. 96These are part of the full distribution, but if you downloaded the 97“core” compiler plus any front ends, you must download the testsuites 98separately. 99</p> 100<p>Second, you must have the testing tools installed. This includes 101<a href="http://www.gnu.org/software/dejagnu/">DejaGnu</a>, Tcl, and Expect; 102the DejaGnu site has links to these. For running the BRIG frontend 103tests, a tool to assemble the binary BRIGs from HSAIL text, 104<a href="https://github.com/HSAFoundation/HSAIL-Tools/">HSAILasm</a> must 105be installed. 106</p> 107<p>If the directories where <code>runtest</code> and <code>expect</code> were 108installed are not in the <code>PATH</code>, you may need to set the following 109environment variables appropriately, as in the following example (which 110assumes that DejaGnu has been installed under <samp>/usr/local</samp>): 111</p> 112<div class="example"> 113<pre class="example">TCL_LIBRARY = /usr/local/share/tcl8.0 114DEJAGNULIBS = /usr/local/share/dejagnu 115</pre></div> 116 117<p>(On systems such as Cygwin, these paths are required to be actual 118paths, not mounts or links; presumably this is due to some lack of 119portability in the DejaGnu code.) 120</p> 121 122<p>Finally, you can run the testsuite (which may take a long time): 123</p><div class="example"> 124<pre class="example">cd <var>objdir</var>; make -k check 125</pre></div> 126 127<p>This will test various components of GCC, such as compiler 128front ends and runtime libraries. While running the testsuite, DejaGnu 129might emit some harmless messages resembling 130‘<samp>WARNING: Couldn't find the global config file.</samp>’ or 131‘<samp>WARNING: Couldn't find tool init file</samp>’ that can be ignored. 132</p> 133<p>If you are testing a cross-compiler, you may want to run the testsuite 134on a simulator as described at <a href="http://gcc.gnu.org/simtest-howto.html">http://gcc.gnu.org/simtest-howto.html</a>. 135</p> 136<div class="section" id="How-can-you-run-the-testsuite-on-selected-tests_003f"> 137<h3 class="section">How can you run the testsuite on selected tests?</h3> 138 139<p>In order to run sets of tests selectively, there are targets 140‘<samp>make check-gcc</samp>’ and language specific ‘<samp>make check-c</samp>’, 141‘<samp>make check-c++</samp>’, ‘<samp>make check-d</samp>’ ‘<samp>make check-fortran</samp>’, 142‘<samp>make check-ada</samp>’, ‘<samp>make check-objc</samp>’, ‘<samp>make check-obj-c++</samp>’, 143‘<samp>make check-lto</samp>’ 144in the <samp>gcc</samp> subdirectory of the object directory. You can also 145just run ‘<samp>make check</samp>’ in a subdirectory of the object directory. 146</p> 147 148<p>A more selective way to just run all <code>gcc</code> execute tests in the 149testsuite is to use 150</p> 151<div class="example"> 152<pre class="example">make check-gcc RUNTESTFLAGS="execute.exp <var>other-options</var>" 153</pre></div> 154 155<p>Likewise, in order to run only the <code>g++</code> “old-deja” tests in 156the testsuite with filenames matching ‘<samp>9805*</samp>’, you would use 157</p> 158<div class="example"> 159<pre class="example">make check-g++ RUNTESTFLAGS="old-deja.exp=9805* <var>other-options</var>" 160</pre></div> 161 162<p>The file-matching expression following <var>filename</var><code>.exp=</code> is treated 163as a series of whitespace-delimited glob expressions so that multiple patterns 164may be passed, although any whitespace must either be escaped or surrounded by 165single quotes if multiple expressions are desired. For example, 166</p> 167<div class="example"> 168<pre class="example">make check-g++ RUNTESTFLAGS="old-deja.exp=9805*\ virtual2.c <var>other-options</var>" 169make check-g++ RUNTESTFLAGS="'old-deja.exp=9805* virtual2.c' <var>other-options</var>" 170</pre></div> 171 172<p>The <samp>*.exp</samp> files are located in the testsuite directories of the GCC 173source, the most important ones being <samp>compile.exp</samp>, 174<samp>execute.exp</samp>, <samp>dg.exp</samp> and <samp>old-deja.exp</samp>. 175To get a list of the possible <samp>*.exp</samp> files, pipe the 176output of ‘<samp>make check</samp>’ into a file and look at the 177‘<samp>Running … .exp</samp>’ lines. 178</p> 179</div> 180<div class="section" id="Passing-options-and-running-multiple-testsuites"> 181<h3 class="section">Passing options and running multiple testsuites</h3> 182 183<p>You can pass multiple options to the testsuite using the 184‘<samp>--target_board</samp>’ option of DejaGNU, either passed as part of 185‘<samp>RUNTESTFLAGS</samp>’, or directly to <code>runtest</code> if you prefer to 186work outside the makefiles. For example, 187</p> 188<div class="example"> 189<pre class="example">make check-g++ RUNTESTFLAGS="--target_board=unix/-O3/-fmerge-constants" 190</pre></div> 191 192<p>will run the standard <code>g++</code> testsuites (“unix” is the target name 193for a standard native testsuite situation), passing 194‘<samp>-O3 -fmerge-constants</samp>’ to the compiler on every test, i.e., 195slashes separate options. 196</p> 197<p>You can run the testsuites multiple times using combinations of options 198with a syntax similar to the brace expansion of popular shells: 199</p> 200<div class="example"> 201<pre class="example">…"--target_board=arm-sim\{-mhard-float,-msoft-float\}\{-O1,-O2,-O3,\}" 202</pre></div> 203 204<p>(Note the empty option caused by the trailing comma in the final group.) 205The following will run each testsuite eight times using the ‘<samp>arm-sim</samp>’ 206target, as if you had specified all possible combinations yourself: 207</p> 208<div class="example"> 209<pre class="example">--target_board='arm-sim/-mhard-float/-O1 \ 210 arm-sim/-mhard-float/-O2 \ 211 arm-sim/-mhard-float/-O3 \ 212 arm-sim/-mhard-float \ 213 arm-sim/-msoft-float/-O1 \ 214 arm-sim/-msoft-float/-O2 \ 215 arm-sim/-msoft-float/-O3 \ 216 arm-sim/-msoft-float' 217</pre></div> 218 219<p>They can be combined as many times as you wish, in arbitrary ways. This 220list: 221</p> 222<div class="example"> 223<pre class="example">…"--target_board=unix/-Wextra\{-O3,-fno-strength\}\{-fomit-frame,\}" 224</pre></div> 225 226<p>will generate four combinations, all involving ‘<samp>-Wextra</samp>’. 227</p> 228<p>The disadvantage to this method is that the testsuites are run in serial, 229which is a waste on multiprocessor systems. For users with GNU Make and 230a shell which performs brace expansion, you can run the testsuites in 231parallel by having the shell perform the combinations and <code>make</code> 232do the parallel runs. Instead of using ‘<samp>--target_board</samp>’, use a 233special makefile target: 234</p> 235<div class="example"> 236<pre class="example">make -j<var>N</var> check-<var>testsuite</var>//<var>test-target</var>/<var>option1</var>/<var>option2</var>/… 237</pre></div> 238 239<p>For example, 240</p> 241<div class="example"> 242<pre class="example">make -j3 check-gcc//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu} 243</pre></div> 244 245<p>will run three concurrent “make-gcc” testsuites, eventually testing all 246ten combinations as described above. Note that this is currently only 247supported in the <samp>gcc</samp> subdirectory. (To see how this works, try 248typing <code>echo</code> before the example given here.) 249</p> 250 251</div> 252<div class="section" id="How-to-interpret-test-results"> 253<h3 class="section">How to interpret test results</h3> 254 255<p>The result of running the testsuite are various <samp>*.sum</samp> and <samp>*.log</samp> 256files in the testsuite subdirectories. The <samp>*.log</samp> files contain a 257detailed log of the compiler invocations and the corresponding 258results, the <samp>*.sum</samp> files summarize the results. These summaries 259contain status codes for all tests: 260</p> 261<ul> 262<li> PASS: the test passed as expected 263</li><li> XPASS: the test unexpectedly passed 264</li><li> FAIL: the test unexpectedly failed 265</li><li> XFAIL: the test failed as expected 266</li><li> UNSUPPORTED: the test is not supported on this platform 267</li><li> ERROR: the testsuite detected an error 268</li><li> WARNING: the testsuite detected a possible problem 269</li></ul> 270 271<p>It is normal for some tests to report unexpected failures. At the 272current time the testing harness does not allow fine grained control 273over whether or not a test is expected to fail. This problem should 274be fixed in future releases. 275</p> 276 277</div> 278<div class="section" id="Submitting-test-results"> 279<h3 class="section">Submitting test results</h3> 280 281<p>If you want to report the results to the GCC project, use the 282<samp>contrib/test_summary</samp> shell script. Start it in the <var>objdir</var> with 283</p> 284<div class="example"> 285<pre class="example"><var>srcdir</var>/contrib/test_summary -p your_commentary.txt \ 286 -m gcc-testresults@gcc.gnu.org |sh 287</pre></div> 288 289<p>This script uses the <code>Mail</code> program to send the results, so 290make sure it is in your <code>PATH</code>. The file <samp>your_commentary.txt</samp> is 291prepended to the testsuite summary and should contain any special 292remarks you have on your results or your build environment. Please 293do not edit the testsuite result block or the subject line, as these 294messages may be automatically processed. 295</p> 296<hr /> 297<p> 298<p><a href="./index.html">Return to the GCC Installation page</a> 299</p> 300 301 302 303 304 305 306<hr></div> 307 308 309 310</body> 311</html> 312