xref: /netbsd-src/external/gpl3/gcc/dist/INSTALL/test.html (revision 4fe0f936ff464bca8e6277bde90f477ef5a4d004)
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
2<html>
3<!-- Created by GNU Texinfo 6.8, https://www.gnu.org/software/texinfo/ -->
4<head>
5<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
6<!-- Copyright (C) 1988-2022 Free Software Foundation, Inc.
7
8Permission is granted to copy, distribute and/or modify this document
9under the terms of the GNU Free Documentation License, Version 1.3 or
10any later version published by the Free Software Foundation; with no
11Invariant Sections, the Front-Cover texts being (a) (see below), and
12with the Back-Cover Texts being (b) (see below).  A copy of the
13license is included in the section entitled "GNU
14Free Documentation License".
15
16(a) The FSF's Front-Cover Text is:
17
18A GNU Manual
19
20(b) The FSF's Back-Cover Text is:
21
22You have freedom to copy and modify this GNU Manual, like GNU
23     software.  Copies published by the Free Software Foundation raise
24     funds for GNU development. -->
25<title>Installing GCC: Testing</title>
26
27<meta name="description" content="Installing GCC: Testing">
28<meta name="keywords" content="Installing GCC: Testing">
29<meta name="resource-type" content="document">
30<meta name="distribution" content="global">
31<meta name="Generator" content="makeinfo">
32<meta name="viewport" content="width=device-width,initial-scale=1">
33
34<style type="text/css">
35<!--
36a.copiable-anchor {visibility: hidden; text-decoration: none; line-height: 0em}
37a.summary-letter {text-decoration: none}
38blockquote.indentedblock {margin-right: 0em}
39div.display {margin-left: 3.2em}
40div.example {margin-left: 3.2em}
41kbd {font-style: oblique}
42pre.display {font-family: inherit}
43pre.format {font-family: inherit}
44pre.menu-comment {font-family: serif}
45pre.menu-preformatted {font-family: serif}
46span.nolinebreak {white-space: nowrap}
47span.roman {font-family: initial; font-weight: normal}
48span.sansserif {font-family: sans-serif; font-weight: normal}
49span:hover a.copiable-anchor {visibility: visible}
50ul.no-bullet {list-style: none}
51-->
52</style>
53
54
55</head>
56
57<body lang="en">
58<h1 class="settitle" align="center">Installing GCC: Testing</h1>
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80<span id="index-Testing"></span>
81<span id="index-Installing-GCC_003a-Testing"></span>
82<span id="index-Testsuite"></span>
83
84<p>Before you install GCC, we encourage you to run the testsuites and to
85compare your results with results from a similar configuration that have
86been submitted to the
87<a href="https://gcc.gnu.org/ml/gcc-testresults/">gcc-testresults mailing list</a>.
88Some of these archived results are linked from the build status lists
89at <a href="https://gcc.gnu.org/buildstat.html">https://gcc.gnu.org/buildstat.html</a>, although not everyone who
90reports a successful build runs the testsuites and submits the results.
91This step is optional and may require you to download additional software,
92but it can give you confidence in your new GCC installation or point out
93problems before you install and start using your new GCC.
94</p>
95<p>First, you must have <a href="download.html">downloaded the testsuites</a>.
96These are part of the full distribution, but if you downloaded the
97&ldquo;core&rdquo; compiler plus any front ends, you must download the testsuites
98separately.
99</p>
100<p>Second, you must have the testing tools installed.  This includes
101<a href="https://www.gnu.org/software/dejagnu/">DejaGnu</a>, Tcl, and Expect;
102the DejaGnu site has links to these.
103Some optional tests also require Python3 and pytest module.
104</p>
105<p>If the directories where <code>runtest</code> and <code>expect</code> were
106installed are not in the <code>PATH</code>, you may need to set the following
107environment variables appropriately, as in the following example (which
108assumes that DejaGnu has been installed under <samp>/usr/local</samp>):
109</p>
110<div class="example">
111<pre class="example">TCL_LIBRARY = /usr/local/share/tcl8.0
112DEJAGNULIBS = /usr/local/share/dejagnu
113</pre></div>
114
115<p>(On systems such as Cygwin, these paths are required to be actual
116paths, not mounts or links; presumably this is due to some lack of
117portability in the DejaGnu code.)
118</p>
119
120<p>Finally, you can run the testsuite (which may take a long time):
121</p><div class="example">
122<pre class="example">cd <var>objdir</var>; make -k check
123</pre></div>
124
125<p>This will test various components of GCC, such as compiler
126front ends and runtime libraries.  While running the testsuite, DejaGnu
127might emit some harmless messages resembling
128&lsquo;<samp>WARNING: Couldn't find the global config file.</samp>&rsquo; or
129&lsquo;<samp>WARNING: Couldn't find tool init file</samp>&rsquo; that can be ignored.
130</p>
131<p>If you are testing a cross-compiler, you may want to run the testsuite
132on a simulator as described at <a href="https://gcc.gnu.org/simtest-howto.html">https://gcc.gnu.org/simtest-howto.html</a>.
133</p>
134<div class="section" id="How-can-you-run-the-testsuite-on-selected-tests_003f">
135<h3 class="section">How can you run the testsuite on selected tests?</h3>
136
137<p>In order to run sets of tests selectively, there are targets
138&lsquo;<samp>make check-gcc</samp>&rsquo; and language specific &lsquo;<samp>make check-c</samp>&rsquo;,
139&lsquo;<samp>make check-c++</samp>&rsquo;, &lsquo;<samp>make check-d</samp>&rsquo; &lsquo;<samp>make check-fortran</samp>&rsquo;,
140&lsquo;<samp>make check-ada</samp>&rsquo;, &lsquo;<samp>make check-objc</samp>&rsquo;, &lsquo;<samp>make check-obj-c++</samp>&rsquo;,
141&lsquo;<samp>make check-lto</samp>&rsquo;
142in the <samp>gcc</samp> subdirectory of the object directory.  You can also
143just run &lsquo;<samp>make check</samp>&rsquo; in a subdirectory of the object directory.
144</p>
145
146<p>A more selective way to just run all <code>gcc</code> execute tests in the
147testsuite is to use
148</p>
149<div class="example">
150<pre class="example">make check-gcc RUNTESTFLAGS=&quot;execute.exp <var>other-options</var>&quot;
151</pre></div>
152
153<p>Likewise, in order to run only the <code>g++</code> &ldquo;old-deja&rdquo; tests in
154the testsuite with filenames matching &lsquo;<samp>9805*</samp>&rsquo;, you would use
155</p>
156<div class="example">
157<pre class="example">make check-g++ RUNTESTFLAGS=&quot;old-deja.exp=9805* <var>other-options</var>&quot;
158</pre></div>
159
160<p>The file-matching expression following <var>filename</var><code>.exp=</code> is treated
161as a series of whitespace-delimited glob expressions so that multiple patterns
162may be passed, although any whitespace must either be escaped or surrounded by
163single quotes if multiple expressions are desired. For example,
164</p>
165<div class="example">
166<pre class="example">make check-g++ RUNTESTFLAGS=&quot;old-deja.exp=9805*\ virtual2.c <var>other-options</var>&quot;
167make check-g++ RUNTESTFLAGS=&quot;'old-deja.exp=9805* virtual2.c' <var>other-options</var>&quot;
168</pre></div>
169
170<p>The <samp>*.exp</samp> files are located in the testsuite directories of the GCC
171source, the most important ones being <samp>compile.exp</samp>,
172<samp>execute.exp</samp>, <samp>dg.exp</samp> and <samp>old-deja.exp</samp>.
173To get a list of the possible <samp>*.exp</samp> files, pipe the
174output of &lsquo;<samp>make check</samp>&rsquo; into a file and look at the
175&lsquo;<samp>Running &hellip;  .exp</samp>&rsquo; lines.
176</p>
177</div>
178<div class="section" id="Passing-options-and-running-multiple-testsuites">
179<h3 class="section">Passing options and running multiple testsuites</h3>
180
181<p>You can pass multiple options to the testsuite using the
182&lsquo;<samp>--target_board</samp>&rsquo; option of DejaGNU, either passed as part of
183&lsquo;<samp>RUNTESTFLAGS</samp>&rsquo;, or directly to <code>runtest</code> if you prefer to
184work outside the makefiles.  For example,
185</p>
186<div class="example">
187<pre class="example">make check-g++ RUNTESTFLAGS=&quot;--target_board=unix/-O3/-fmerge-constants&quot;
188</pre></div>
189
190<p>will run the standard <code>g++</code> testsuites (&ldquo;unix&rdquo; is the target name
191for a standard native testsuite situation), passing
192&lsquo;<samp>-O3 -fmerge-constants</samp>&rsquo; to the compiler on every test, i.e.,
193slashes separate options.
194</p>
195<p>You can run the testsuites multiple times using combinations of options
196with a syntax similar to the brace expansion of popular shells:
197</p>
198<div class="example">
199<pre class="example">&hellip;&quot;--target_board=arm-sim\{-mhard-float,-msoft-float\}\{-O1,-O2,-O3,\}&quot;
200</pre></div>
201
202<p>(Note the empty option caused by the trailing comma in the final group.)
203The following will run each testsuite eight times using the &lsquo;<samp>arm-sim</samp>&rsquo;
204target, as if you had specified all possible combinations yourself:
205</p>
206<div class="example">
207<pre class="example">--target_board='arm-sim/-mhard-float/-O1 \
208                arm-sim/-mhard-float/-O2 \
209                arm-sim/-mhard-float/-O3 \
210                arm-sim/-mhard-float \
211                arm-sim/-msoft-float/-O1 \
212                arm-sim/-msoft-float/-O2 \
213                arm-sim/-msoft-float/-O3 \
214                arm-sim/-msoft-float'
215</pre></div>
216
217<p>They can be combined as many times as you wish, in arbitrary ways.  This
218list:
219</p>
220<div class="example">
221<pre class="example">&hellip;&quot;--target_board=unix/-Wextra\{-O3,-fno-strength\}\{-fomit-frame,\}&quot;
222</pre></div>
223
224<p>will generate four combinations, all involving &lsquo;<samp>-Wextra</samp>&rsquo;.
225</p>
226<p>The disadvantage to this method is that the testsuites are run in serial,
227which is a waste on multiprocessor systems.  For users with GNU Make and
228a shell which performs brace expansion, you can run the testsuites in
229parallel by having the shell perform the combinations and <code>make</code>
230do the parallel runs.  Instead of using &lsquo;<samp>--target_board</samp>&rsquo;, use a
231special makefile target:
232</p>
233<div class="example">
234<pre class="example">make -j<var>N</var> check-<var>testsuite</var>//<var>test-target</var>/<var>option1</var>/<var>option2</var>/&hellip;
235</pre></div>
236
237<p>For example,
238</p>
239<div class="example">
240<pre class="example">make -j3 check-gcc//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu}
241</pre></div>
242
243<p>will run three concurrent &ldquo;make-gcc&rdquo; testsuites, eventually testing all
244ten combinations as described above.  Note that this is currently only
245supported in the <samp>gcc</samp> subdirectory.  (To see how this works, try
246typing <code>echo</code> before the example given here.)
247</p>
248
249</div>
250<div class="section" id="How-to-interpret-test-results">
251<h3 class="section">How to interpret test results</h3>
252
253<p>The result of running the testsuite are various <samp>*.sum</samp> and <samp>*.log</samp>
254files in the testsuite subdirectories.  The <samp>*.log</samp> files contain a
255detailed log of the compiler invocations and the corresponding
256results, the <samp>*.sum</samp> files summarize the results.  These summaries
257contain status codes for all tests:
258</p>
259<ul>
260<li> PASS: the test passed as expected
261</li><li> XPASS: the test unexpectedly passed
262</li><li> FAIL: the test unexpectedly failed
263</li><li> XFAIL: the test failed as expected
264</li><li> UNSUPPORTED: the test is not supported on this platform
265</li><li> ERROR: the testsuite detected an error
266</li><li> WARNING: the testsuite detected a possible problem
267</li></ul>
268
269<p>It is normal for some tests to report unexpected failures.  At the
270current time the testing harness does not allow fine grained control
271over whether or not a test is expected to fail.  This problem should
272be fixed in future releases.
273</p>
274
275</div>
276<div class="section" id="Submitting-test-results">
277<h3 class="section">Submitting test results</h3>
278
279<p>If you want to report the results to the GCC project, use the
280<samp>contrib/test_summary</samp> shell script.  Start it in the <var>objdir</var> with
281</p>
282<div class="example">
283<pre class="example"><var>srcdir</var>/contrib/test_summary -p your_commentary.txt \
284    -m gcc-testresults@gcc.gnu.org |sh
285</pre></div>
286
287<p>This script uses the <code>Mail</code> program to send the results, so
288make sure it is in your <code>PATH</code>.  The file <samp>your_commentary.txt</samp> is
289prepended to the testsuite summary and should contain any special
290remarks you have on your results or your build environment.  Please
291do not edit the testsuite result block or the subject line, as these
292messages may be automatically processed.
293</p>
294<hr />
295<p>
296<p><a href="./index.html">Return to the GCC Installation page</a>
297</p>
298
299
300
301
302
303<hr></div>
304
305
306
307</body>
308</html>
309