1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Testing</title><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><meta name="keywords" content="ISO C++, test, testsuite, performance, conformance, ABI, exception safety" /><meta name="keywords" content="ISO C++, library" /><meta name="keywords" content="ISO C++, runtime, library" /><link rel="home" href="../index.html" title="The GNU C++ Library" /><link rel="up" href="appendix_porting.html" title="Appendix B. Porting and Maintenance" /><link rel="prev" href="internals.html" title="Porting to New Hardware or Operating Systems" /><link rel="next" href="abi.html" title="ABI Policy and Guidelines" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Testing</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="internals.html">Prev</a> </td><th width="60%" align="center">Appendix B. 3 Porting and Maintenance 4 5</th><td width="20%" align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="manual.intro.setup.test"></a>Testing</h2></div></div></div><p> 6The libstdc++ testsuite includes testing for standard conformance, 7regressions, ABI, and performance. 8</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.organization"></a>Test Organization</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.layout"></a>Directory Layout</h4></div></div></div><p> 9 The directory 10 <code class="filename"><em class="replaceable"><code>gccsrcdir</code></em>/libstdc++-v3/testsuite</code> 11 contains the individual test cases organized in sub-directories 12 corresponding to clauses of the C++ standard (detailed below), 13 the DejaGnu test harness support files, and sources to various 14 testsuite utilities that are packaged in a separate testing library. 15</p><p> 16 All test cases for functionality required by the runtime components 17 of the C++ standard (ISO 14882) are files within the following 18 directories: 19 20 </p><pre class="programlisting"> 21 17_intro 22 18_support 23 19_diagnostics 24 20_util 25 21_strings 26 22_locale 27 23_containers 28 24_iterators 29 25_algorithms 30 26_numerics 31 27_io 32 28_regex 33 29_atomics 34 30_threads 35 </pre><p> 36</p><p> 37 In addition, the following directories include test files: 38 39</p><div class="variablelist"><dl class="variablelist compact"><dt><span class="term"><code class="filename">tr1</code></span></dt><dd>Tests for components as described by the Technical Report 40 on Standard Library Extensions (TR1). 41 </dd><dt><span class="term"><code class="filename">backward</code></span></dt><dd>Tests for backwards compatibility and deprecated features. 42 </dd><dt><span class="term"><code class="filename">demangle</code></span></dt><dd>Tests for <code class="function">__cxa_demangle</code>, the IA-64 C++ ABI 43 demangler. 44 </dd><dt><span class="term"><code class="filename">ext</code></span></dt><dd>Tests for extensions.</dd><dt><span class="term"><code class="filename">performance</code></span></dt><dd>Tests for performance analysis, and performance regressions. 45 </dd></dl></div><p> 46 </p><p> 47 Some directories don't have test files, but instead contain 48 auxiliary information: 49 50</p><div class="variablelist"><dl class="variablelist compact"><dt><span class="term"><code class="filename">config</code></span></dt><dd>Files for the DejaGnu test harness.</dd><dt><span class="term"><code class="filename">lib</code></span></dt><dd>Files for the DejaGnu test harness.</dd><dt><span class="term"><code class="filename">libstdc++*</code></span></dt><dd>Files for the DejaGnu test harness.</dd><dt><span class="term"><code class="filename">data</code></span></dt><dd>Sample text files for testing input and output.</dd><dt><span class="term"><code class="filename">util</code></span></dt><dd>Files for libtestc++, utilities and testing routines.</dd></dl></div><p> 51 </p><p> 52 Within a directory that includes test files, there may be 53 additional subdirectories, or files. Originally, test cases 54 were appended to one file that represented a particular section 55 of the chapter under test, and was named accordingly. For 56 instance, to test items related to <code class="code"> 21.3.6.1 - 57 <code class="function">basic_string::find</code> [lib.string::find]</code> 58 in the standard, the following was used: 59</p><pre class="programlisting"> 21_strings/find.cc </pre><p> 60 However, that practice soon became a liability as the test cases 61 became huge and unwieldy, and testing new or extended 62 functionality (like wide characters or named locales) became 63 frustrating, leading to aggressive pruning of test cases on some 64 platforms that covered up implementation errors. Now, the test 65 suite has a policy of one file, one test case, which solves the 66 above issues and gives finer grained results and more manageable 67 error debugging. As an example, the test case quoted above 68 becomes: 69</p><pre class="programlisting"> 21_strings/basic_string/find/char/1.cc 70 21_strings/basic_string/find/char/2.cc 71 21_strings/basic_string/find/char/3.cc 72 21_strings/basic_string/find/wchar_t/1.cc 73 21_strings/basic_string/find/wchar_t/2.cc 74 21_strings/basic_string/find/wchar_t/3.cc</pre><p> 75 </p><p> 76 All new tests should be written with the policy of "one test 77 case, one file" in mind. 78 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.naming"></a>Naming Conventions</h4></div></div></div><p> 79 In addition, there are some special names and suffixes that are 80 used within the testsuite to designate particular kinds of 81 tests. 82 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term"><code class="filename">_xin.cc</code></span></dt><dd> 83 This test case expects some kind of interactive input in order 84 to finish or pass. At the moment, the interactive tests are not 85 run by default. Instead, they are run by hand, like: 86 <pre class="programlisting"> 87g++ 27_io/objects/char/3_xin.cc 88cat 27_io/objects/char/3_xin.in | a.out</pre></dd><dt><span class="term"><code class="filename">.in</code></span></dt><dd> 89 This file contains the expected input for the corresponding <span class="emphasis"><em> 90 _xin.cc</em></span> test case. 91 </dd><dt><span class="term"><code class="filename">_neg.cc</code></span></dt><dd> 92 This test case is expected to fail: it's a negative test. At the 93 moment, these are almost always compile time errors. 94 </dd><dt><span class="term"><code class="filename">char</code></span></dt><dd> 95 This can either be a directory name or part of a longer file 96 name, and indicates that this file, or the files within this 97 directory are testing the <code class="code">char</code> instantiation of a 98 template. 99 </dd><dt><span class="term"><code class="filename">wchar_t</code></span></dt><dd> 100 This can either be a directory name or part of a longer file 101 name, and indicates that this file, or the files within this 102 directory are testing the <code class="code">wchar_t</code> instantiation of 103 a template. Some hosts do not support <code class="code">wchar_t</code> 104 functionality, so for these targets, all of these tests will not 105 be run. 106 </dd><dt><span class="term"><code class="filename">thread</code></span></dt><dd> 107 This can either be a directory name or part of a longer file 108 name, and indicates that this file, or the files within this 109 directory are testing situations where multiple threads are 110 being used. 111 </dd><dt><span class="term"><code class="filename">performance</code></span></dt><dd> 112 This can either be an enclosing directory name or part of a 113 specific file name. This indicates a test that is used to 114 analyze runtime performance, for performance regression testing, 115 or for other optimization related analysis. At the moment, these 116 test cases are not run by default. 117 </dd></dl></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.run"></a>Running the Testsuite</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.basic"></a>Basic</h4></div></div></div><p> 118 You can check the status of the build without installing it 119 using the DejaGnu harness, much like the rest of the gcc 120 tools, i.e. 121 <strong class="userinput"><code>make check</code></strong> 122 in the 123 <code class="filename"><em class="replaceable"><code>libbuilddir</code></em></code> 124 directory, or 125 <strong class="userinput"><code>make check-target-libstdc++-v3</code></strong> 126 in the 127 <code class="filename"><em class="replaceable"><code>gccbuilddir</code></em></code> 128 directory. 129 </p><p> 130 These commands are functionally equivalent and will create a 131 '<code class="filename">testsuite</code>' directory underneath 132 <code class="filename"><em class="replaceable"><code>libbuilddir</code></em></code> 133 containing the results of the 134 tests. Two results files will be generated: 135 <code class="filename">libstdc++.sum</code>, which is a PASS/FAIL summary 136 for each test, and 137 <code class="filename">libstdc++.log</code> which is a log of 138 the exact command-line passed to the compiler, the compiler 139 output, and the executable output (if any) for each test. 140 </p><p> 141 Archives of test results for various versions and platforms are 142 available on the GCC website in the <a class="link" href="http://gcc.gnu.org/gcc-4.3/buildstat.html" target="_top">build 143 status</a> section of each individual release, and are also 144 archived on a daily basis on the <a class="link" href="http://gcc.gnu.org/ml/gcc-testresults/current" target="_top">gcc-testresults</a> 145 mailing list. Please check either of these places for a similar 146 combination of source version, operating system, and host CPU. 147 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.variations"></a>Variations</h4></div></div></div><p> 148 There are several options for running tests, including testing 149 the regression tests, testing a subset of the regression tests, 150 testing the performance tests, testing just compilation, testing 151 installed tools, etc. In addition, there is a special rule for 152 checking the exported symbols of the shared library. 153 </p><p> 154 To debug the DejaGnu test harness during runs, try invoking with a 155 specific argument to the variable <code class="varname">RUNTESTFLAGS</code>, 156 like so: 157</p><pre class="programlisting"> 158 make check-target-libstdc++-v3 RUNTESTFLAGS="-v" 159</pre><p> 160 or 161</p><pre class="programlisting"> 162 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" 163</pre><p> 164 </p><p> 165 To run a subset of the library tests, you can either generate the 166 <code class="filename">testsuite_files</code> file (described below) by running 167 <strong class="userinput"><code>make testsuite_files</code></strong> in the 168 <code class="filename"><em class="replaceable"><code>libbuilddir</code></em>/testsuite</code> 169 directory, then edit the 170 file to remove the tests you don't want and then run the testsuite as 171 normal, or you can specify a testsuite and a subset of tests in the 172 <code class="varname">RUNTESTFLAGS</code> variable. 173 </p><p> 174 For example, to run only the tests for containers you could use: 175 176</p><pre class="programlisting"> 177 make check-target-libstdc++-v3 RUNTESTFLAGS="conformance.exp=23_containers/*" 178</pre><p> 179 </p><p> 180 When combining this with other options in <code class="varname">RUNTESTFLAGS</code> 181 the <code class="option">testsuite.exp=testfiles</code> options must come first. 182 </p><p> 183 There are two ways to run on a simulator: set up <code class="envar">DEJAGNU</code> 184 to point to a specially crafted <code class="filename">site.exp</code>, 185 or pass down <code class="option">--target_board</code> flags. 186 </p><p> 187 Example flags to pass down for various embedded builds are as follows: 188 189</p><pre class="programlisting"> 190 --target=powerpc-eabisim <span class="emphasis"><em>(libgloss/sim)</em></span> 191 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" 192 193 --target=calmrisc32 <span class="emphasis"><em>(libgloss/sid)</em></span> 194 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" 195 196 --target=xscale-elf <span class="emphasis"><em>(newlib/sim)</em></span> 197 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" 198</pre><p> 199 </p><p> 200 Also, here is an example of how to run the libstdc++ testsuite 201 for a multilibed build directory with different ABI settings: 202 203 </p><pre class="programlisting"> 204 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"' 205</pre><p> 206 </p><p> 207 You can run the tests with a compiler and library that have 208 already been installed. Make sure that the compiler (e.g., 209 <span class="command"><strong>g++</strong></span>) is in your <code class="envar">PATH</code>. If you are 210 using shared libraries, then you must also ensure that the 211 directory containing the shared version of libstdc++ is in your 212 <code class="envar">LD_LIBRARY_PATH</code>, or 213 <a class="link" href="using_dynamic_or_shared.html#manual.intro.using.linkage.dynamic" title="Finding Dynamic or Shared Libraries">equivalent</a>. 214 If your GCC source tree is at 215 <code class="filename">/path/to/gcc</code>, 216 then you can run the tests as follows: 217 218</p><pre class="programlisting"> 219 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite 220</pre><p> 221 </p><p> 222 The testsuite will create a number of files in the directory in 223 which you run this command,. Some of those files might use the 224 same name as files created by other testsuites (like the ones 225 for GCC and G++), so you should not try to run all the 226 testsuites in parallel from the same directory. 227 </p><p> 228 In addition, there are some testing options that are mostly of 229 interest to library maintainers and system integrators. As such, 230 these tests may not work on all CPU and host combinations, and 231 may need to be executed in the 232 <code class="filename"><em class="replaceable"><code>libbuilddir</code></em>/testsuite</code> 233 directory. These 234 options include, but are not necessarily limited to, the 235 following: 236 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term"><strong class="userinput"><code> 237 make testsuite_files 238 </code></strong></span></dt><dd><p> 239 Five files are generated that determine what test files 240 are run. These files are: 241 242 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term"> <code class="filename">testsuite_files</code> </span></dt><dd> 243 This is a list of all the test cases that will be run. Each 244 test case is on a separate line, given with an absolute path 245 from the 246 <code class="filename"><em class="replaceable"><code>libsrcdir</code></em>/testsuite</code> 247 directory. 248 </dd><dt><span class="term"> <code class="filename">testsuite_files_interactive</code> </span></dt><dd> 249 This is a list of all the interactive test cases, using the 250 same format as the file list above. These tests are not run 251 by default. 252 </dd><dt><span class="term"> <code class="filename">testsuite_files_performance</code> </span></dt><dd> 253 This is a list of all the performance test cases, using the 254 same format as the file list above. These tests are not run 255 by default. 256 </dd><dt><span class="term"> <code class="filename">testsuite_thread</code> </span></dt><dd> 257 This file indicates that the host system can run tests which 258 involved multiple threads. 259 </dd><dt><span class="term"> <code class="filename">testsuite_wchar_t</code> </span></dt><dd> 260 This file indicates that the host system can run the 261 <code class="code">wchar_t</code> tests, and corresponds to the macro 262 definition <code class="literal">_GLIBCXX_USE_WCHAR_T</code> in the 263 file <code class="filename">c++config.h</code>. 264 </dd></dl></div><p> 265 </p></dd><dt><span class="term"><strong class="userinput"><code> 266 make check-abi 267 </code></strong></span></dt><dd><p> 268 The library ABI can be tested. This involves testing the shared 269 library against a baseline list of symbol exports that defines the 270 previous version of the ABI. The tests require that no exported 271 symbols are removed, no new symbols are added to the old symbol 272 versions, and any new symbols have the latest symbol version. 273 See <a class="link" href="abi.html#abi.versioning" title="Versioning">Versioning</a> for more details 274 of the ABI version history. 275 </p></dd><dt><span class="term"><strong class="userinput"><code> 276 make new-abi-baseline 277 </code></strong></span></dt><dd><p> 278 Generate a new baseline set of symbols exported from the library 279 (written to a file under 280 <code class="filename"><em class="replaceable"><code>libsrcdir</code></em>/config/abi/post/<em class="replaceable"><code>target</code></em>/</code>). 281 A different baseline symbols file is needed for each architecture and 282 is used by the <code class="literal">check-abi</code> target described above. 283 The files are usually re-generated by target maintainers for releases. 284 </p></dd><dt><span class="term"><strong class="userinput"><code> 285 make check-compile 286 </code></strong></span></dt><dd><p> 287 This rule compiles, but does not link or execute, the 288 <code class="filename">testsuite_files</code> test cases and displays the 289 output on stdout. 290 </p></dd><dt><span class="term"><strong class="userinput"><code> 291 make check-performance 292 </code></strong></span></dt><dd><p> 293 This rule runs through the 294 <code class="filename">testsuite_files_performance</code> test cases and 295 collects information for performance analysis and can be used to 296 spot performance regressions. Various timing information is 297 collected, as well as number of hard page faults, and memory 298 used. This is not run by default, and the implementation is in 299 flux. 300 </p></dd><dt><span class="term"><strong class="userinput"><code> 301 make check-debug 302 </code></strong></span></dt><dd><p> 303 This rule runs through the test suite under the 304 <a class="link" href="debug_mode.html" title="Chapter 17. Debug Mode">debug mode</a>. 305 </p></dd><dt><span class="term"><strong class="userinput"><code> 306 make check-parallel 307 </code></strong></span></dt><dd><p> 308 This rule runs through the test suite under the 309 <a class="link" href="parallel_mode.html" title="Chapter 18. Parallel Mode">parallel mode</a>. 310 </p></dd></dl></div><p> 311 We are interested in any strange failures of the testsuite; 312 please email the main libstdc++ mailing list if you see 313 something odd or have questions. 314 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.permutations"></a>Permutations</h4></div></div></div><p> 315 The tests will be compiled with a set of default compiler flags defined 316 by the 317 <code class="filename"><em class="replaceable"><code>libbuilddir</code></em>/scripts/testsuite_flags</code> 318 file, as well as options specified in individual tests. You can run 319 the tests with different options by adding them to the output of 320 the <code class="option">--cxxflags</code> option of that script, or by setting 321 the <code class="varname">CXXFLAGS</code> variable when running 322 <span class="command"><strong>make</strong></span>, or via options for the DejaGnu test framework 323 (described below). The latter approach uses the 324 <code class="option">--target_board</code> option that was shown earlier, 325 but requires DejaGnu version 1.5.3 or newer to work reliably, so that the 326 <code class="literal">dg-options</code> in the test aren't overridden. 327 For example, to run the tests with 328 <code class="option">-O1 -D_GLIBCXX_ASSERTIONS</code> 329 you could use: 330</p><pre class="programlisting"> make check RUNTESTFLAGS=--target_board=unix/-O1/-D_GLIBCXX_ASSERTIONS</pre><p> 331 </p><p> 332 The <code class="option">--target_board</code> option can also be used to run the 333 tests multiple times in different variations. For example, to run the 334 entire testsuite three times using <code class="option">-O3</code> but with 335 different <code class="option">-std</code> options: 336</p><pre class="programlisting"> make check 'RUNTESTFLAGS=--target_board=unix/-O3\"{-std=gnu++98,-std=gnu++11,-std=gnu++14}\"'</pre><p> 337 N.B. that set of variations could also be written as 338 <code class="literal">unix/-O3\"{-std=gnu++98,-std=gnu++11,}\"</code> so that 339 the third variation would use the default for <code class="option">-std</code> 340 (which is <code class="option">-std=gnu++14</code> as of GCC 6). 341 </p><p> 342 To run the libstdc++ test suite under the 343 <a class="link" href="debug_mode.html" title="Chapter 17. Debug Mode">debug mode</a>, use 344 <strong class="userinput"><code>make check-debug</code></strong>. Alternatively, edit 345 <code class="filename"><em class="replaceable"><code>libbuilddir</code></em>/scripts/testsuite_flags</code> 346 to add the compile-time flag <code class="option">-D_GLIBCXX_DEBUG</code> to the 347 result printed by the <code class="option">--cxxflags</code> 348 option. Additionally, add the 349 <code class="option">-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on 350 pedantic checking. The libstdc++ test suite should produce 351 the same results under debug mode that it does under release mode: 352 any deviation indicates an error in either the library or the test suite. 353 Note, however, that the number of tests that PASS may change, because 354 some test cases are skipped in normal mode, and some are skipped in 355 debug mode, as determined by the 356 <code class="literal">dg-require-<em class="replaceable"><code>support</code></em></code> 357 directives described below. 358 </p><p> 359 The <a class="link" href="parallel_mode.html" title="Chapter 18. Parallel Mode">parallel 360 mode</a> can be tested using 361 <strong class="userinput"><code>make check-parallel</code></strong>, or in much the same manner 362 as the debug mode, substituting 363 <code class="option">-D_GLIBCXX_PARALLEL</code> for 364 <code class="option">-D_GLIBCXX_DEBUG</code> in the previous paragraph. 365 </p><p> 366 Or, just run the testsuite 367 <code class="option">-D_GLIBCXX_DEBUG</code> or <code class="option">-D_GLIBCXX_PARALLEL</code> 368 in <code class="varname">CXXFLAGS</code> or <code class="varname">RUNTESTFLAGS</code>. 369 </p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.new_tests"></a>Writing a new test case</h3></div></div></div><p> 370 The first step in making a new test case is to choose the correct 371 directory and file name, given the organization as previously 372 described. 373 </p><p> 374 All files are copyright the FSF, and GPL'd: this is very 375 important. The first copyright year should correspond to the date 376 the file was checked in to version control. If a test is copied from 377 an existing file it should retain the copyright years from the 378 original file. 379 </p><p> 380 The DejaGnu instructions say to always return <code class="literal">0</code> 381 from <code class="function">main</code> to indicate success. Strictly speaking 382 this is redundant in C++, since returning from <code class="function">main</code> 383 is defined to return <code class="literal">0</code>. Most tests still have an 384 explicit return. 385 </p><p> 386 A bunch of utility functions and classes have already been 387 abstracted out into the testsuite utility library, <code class="code"> 388 libtestc++</code>. To use this functionality, just include the 389 appropriate header file: the library or specific object files will 390 automatically be linked in as part of the testsuite run. 391 </p><p> 392 Tests that need to perform runtime checks should use the 393 <code class="literal">VERIFY</code> macro, defined in the 394 <code class="filename"><testsuite_hooks.h></code> header. 395 This expands to a custom assertion using 396 <code class="function">__builtin_printf</code> and 397 <code class="function">__builtin_abort</code> 398 (to avoid using <code class="literal">assert</code> and being affected by 399 <code class="literal">NDEBUG</code>). 400 </p><p> 401 Prior to GCC 7.1, <code class="literal">VERIFY</code> was defined differently. 402 It usually expanded to the standard <code class="literal">assert</code> macro, but 403 allowed targets to define it to something different. In order to support 404 the alternative expansions of <code class="literal">VERIFY</code>, before any use 405 of the macro there needed to be a variable called <code class="varname">test</code> 406 in scope, which was usually defined like so (the attribute avoids 407 warnings about an unused variable): 408 </p><pre class="programlisting"> 409 bool test __attribute__((unused)) = true; 410 </pre><p> 411 This is no longer needed, and should not be added to new tests. 412 </p><p> 413 The testsuite uses the DejaGnu framework to compile and run the tests. 414 Test cases are normal C++ files which contain special directives in 415 comments. These directives look like <code class="literal">{ dg-* ... }</code> 416 and tell DejaGnu what to do and what kinds of behavior are to be expected 417 for a test. The core DejaGnu directives are documented in the 418 <code class="filename">dg.exp</code> file installed by DejaGnu. 419 The GCC testsuites support additional directives 420 as described in the GCC internals documentation, see <a class="link" href="https://gcc.gnu.org/onlinedocs/gccint/Directives.html" target="_top">Syntax 421 and Descriptions of test directives</a>. GCC also defines many <a class="link" href="https://gcc.gnu.org/onlinedocs/gccint/Effective-Target-Keywords.html" target="_top"> 422 Keywords describing target attributes</a> (a.k.a effective targets) 423 which can be used where a target <em class="replaceable"><code>selector</code></em> can 424 appear. 425 </p><p> 426 Some directives commonly used in the libstdc++ testsuite are: 427 428</p><div class="variablelist"><dl class="variablelist"><dt><span class="term"><code class="literal">{ dg-do <em class="replaceable"><code>do-what-keyword</code></em> [{ target/xfail <em class="replaceable"><code>selector</code></em> }] }</code></span></dt><dd>Where <em class="replaceable"><code>do-what-keyword</code></em> is usually 429 one of <code class="literal">run</code> (which is the default), 430 <code class="literal">compile</code>, or <code class="literal">link</code>, 431 and typical selectors are targets such as <code class="literal">*-*-gnu*</code> 432 or an effective target such as <code class="literal">c++11</code>. 433 </dd><dt><span class="term"><code class="literal">{ dg-require-<em class="replaceable"><code>support</code></em> args }</code></span></dt><dd>Skip the test if the target does not provide the required support. 434 See below for values of <em class="replaceable"><code>support</code></em>. 435 </dd><dt><span class="term"><code class="literal">{ dg-options <em class="replaceable"><code>options</code></em> [{ target <em class="replaceable"><code>selector</code></em> }] }</code></span></dt><dd></dd><dt><span class="term"><code class="literal">{ dg-error <em class="replaceable"><code>regexp</code></em> [ <em class="replaceable"><code>comment</code></em> [{ target/xfail <em class="replaceable"><code>selector</code></em> } [<em class="replaceable"><code>line</code></em>] ]] }</code></span></dt><dd></dd><dt><span class="term"><code class="literal">{ dg-excess-errors <em class="replaceable"><code>comment</code></em> [{ target/xfail <em class="replaceable"><code>selector</code></em> }] }</code></span></dt><dd></dd></dl></div><p> 436 For full details of these and other directives see the main GCC DejaGnu 437 documentation in the internals manual. 438 </p><p> 439 Test cases that use features of a particular C++ standard should specify 440 the minimum required standard as an effective target: 441</p><pre class="programlisting"> // { dg-do run { target c++11 } }</pre><p> 442 or 443</p><pre class="programlisting"> // { dg-require-effective-target c++11 }</pre><p> 444 Specifying the minimum required standard for a test allows it to be run 445 using later standards, so that we can verify that C++11 components still 446 work correctly when compiled as C++14 or later. Specifying a minimum also 447 means the test will be skipped if the test is compiled using 448 an older standard, e.g. using 449 <code class="option">RUNTESTFLAGS=--target_board=unix/-std=gnu++98</code>. 450 </p><p> 451 It is possible to indicate that a test should <span class="emphasis"><em>only</em></span> 452 be run for a specific standard (and not later standards) using an 453 effective target like <code class="literal">c++11_only</code>. However, this means 454 the test will be skipped by default (because the default mode is 455 <code class="literal">gnu++14</code>), and so will only run when 456 <code class="option">-std=gnu++11</code> or <code class="option">-std=c++11</code> is used 457 explicitly. For tests that require a specific standard it is better to 458 use a <code class="literal">dg-options</code> directive: 459</p><pre class="programlisting"> // { dg-options "-std=gnu++11" }</pre><p> 460 This means the test will not get skipped by default, and will always use 461 the specific standard dialect that the test requires. This isn't needed 462 often, and most tests should use an effective target to specify a 463 minimum standard instead, to allow them to be tested for all 464 possible variations. 465 </p><p> 466 Similarly, tests which depend on a newer standard than the default 467 must use <code class="literal">dg-options</code> instead of (or in addition to) 468 an effective target, so that they are not skipped by default. 469 For example, tests for C++17 features should use 470</p><pre class="programlisting"> // { dg-options "-std=gnu++17" }</pre><p> 471 before any <code class="literal">dg-do</code> such as: 472</p><pre class="programlisting"> // { dg-do run "c++17" }</pre><p> 473 The <code class="literal">dg-options</code> directive must come first, so that 474 the <code class="literal">-std</code> flag has already been added to the options 475 before checking the <code class="literal">c++17</code> target. 476 </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="tests.dg.examples"></a>Examples of Test Directives</h4></div></div></div><p> 477Example 1: Testing compilation only: 478</p><pre class="programlisting"> 479// { dg-do compile } 480</pre><p> 481 482Example 2: Testing for expected warnings on line 36, which all targets fail: 483</p><pre class="programlisting"> 484// { dg-warning "string literals" "" { xfail *-*-* } 36 } 485</pre><p> 486 487Example 3: Testing for expected warnings on line 36: 488</p><pre class="programlisting"> 489// { dg-warning "string literals" "" { target *-*-* } 36 } 490</pre><p> 491 492Example 4: Testing for compilation errors on line 41: 493</p><pre class="programlisting"> 494// { dg-do compile } 495// { dg-error "no match for" "" { target *-*-* } 41 } 496</pre><p> 497 498Example 5: Testing with special command line settings, or without the 499use of pre-compiled headers, in particular the 500<code class="filename">stdc++.h.gch</code> file. Any 501options here will override the <code class="varname">DEFAULT_CXXFLAGS</code> and 502<code class="varname">PCH_CXXFLAGS</code> set up in the <code class="filename">normal.exp</code> 503file: 504</p><pre class="programlisting"> 505// { dg-options "-O0" { target *-*-* } } 506</pre><p> 507 508Example 6: Compiling and linking a test only for C++14 and later, and only 509if Debug Mode is active: 510</p><pre class="programlisting"> 511// { dg-do link { target c++14 } } 512// { dg-require-debug-mode "" } 513</pre><p> 514 515Example 7: Running a test only on x86 targets, and only for C++11 and later, 516with specific options, and additional options for 32-bit x86: 517</p><pre class="programlisting"> 518// { dg-options "-fstrict-enums" } 519// { dg-additional-options "-march=i486" { target ia32 } } 520// { dg-do run { target { ia32 || x86_64-*-* } } } 521// { dg-require-effective-target "c++11" } 522</pre><p> 523 </p><p> 524 More examples can be found in the 525 <code class="filename">libstdc++-v3/testsuite/*/*.cc</code> files. 526 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="tests.dg.directives"></a>Directives Specific to Libstdc++ Tests</h4></div></div></div><p> 527 In addition to the usual <a class="link" href="https://gcc.gnu.org/onlinedocs/gccint/Require-Support.html" target="_top">Variants 528 of <code class="literal">dg-require-<em class="replaceable"><code>support</code></em></code></a> 529 several more directives are available for use in libstdc++ tests, 530 including the following: 531 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term"><code class="literal">dg-require-namedlocale</code> <em class="replaceable"><code>name</code></em></span></dt><dd><p>The named locale must be available. 532 </p></dd><dt><span class="term"><code class="literal">dg-require-debug-mode ""</code></span></dt><dd><p>Skip the test if the Debug Mode is not active 533 (as determined by the <code class="literal">_GLIBCXX_DEBUG</code> macro). 534 </p></dd><dt><span class="term"><code class="literal">dg-require-parallel-mode ""</code></span></dt><dd><p>Skip the test if the Parallel Mode is not active 535 (as determined by the <code class="literal">_GLIBCXX_PARALLEL</code> macro). 536 </p></dd><dt><span class="term"><code class="literal">dg-require-profile-mode ""</code></span></dt><dd><p>Skip the test if the Profile Mode is not active 537 (as determined by the <code class="literal">_GLIBCXX_PROFILE</code> macro). 538 </p></dd><dt><span class="term"><code class="literal">dg-require-normal-mode ""</code></span></dt><dd><p>Skip the test if any of Debug, Parallel or Profile 539 Mode is active. 540 </p></dd><dt><span class="term"><code class="literal">dg-require-atomic-builtins ""</code></span></dt><dd><p>Skip the test if atomic operations on <span class="type">bool</span> 541 and <span class="type">int</span> are not lock-free. 542 </p></dd><dt><span class="term"><code class="literal">dg-require-gthreads ""</code></span></dt><dd><p>Skip the test if the C++11 thread library is not 543 supported, as determined by the <code class="literal">_GLIBCXX_HAS_GTHREADS</code> 544 macro. 545 </p></dd><dt><span class="term"><code class="literal">dg-require-gthreads-timed ""</code></span></dt><dd><p>Skip the test if C++11 timed mutexes are not supported, 546 as determined by the <code class="literal">_GLIBCXX_HAS_GTHREADS</code> and 547 <code class="literal">_GTHREAD_USE_MUTEX_TIMEDLOCK</code> macros. 548 </p></dd><dt><span class="term"><code class="literal">dg-require-string-conversions ""</code></span></dt><dd><p>Skip the test if the C++11 <code class="function">to_string</code> 549 and <code class="function">stoi</code>, <code class="function">stod</code> etc. functions 550 are not fully supported (including wide character versions). 551 </p></dd><dt><span class="term"><code class="literal">dg-require-filesystem-ts ""</code></span></dt><dd><p>Skip the test if the Filesystem TS is not supported. 552 </p></dd></dl></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.harness"></a>Test Harness and Utilities</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.dejagnu"></a>DejaGnu Harness Details</h4></div></div></div><p> 553 Underlying details of testing for conformance and regressions are 554 abstracted via the GNU DejaGnu package. This is similar to the 555 rest of GCC. 556 </p><p>This is information for those looking at making changes to the testsuite 557structure, and/or needing to trace DejaGnu's actions with 558<code class="option">--verbose</code>. 559This will not be useful to people who are "merely" adding new tests 560to the existing structure. 561</p><p>The first key point when working with DejaGnu is the idea of a "tool". 562Files, directories, and functions are all implicitly used when they are 563named after the tool in use. Here, the tool will always be "libstdc++". 564</p><p>The <code class="code">lib</code> subdir contains support routines. The 565<code class="code">lib/libstdc++.exp</code> file ("support library") is loaded 566automagically, and must explicitly load the others. For example, files can 567be copied from the core compiler's support directory into <code class="code">lib</code>. 568</p><p>Some routines in <code class="code">lib/libstdc++.exp</code> are callbacks, some are 569our own. Callbacks must be prefixed with the name of the tool. To easily 570distinguish the others, by convention our own routines are named "v3-*". 571</p><p>The next key point when working with DejaGnu is "test files". Any 572directory whose name starts with the tool name will be searched for test files. 573(We have only one.) In those directories, any <code class="code">.exp</code> file is 574considered a test file, and will be run in turn. Our main test file is called 575<code class="code">normal.exp</code>; it runs all the tests in testsuite_files using the 576callbacks loaded from the support library. 577</p><p>The <code class="code">config</code> directory is searched for any particular "target 578board" information unique to this library. This is currently unused and sets 579only default variables. 580</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.utils"></a>Utilities</h4></div></div></div><p> 581 </p><p> 582 The testsuite directory also contains some files that implement 583 functionality that is intended to make writing test cases easier, 584 or to avoid duplication, or to provide error checking in a way that 585 is consistent across platforms and test harnesses. A stand-alone 586 executable, called <span class="emphasis"><em>abi_check</em></span>, and a static 587 library called <span class="emphasis"><em>libtestc++</em></span> are 588 constructed. Both of these items are not installed, and only used 589 during testing. 590 </p><p> 591 These files include the following functionality: 592 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 593 <span class="emphasis"><em>testsuite_abi.h</em></span>, 594 <span class="emphasis"><em>testsuite_abi.cc</em></span>, 595 <span class="emphasis"><em>testsuite_abi_check.cc</em></span> 596 </p><p> 597 Creates the executable <span class="emphasis"><em>abi_check</em></span>. 598 Used to check correctness of symbol versioning, visibility of 599 exported symbols, and compatibility on symbols in the shared 600 library, for hosts that support this feature. More information 601 can be found in the ABI documentation <a class="link" href="abi.html" title="ABI Policy and Guidelines">here</a> 602 </p></li><li class="listitem"><p> 603 <span class="emphasis"><em>testsuite_allocator.h</em></span>, 604 <span class="emphasis"><em>testsuite_allocator.cc</em></span> 605 </p><p> 606 Contains specialized allocators that keep track of construction 607 and destruction. Also, support for overriding global new and 608 delete operators, including verification that new and delete 609 are called during execution, and that allocation over max_size 610 fails. 611 </p></li><li class="listitem"><p> 612 <span class="emphasis"><em>testsuite_character.h</em></span> 613 </p><p> 614 Contains <code class="code">std::char_traits</code> and 615 <code class="code">std::codecvt</code> specializations for a user-defined 616 POD. 617 </p></li><li class="listitem"><p> 618 <span class="emphasis"><em>testsuite_hooks.h</em></span>, 619 <span class="emphasis"><em>testsuite_hooks.cc</em></span> 620 </p><p> 621 A large number of utilities, including: 622 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>VERIFY</p></li><li class="listitem"><p>set_memory_limits</p></li><li class="listitem"><p>verify_demangle</p></li><li class="listitem"><p>run_tests_wrapped_locale</p></li><li class="listitem"><p>run_tests_wrapped_env</p></li><li class="listitem"><p>try_named_locale</p></li><li class="listitem"><p>try_mkfifo</p></li><li class="listitem"><p>func_callback</p></li><li class="listitem"><p>counter</p></li><li class="listitem"><p>copy_tracker</p></li><li class="listitem"><p>copy_constructor</p></li><li class="listitem"><p>assignment_operator</p></li><li class="listitem"><p>destructor</p></li><li class="listitem"><p>pod_char, pod_int and associated char_traits specializations</p></li></ul></div></li><li class="listitem"><p> 623 <span class="emphasis"><em>testsuite_io.h</em></span> 624 </p><p> 625 Error, exception, and constraint checking for 626 <code class="code">std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>. 627 </p></li><li class="listitem"><p> 628 <span class="emphasis"><em>testsuite_iterators.h</em></span> 629 </p><p> 630 Wrappers for various iterators. 631 </p></li><li class="listitem"><p> 632 <span class="emphasis"><em>testsuite_performance.h</em></span> 633 </p><p> 634 A number of class abstractions for performance counters, and 635 reporting functions including: 636 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>time_counter</p></li><li class="listitem"><p>resource_counter</p></li><li class="listitem"><p>report_performance</p></li></ul></div></li></ul></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="test.special"></a>Special Topics</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="test.exception.safety"></a> 637 Qualifying Exception Safety Guarantees 638 <a id="id-1.3.6.3.5.7.2.1.1.1" class="indexterm"></a> 639</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.overview"></a>Overview</h5></div></div></div><p> 640 Testing is composed of running a particular test sequence, 641 and looking at what happens to the surrounding code when 642 exceptions are thrown. Each test is composed of measuring 643 initial state, executing a particular sequence of code under 644 some instrumented conditions, measuring a final state, and 645 then examining the differences between the two states. 646 </p><p> 647 Test sequences are composed of constructed code sequences 648 that exercise a particular function or member function, and 649 either confirm no exceptions were generated, or confirm the 650 consistency/coherency of the test subject in the event of a 651 thrown exception. 652 </p><p> 653 Random code paths can be constructed using the basic test 654 sequences and instrumentation as above, only combined in a 655 random or pseudo-random way. 656 </p><p> To compute the code paths that throw, test instruments 657 are used that throw on allocation events 658 (<code class="classname">__gnu_cxx::throw_allocator_random</code> 659 and <code class="classname">__gnu_cxx::throw_allocator_limit</code>) 660 and copy, assignment, comparison, increment, swap, and 661 various operators 662 (<code class="classname">__gnu_cxx::throw_type_random</code> 663 and <code class="classname">__gnu_cxx::throw_type_limit</code>). Looping 664 through a given test sequence and conditionally throwing in 665 all instrumented places. Then, when the test sequence 666 completes without an exception being thrown, assume all 667 potential error paths have been exercised in a sequential 668 manner. 669 </p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.status"></a> 670 Existing tests 671</h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 672 Ad Hoc 673 </p><p> 674 For example, 675 <code class="filename">testsuite/23_containers/list/modifiers/3.cc</code>. 676 </p></li><li class="listitem"><p> 677 Policy Based Data Structures 678 </p><p> 679 For example, take the test 680 functor <code class="classname">rand_reg_test</code> in 681 in <code class="filename">testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</code>. This uses <code class="classname">container_rand_regression_test</code> in 682<code class="filename">testsuite/util/regression/rand/assoc/container_rand_regression_test.h</code>. 683 684 </p><p> 685 Which has several tests for container member functions, 686Includes control and test container objects. Configuration includes 687random seed, iterations, number of distinct values, and the 688probability that an exception will be thrown. Assumes instantiating 689container uses an extension 690allocator, <code class="classname">__gnu_cxx::throw_allocator_random</code>, 691as the allocator type. 692 </p></li><li class="listitem"><p> 693 C++11 Container Requirements. 694 </p><p> 695 Coverage is currently limited to testing container 696 requirements for exception safety, 697 although <code class="classname">__gnu_cxx::throw_type</code> meets 698 the additional type requirements for testing numeric data 699 structures and instantiating algorithms. 700 </p><p> 701 Of particular interest is extending testing to algorithms and 702 then to parallel algorithms. Also io and locales. 703 </p><p> 704 The test instrumentation should also be extended to add 705 instrumentation to <code class="classname">iterator</code> 706 and <code class="classname">const_iterator</code> types that throw 707 conditionally on iterator operations. 708 </p></li></ul></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="test.exception.safety.containers"></a> 709C++11 Requirements Test Sequence Descriptions 710</h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 711 Basic 712 </p><p> 713 Basic consistency on exception propagation tests. For 714 each container, an object of that container is constructed, 715 a specific member function is exercised in 716 a <code class="literal">try</code> block, and then any thrown 717 exceptions lead to error checking in the appropriate 718 <code class="literal">catch</code> block. The container's use of 719 resources is compared to the container's use prior to the 720 test block. Resource monitoring is limited to allocations 721 made through the container's <span class="type">allocator_type</span>, 722 which should be sufficient for container data 723 structures. Included in these tests are member functions 724 are <span class="type">iterator</span> and <span class="type">const_iterator</span> 725 operations, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>, <code class="function">erase</code>, <code class="function">swap</code>, <code class="function">clear</code>, 726 and <code class="function">rehash</code>. The container in question is 727 instantiated with two instrumented template arguments, 728 with <code class="classname">__gnu_cxx::throw_allocator_limit</code> 729 as the allocator type, and 730 with <code class="classname">__gnu_cxx::throw_type_limit</code> as 731 the value type. This allows the test to loop through 732 conditional throw points. 733 </p><p> 734 The general form is demonstrated in 735 <code class="filename">testsuite/23_containers/list/requirements/exception/basic.cc 736 </code>. The instantiating test object is <code class="classname">__gnu_test::basic_safety</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>. 737 </p></li><li class="listitem"><p> 738 Generation Prohibited 739 </p><p> 740 Exception generation tests. For each container, an object of 741 that container is constructed and all member functions 742 required to not throw exceptions are exercised. Included in 743 these tests are member functions 744 are <span class="type">iterator</span> and <span class="type">const_iterator</span> operations, <code class="function">erase</code>, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">swap</code>, 745 and <code class="function">clear</code>. The container in question is 746 instantiated with two instrumented template arguments, 747 with <code class="classname">__gnu_cxx::throw_allocator_random</code> 748 as the allocator type, and 749 with <code class="classname">__gnu_cxx::throw_type_random</code> as 750 the value type. This test does not loop, an instead is sudden 751 death: first error fails. 752 </p><p> 753 The general form is demonstrated in 754 <code class="filename">testsuite/23_containers/list/requirements/exception/generation_prohibited.cc 755 </code>. The instantiating test object is <code class="classname">__gnu_test::generation_prohibited</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>. 756 </p></li><li class="listitem"><p> 757 Propagation Consistent 758 </p><p> 759 Container rollback on exception propagation tests. For 760 each container, an object of that container is constructed, 761 a specific member function that requires rollback to a previous 762 known good state is exercised in 763 a <code class="literal">try</code> block, and then any thrown 764 exceptions lead to error checking in the appropriate 765 <code class="literal">catch</code> block. The container is compared to 766 the container's last known good state using such parameters 767 as size, contents, and iterator references. Included in these 768 tests are member functions 769 are <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>, 770 and <code class="function">rehash</code>. The container in question is 771 instantiated with two instrumented template arguments, 772 with <code class="classname">__gnu_cxx::throw_allocator_limit</code> 773 as the allocator type, and 774 with <code class="classname">__gnu_cxx::throw_type_limit</code> as 775 the value type. This allows the test to loop through 776 conditional throw points. 777 </p><p> 778 The general form demonstrated in 779 <code class="filename">testsuite/23_containers/list/requirements/exception/propagation_coherent.cc 780 </code>. The instantiating test object is <code class="classname">__gnu_test::propagation_coherent</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>. 781 </p></li></ul></div></div></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="internals.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="appendix_porting.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Porting to New Hardware or Operating Systems </td><td width="20%" align="center"><a accesskey="h" href="../index.html">Home</a></td><td width="40%" align="right" valign="top"> ABI Policy and Guidelines</td></tr></table></div></body></html>