1*404b540aSrobert<?xml version="1.0" encoding="ISO-8859-1"?> 2*404b540aSrobert<!DOCTYPE html 3*404b540aSrobert PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 4*404b540aSrobert "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 5*404b540aSrobert 6*404b540aSrobert<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> 7*404b540aSrobert<head> 8*404b540aSrobert <meta name="AUTHOR" content="bkoz@gcc.gnu.org (Benjamin Kosnik)" /> 9*404b540aSrobert <meta name="KEYWORDS" content="c++, libstdc++, test, regression, g++" /> 10*404b540aSrobert <meta name="DESCRIPTION" content="README for the GNU libstdc++ effort." /> 11*404b540aSrobert <meta name="GENERATOR" content="vi and eight fingers" /> 12*404b540aSrobert <title>libstdc++-v3 Testing Instructions</title> 13*404b540aSrobert<link rel="StyleSheet" href="lib3styles.css" /> 14*404b540aSrobert</head> 15*404b540aSrobert<body> 16*404b540aSrobert 17*404b540aSrobert<h1 class="centered"><a name="top">Testing Details</a></h1> 18*404b540aSrobert 19*404b540aSrobert<p class="fineprint"><em> 20*404b540aSrobert The latest version of this document is always available at 21*404b540aSrobert <a href="http://gcc.gnu.org/onlinedocs/libstdc++/test.html"> 22*404b540aSrobert http://gcc.gnu.org/onlinedocs/libstdc++/test.html</a>. 23*404b540aSrobert</em></p> 24*404b540aSrobert 25*404b540aSrobert<p><em> 26*404b540aSrobert To the <a href="http://gcc.gnu.org/libstdc++/">libstdc++-v3 homepage</a>. 27*404b540aSrobert</em></p> 28*404b540aSrobert 29*404b540aSrobert<!-- ####################################################### --> 30*404b540aSrobert<hr /> 31*404b540aSrobert<h2>Contents</h2> 32*404b540aSrobert<ul> 33*404b540aSrobert <li><a href="#org">Testsuite organization and naming conventions</a></li> 34*404b540aSrobert <li><a href="#util">Utilities: abicheck and libtestc++</a></li> 35*404b540aSrobert <li><a href="#new">How to write a new test case</a></li> 36*404b540aSrobert <li><a href="#check">Options for running the tests</a></li> 37*404b540aSrobert <li><a href="#debug">Running debug-mode tests</a></li> 38*404b540aSrobert <li><a href="#future">Future</a></li> 39*404b540aSrobert <li><a href="#internals">DejaGNU internals</a></li> 40*404b540aSrobert</ul> 41*404b540aSrobert 42*404b540aSrobert<hr /> 43*404b540aSrobert 44*404b540aSrobert<!-- ####################################################### --> 45*404b540aSrobert 46*404b540aSrobert<h2><a name="org">Testsuite organization and naming conventions</a></h2> 47*404b540aSrobert <p> 48*404b540aSrobert The directory <em>libsrcdir/testsuite</em> contains the 49*404b540aSrobert individual test cases organized in sub-directories corresponding 50*404b540aSrobert to chapters of the C++ standard (detailed below), the dejagnu 51*404b540aSrobert test harness support files, and sources to various testsuite 52*404b540aSrobert utilities that are packaged in a separate testing library. 53*404b540aSrobert </p> 54*404b540aSrobert 55*404b540aSrobert <p> All test cases for functionality required by the runtime 56*404b540aSrobert components of the C++ standard (ISO 14882) are files within the 57*404b540aSrobert following directories. 58*404b540aSrobert </p> 59*404b540aSrobert 60*404b540aSrobert <pre> 61*404b540aSrobert17_intro 62*404b540aSrobert18_support 63*404b540aSrobert19_diagnostics 64*404b540aSrobert20_util 65*404b540aSrobert21_strings 66*404b540aSrobert22_locale 67*404b540aSrobert23_containers 68*404b540aSrobert25_algorithms 69*404b540aSrobert26_numerics 70*404b540aSrobert27_io 71*404b540aSrobert </pre> 72*404b540aSrobert 73*404b540aSrobert <p> 74*404b540aSrobert In addition, the following directories include test files: 75*404b540aSrobert </p> 76*404b540aSrobert 77*404b540aSrobert <pre> 78*404b540aSroberttr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1). 79*404b540aSrobertbackward Tests for backwards compatibility and deprecated features. 80*404b540aSrobertdemangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler 81*404b540aSrobertext Tests for extensions. 82*404b540aSrobertperformance Tests for performance analysis, and performance regressions. 83*404b540aSrobertthread Tests for threads. 84*404b540aSrobert </pre> 85*404b540aSrobert 86*404b540aSrobert <p> 87*404b540aSrobert Some directories don't have test files, but instead contain 88*404b540aSrobert auxiliary information (<a href="#internals">more information</a>): 89*404b540aSrobert </p> 90*404b540aSrobert 91*404b540aSrobert <pre> 92*404b540aSrobertconfig Files for the dejagnu test harness. 93*404b540aSrobertlib Files for the dejagnu test harness. 94*404b540aSrobertlibstdc++* Files for the dejagnu test harness. 95*404b540aSrobertdata Sample text files for testing input and output. 96*404b540aSrobertutil Files for libtestc++, utilities and testing routines. 97*404b540aSrobert </pre> 98*404b540aSrobert 99*404b540aSrobert <p> 100*404b540aSrobert Within a directory that includes test files, there may be 101*404b540aSrobert additional subdirectories, or files. Originally, test cases 102*404b540aSrobert were appended to one file that represented a particular section 103*404b540aSrobert of the chapter under test, and was named accordingly. For 104*404b540aSrobert instance, to test items related to <code> 21.3.6.1 - 105*404b540aSrobert basic_string::find [lib.string::find]</code> in the standard, 106*404b540aSrobert the following was used: 107*404b540aSrobert </p> 108*404b540aSrobert <pre> 109*404b540aSrobert21_strings/find.cc 110*404b540aSrobert </pre> 111*404b540aSrobert <p> 112*404b540aSrobert However, that practice soon became a liability as the test cases 113*404b540aSrobert became huge and unwieldy, and testing new or extended 114*404b540aSrobert functionality (like wide characters or named locales) became 115*404b540aSrobert frustrating, leading to aggressive pruning of test cases on some 116*404b540aSrobert platforms that covered up implementation errors. Now, the test 117*404b540aSrobert suite has a policy of one file, one test case, which solves the 118*404b540aSrobert above issues and gives finer grained results and more manageable 119*404b540aSrobert error debugging. As an example, the test case quoted above 120*404b540aSrobert becomes: 121*404b540aSrobert </p> 122*404b540aSrobert <pre> 123*404b540aSrobert21_strings/basic_string/find/char/1.cc 124*404b540aSrobert21_strings/basic_string/find/char/2.cc 125*404b540aSrobert21_strings/basic_string/find/char/3.cc 126*404b540aSrobert21_strings/basic_string/find/wchar_t/1.cc 127*404b540aSrobert21_strings/basic_string/find/wchar_t/2.cc 128*404b540aSrobert21_strings/basic_string/find/wchar_t/3.cc 129*404b540aSrobert </pre> 130*404b540aSrobert 131*404b540aSrobert <p> 132*404b540aSrobert All new tests should be written with the policy of one test 133*404b540aSrobert case, one file in mind. 134*404b540aSrobert </p> 135*404b540aSrobert 136*404b540aSrobert <p> 137*404b540aSrobert In addition, there are some special names and suffixes that are 138*404b540aSrobert used within the testsuite to designate particular kinds of 139*404b540aSrobert tests. 140*404b540aSrobert </p> 141*404b540aSrobert 142*404b540aSrobert<ul> 143*404b540aSrobert<li> 144*404b540aSrobert <em>_xin.cc</em> 145*404b540aSrobert <p> 146*404b540aSrobert This test case expects some kind of interactive input in order 147*404b540aSrobert to finish or pass. At the moment, the interactive tests are not 148*404b540aSrobert run by default. Instead, they are run by hand, like: 149*404b540aSrobert </p> 150*404b540aSrobert <pre> 151*404b540aSrobertg++ 27_io/objects/char/3_xin.cc 152*404b540aSrobertcat 27_io/objects/char/3_xin.in | a.out 153*404b540aSrobert </pre> 154*404b540aSrobert</li> 155*404b540aSrobert<li> 156*404b540aSrobert <em>.in</em> 157*404b540aSrobert <p> 158*404b540aSrobert This file contains the expected input for the corresponding <em> 159*404b540aSrobert _xin.cc</em> test case. 160*404b540aSrobert </p> 161*404b540aSrobert</li> 162*404b540aSrobert<li> 163*404b540aSrobert <em>_neg.cc</em> 164*404b540aSrobert <p> 165*404b540aSrobert This test case is expected to fail: it's a negative test. At the 166*404b540aSrobert moment, these are almost always compile time errors. 167*404b540aSrobert </p> 168*404b540aSrobert</li> 169*404b540aSrobert<li> 170*404b540aSrobert <em>char</em> 171*404b540aSrobert <p> 172*404b540aSrobert This can either be a directory name or part of a longer file 173*404b540aSrobert name, and indicates that this file, or the files within this 174*404b540aSrobert directory are testing the <code>char</code> instantiation of a 175*404b540aSrobert template. 176*404b540aSrobert </p> 177*404b540aSrobert</li> 178*404b540aSrobert<li> 179*404b540aSrobert <em>wchar_t</em> 180*404b540aSrobert <p> 181*404b540aSrobert This can either be a directory name or part of a longer file 182*404b540aSrobert name, and indicates that this file, or the files within this 183*404b540aSrobert directory are testing the <code>wchar_t</code> instantiation of 184*404b540aSrobert a template. Some hosts do not support <code>wchar_t</code> 185*404b540aSrobert functionality, so for these targets, all of these tests will not 186*404b540aSrobert be run. 187*404b540aSrobert </p> 188*404b540aSrobert</li> 189*404b540aSrobert<li> 190*404b540aSrobert <em>thread</em> 191*404b540aSrobert <p> 192*404b540aSrobert This can either be a directory name or part of a longer file 193*404b540aSrobert name, and indicates that this file, or the files within this 194*404b540aSrobert directory are testing situations where multiple threads are 195*404b540aSrobert being used. 196*404b540aSrobert </p> 197*404b540aSrobert</li> 198*404b540aSrobert<li> 199*404b540aSrobert <em>performance</em> 200*404b540aSrobert <p> 201*404b540aSrobert This can either be an enclosing directory name or part of a 202*404b540aSrobert specific file name. This indicates a test that is used to 203*404b540aSrobert analyze runtime performance, for performance regression testing, 204*404b540aSrobert or for other optimization related analysis. At the moment, these 205*404b540aSrobert test cases are not run by default. 206*404b540aSrobert </p> 207*404b540aSrobert</li> 208*404b540aSrobert</ul> 209*404b540aSrobert 210*404b540aSrobert<hr /> 211*404b540aSrobert<h2><a name="util">Utilities: abi_check and libtestc++</a></h2> 212*404b540aSrobert <p> 213*404b540aSrobert The testsuite directory also contains some files that implement 214*404b540aSrobert functionality that is intended to make writing test cases easier, 215*404b540aSrobert or to avoid duplication, or to provide error checking in a way that 216*404b540aSrobert is consistent across platforms and test harnesses. A stand-alone 217*404b540aSrobert executable, called <em>abi_check</em>, and a static library called 218*404b540aSrobert <em>libtestc++</em> are constructed. Both of these items are not 219*404b540aSrobert installed, and only used during testing. 220*404b540aSrobert </p> 221*404b540aSrobert 222*404b540aSrobert <p> 223*404b540aSrobert These files include the following functionality: 224*404b540aSrobert </p> 225*404b540aSrobert 226*404b540aSrobert <ul> 227*404b540aSrobert <li> 228*404b540aSrobert <em>testsuite_abi.h</em>, 229*404b540aSrobert <em>testsuite_abi.cc</em>, 230*404b540aSrobert <em>testsuite_abi_check.cc</em> 231*404b540aSrobert <p> 232*404b540aSrobert Creates the executable <em>abi_check</em>. 233*404b540aSrobert Used to check correctness of symbol versioning, visibility of 234*404b540aSrobert exported symbols, and compatibility on symbols in the shared 235*404b540aSrobert library, for hosts that support this feature. More information 236*404b540aSrobert can be found in the ABI documentation <a href="abi.html"> here</a> 237*404b540aSrobert </p> 238*404b540aSrobert </li> 239*404b540aSrobert <li> 240*404b540aSrobert <em>testsuite_allocator.h</em>, 241*404b540aSrobert <em>testsuite_allocator.cc</em> 242*404b540aSrobert <p> 243*404b540aSrobert Contains specialized allocators that keep track of construction 244*404b540aSrobert and destruction. Also, support for overriding global new and 245*404b540aSrobert delete operators, including verification that new and delete 246*404b540aSrobert are called during execution, and that allocation over max_size 247*404b540aSrobert fails. 248*404b540aSrobert </p> 249*404b540aSrobert </li> 250*404b540aSrobert <li> 251*404b540aSrobert <em>testsuite_character.h</em> 252*404b540aSrobert <p> 253*404b540aSrobert Contains <code>std::char_traits</code> and 254*404b540aSrobert <code>std::codecvt</code> specializations for a user-defined 255*404b540aSrobert POD. 256*404b540aSrobert </p> 257*404b540aSrobert </li> 258*404b540aSrobert <li> 259*404b540aSrobert <em>testsuite_hooks.h</em>, 260*404b540aSrobert <em>testsuite_hooks.cc</em> 261*404b540aSrobert <p> 262*404b540aSrobert A large number of utilities, including: 263*404b540aSrobert </p> 264*404b540aSrobert <ul> 265*404b540aSrobert <li>VERIFY</li> 266*404b540aSrobert <li>set_memory_limits</li> 267*404b540aSrobert <li>verify_demangle</li> 268*404b540aSrobert <li>run_tests_wrapped_locale</li> 269*404b540aSrobert <li>run_tests_wrapped_env</li> 270*404b540aSrobert <li>try_named_locale</li> 271*404b540aSrobert <li>try_mkfifo</li> 272*404b540aSrobert <li>func_callback</li> 273*404b540aSrobert <li>counter</li> 274*404b540aSrobert <li>copy_tracker</li> 275*404b540aSrobert <li>copy_constructor</li> 276*404b540aSrobert <li>assignment_operator</li> 277*404b540aSrobert <li>destructor</li> 278*404b540aSrobert <li>pod_char, pod_int and associated char_traits specializations</li> 279*404b540aSrobert </ul> 280*404b540aSrobert <p></p> 281*404b540aSrobert </li> 282*404b540aSrobert <li> 283*404b540aSrobert <em>testsuite_io.h</em> 284*404b540aSrobert <p> 285*404b540aSrobert Error, exception, and constraint checking for 286*404b540aSrobert <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>. 287*404b540aSrobert </p> 288*404b540aSrobert </li> 289*404b540aSrobert <li> 290*404b540aSrobert <em>testsuite_iterators.h</em> 291*404b540aSrobert <p> 292*404b540aSrobert Wrappers for various iterators. 293*404b540aSrobert </p> 294*404b540aSrobert </li> 295*404b540aSrobert <li> 296*404b540aSrobert <em>testsuite_performance.h</em> 297*404b540aSrobert <p> 298*404b540aSrobert A number of class abstractions for performance counters, and 299*404b540aSrobert reporting functions including: 300*404b540aSrobert </p> 301*404b540aSrobert <ul> 302*404b540aSrobert <li>time_counter</li> 303*404b540aSrobert <li>resource_counter</li> 304*404b540aSrobert <li>report_performance</li> 305*404b540aSrobert </ul> 306*404b540aSrobert <p></p> 307*404b540aSrobert </li> 308*404b540aSrobert </ul> 309*404b540aSrobert 310*404b540aSrobert<hr /> 311*404b540aSrobert<h2><a name="new">How to write a new test case</a></h2> 312*404b540aSrobert 313*404b540aSrobert <p> 314*404b540aSrobert The first step in making a new test case is to choose the correct 315*404b540aSrobert directory and file name, given the organization as previously 316*404b540aSrobert described. 317*404b540aSrobert </p> 318*404b540aSrobert 319*404b540aSrobert <p> 320*404b540aSrobert All files are copyright the FSF, and GPL'd: this is very 321*404b540aSrobert important. The first copyright year should correspond to the date 322*404b540aSrobert the file was checked in to CVS. 323*404b540aSrobert </p> 324*404b540aSrobert 325*404b540aSrobert <p> 326*404b540aSrobert As per the dejagnu instructions, always return 0 from main to 327*404b540aSrobert indicate success. 328*404b540aSrobert </p> 329*404b540aSrobert 330*404b540aSrobert <p> 331*404b540aSrobert A bunch of utility functions and classes have already been 332*404b540aSrobert abstracted out into the testsuite utility library, <code> 333*404b540aSrobert libtestc++</code>. To use this functionality, just include the 334*404b540aSrobert appropriate header file: the library or specific object files will 335*404b540aSrobert automatically be linked in as part of the testsuite run. 336*404b540aSrobert </p> 337*404b540aSrobert 338*404b540aSrobert <p> 339*404b540aSrobert For a test that needs to take advantage of the dejagnu test 340*404b540aSrobert harness, what follows below is a list of special keyword that 341*404b540aSrobert harness uses. Basically, a test case contains dg-keywords (see 342*404b540aSrobert dg.exp) indicating what to do and what kinds of behavior are to be 343*404b540aSrobert expected. New test cases should be written with the new style 344*404b540aSrobert DejaGnu framework in mind. 345*404b540aSrobert </p> 346*404b540aSrobert 347*404b540aSrobert <p> 348*404b540aSrobert To ease transition, here is the list of dg-keyword documentation 349*404b540aSrobert lifted from dg.exp. 350*404b540aSrobert </p> 351*404b540aSrobert 352*404b540aSrobert<pre> 353*404b540aSrobert# The currently supported options are: 354*404b540aSrobert# 355*404b540aSrobert# dg-prms-id N 356*404b540aSrobert# set prms_id to N 357*404b540aSrobert# 358*404b540aSrobert# dg-options "options ..." [{ target selector }] 359*404b540aSrobert# specify special options to pass to the tool (eg: compiler) 360*404b540aSrobert# 361*404b540aSrobert# dg-do do-what-keyword [{ target/xfail selector }] 362*404b540aSrobert# `do-what-keyword' is tool specific and is passed unchanged to 363*404b540aSrobert# ${tool}-dg-test. An example is gcc where `keyword' can be any of: 364*404b540aSrobert# preprocess|compile|assemble|link|run 365*404b540aSrobert# and will do one of: produce a .i, produce a .s, produce a .o, 366*404b540aSrobert# produce an a.out, or produce an a.out and run it (the default is 367*404b540aSrobert# compile). 368*404b540aSrobert# 369*404b540aSrobert# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]] 370*404b540aSrobert# indicate an error message <regexp> is expected on this line 371*404b540aSrobert# (the test fails if it doesn't occur) 372*404b540aSrobert# Linenum=0 for general tool messages (eg: -V arg missing). 373*404b540aSrobert# "." means the current line. 374*404b540aSrobert# 375*404b540aSrobert# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]] 376*404b540aSrobert# indicate a warning message <regexp> is expected on this line 377*404b540aSrobert# (the test fails if it doesn't occur) 378*404b540aSrobert# 379*404b540aSrobert# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]] 380*404b540aSrobert# indicate a bogus error message <regexp> use to occur here 381*404b540aSrobert# (the test fails if it does occur) 382*404b540aSrobert# 383*404b540aSrobert# dg-build regexp comment [{ target/xfail selector }] 384*404b540aSrobert# indicate the build use to fail for some reason 385*404b540aSrobert# (errors covered here include bad assembler generated, tool crashes, 386*404b540aSrobert# and link failures) 387*404b540aSrobert# (the test fails if it does occur) 388*404b540aSrobert# 389*404b540aSrobert# dg-excess-errors comment [{ target/xfail selector }] 390*404b540aSrobert# indicate excess errors are expected (any line) 391*404b540aSrobert# (this should only be used sparingly and temporarily) 392*404b540aSrobert# 393*404b540aSrobert# dg-output regexp [{ target selector }] 394*404b540aSrobert# indicate the expected output of the program is <regexp> 395*404b540aSrobert# (there may be multiple occurrences of this, they are concatenated) 396*404b540aSrobert# 397*404b540aSrobert# dg-final { tcl code } 398*404b540aSrobert# add some tcl code to be run at the end 399*404b540aSrobert# (there may be multiple occurrences of this, they are concatenated) 400*404b540aSrobert# (unbalanced braces must be \-escaped) 401*404b540aSrobert# 402*404b540aSrobert# "{ target selector }" is a list of expressions that determine whether the 403*404b540aSrobert# test succeeds or fails for a particular target, or in some cases whether the 404*404b540aSrobert# option applies for a particular target. If the case of `dg-do' it specifies 405*404b540aSrobert# whether the test case is even attempted on the specified target. 406*404b540aSrobert# 407*404b540aSrobert# The target selector is always optional. The format is one of: 408*404b540aSrobert# 409*404b540aSrobert# { xfail *-*-* ... } - the test is expected to fail for the given targets 410*404b540aSrobert# { target *-*-* ... } - the option only applies to the given targets 411*404b540aSrobert# 412*404b540aSrobert# At least one target must be specified, use *-*-* for "all targets". 413*404b540aSrobert# At present it is not possible to specify both `xfail' and `target'. 414*404b540aSrobert# "native" may be used in place of "*-*-*". 415*404b540aSrobert 416*404b540aSrobertExample 1: Testing compilation only 417*404b540aSrobert// { dg-do compile } 418*404b540aSrobert 419*404b540aSrobertExample 2: Testing for expected warnings on line 36, which all targets fail 420*404b540aSrobert// { dg-warning "string literals" "" { xfail *-*-* } 36 421*404b540aSrobert 422*404b540aSrobertExample 3: Testing for expected warnings on line 36 423*404b540aSrobert// { dg-warning "string literals" "" { target *-*-* } 36 424*404b540aSrobert 425*404b540aSrobertExample 4: Testing for compilation errors on line 41 426*404b540aSrobert// { dg-do compile } 427*404b540aSrobert// { dg-error "no match for" "" { target *-*-* } 41 } 428*404b540aSrobert 429*404b540aSrobertExample 5: Testing with special command line settings, or without the 430*404b540aSrobertuse of pre-compiled headers, in particular the stdc++.h.gch file. Any 431*404b540aSrobertoptions here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set 432*404b540aSrobertup in the normal.exp file. 433*404b540aSrobert// { dg-options "-O0" { target *-*-* } } 434*404b540aSrobert</pre> 435*404b540aSrobert 436*404b540aSrobert <p> 437*404b540aSrobert More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. 438*404b540aSrobert </p> 439*404b540aSrobert 440*404b540aSrobert<hr /> 441*404b540aSrobert<h2><a name="check">Options for running the tests</a></h2> 442*404b540aSrobert 443*404b540aSrobert <p> There are several options for running tests, including testing 444*404b540aSrobert the regression tests, testing a subset of the regression tests, 445*404b540aSrobert testing the performance tests, testing just compilation, testing 446*404b540aSrobert installed tools, etc. In addition, there is a special rule for 447*404b540aSrobert checking the exported symbols of the shared library. 448*404b540aSrobert </p> 449*404b540aSrobert 450*404b540aSrobert <p>You can check the status of the build without installing it 451*404b540aSrobert using the dejagnu harness, much like the rest of the gcc tools.</p> 452*404b540aSrobert <pre> make check</pre> 453*404b540aSrobert <p>in the <em>libbuilddir</em> directory.</p> 454*404b540aSrobert <p>or</p> 455*404b540aSrobert <pre> make check-target-libstdc++-v3</pre> 456*404b540aSrobert <p>in the <em>gccbuilddir</em> directory.</p> 457*404b540aSrobert 458*404b540aSrobert <p> 459*404b540aSrobert These commands are functionally equivalent and will create a 460*404b540aSrobert 'testsuite' directory underneath <em>libbuilddir</em> containing 461*404b540aSrobert the results of the tests. Two results files will be generated: 462*404b540aSrobert <em> libstdc++.sum</em>, which is a PASS/FAIL summary for each 463*404b540aSrobert test, and <em>libstdc++.log</em> which is a log of the exact 464*404b540aSrobert command line passed to the compiler, the compiler output, and 465*404b540aSrobert the executable output (if any). 466*404b540aSrobert </p> 467*404b540aSrobert 468*404b540aSrobert 469*404b540aSrobert<p> 470*404b540aSrobertTo debug the dejagnu test harness during runs, try invoking with a 471*404b540aSrobertspecific argument to the variable RUNTESTFLAGS, as below. 472*404b540aSrobert</p> 473*404b540aSrobert 474*404b540aSrobert<pre> 475*404b540aSrobertmake check-target-libstdc++-v3 RUNTESTFLAGS="-v" 476*404b540aSrobert</pre> 477*404b540aSrobertor 478*404b540aSrobert<pre> 479*404b540aSrobertmake check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" 480*404b540aSrobert</pre> 481*404b540aSrobert 482*404b540aSrobert<p> To run a subset of the library tests, try using a command like the 483*404b540aSrobertfollowing from the <em>libbuilddir/testsuite</em> directory: 484*404b540aSrobert</p> 485*404b540aSrobert<pre> 486*404b540aSrobertruntest --tool libstdc++ normal.exp="`find $srcdir/17_intro -name *.cc`" 487*404b540aSrobert</pre> 488*404b540aSrobert 489*404b540aSrobert 490*404b540aSrobert<p> 491*404b540aSrobertThere are two ways to run on a simulator: set up DEJAGNU to point to a 492*404b540aSrobertspecially crafted site.exp, or pass down --target_board flags. 493*404b540aSrobert</p> 494*404b540aSrobertExample flags to pass down for various embedded builds are as follows: 495*404b540aSrobert<pre> 496*404b540aSrobert--target=powerpc-eabism (libgloss/sim) 497*404b540aSrobertmake check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" 498*404b540aSrobert 499*404b540aSrobert--target=calmrisc32 (libgloss/sid) 500*404b540aSrobertmake check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" 501*404b540aSrobert 502*404b540aSrobert--target=xscale-elf (newlib/sim) 503*404b540aSrobertmake check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" 504*404b540aSrobert</pre> 505*404b540aSrobert 506*404b540aSrobert<p> Also, here is an example of how to run the libstdc++ testsuite for a 507*404b540aSrobertmultilibed build directory with different ABI settings: 508*404b540aSrobert</p> 509*404b540aSrobert<pre> 510*404b540aSrobertmake check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"' 511*404b540aSrobert</pre> 512*404b540aSrobert 513*404b540aSrobert<p> 514*404b540aSrobertYou can run the tests with a compiler and library that have already 515*404b540aSrobertbeen installed. Make sure that the compiler (e.g., <code>g++</code>) 516*404b540aSrobertis in your <code>PATH</code>. If you are using shared libraries, then 517*404b540aSrobertyou must also ensure that the directory containing the shared version 518*404b540aSrobertof libstdc++ is in your <code>LD_LIBRARY_PATH</code>, or equivalent. 519*404b540aSrobertIf your GCC source tree is at <code>/path/to/gcc</code>, then you can 520*404b540aSrobertrun the tests as follows: 521*404b540aSrobert<pre> 522*404b540aSrobertruntest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite 523*404b540aSrobert</pre> 524*404b540aSrobertThe testsuite will create a number of files in the directory in which you 525*404b540aSrobertrun this command,. Some of those files might use the same name as 526*404b540aSrobertfiles created by other testsuites (like the ones for GCC and G++), so 527*404b540aSrobertyou should not try to run all the testsuites in parallel from the same 528*404b540aSrobertdirectory. 529*404b540aSrobert</p> 530*404b540aSrobert 531*404b540aSrobert <p> In addition, there are some testing options that are mostly of 532*404b540aSrobert interest to library maintainers and system integrators. As such, 533*404b540aSrobert these tests may not work on all cpu and host combinations, and may need to 534*404b540aSrobert be executed in the <em>libbuilddir/testsuite</em> directory. These options 535*404b540aSrobert include, but are not necessarily limited to, the following: 536*404b540aSrobert </p> 537*404b540aSrobert 538*404b540aSrobert <pre> 539*404b540aSrobert make testsuite_files</pre> 540*404b540aSrobert <p> 541*404b540aSrobert Five files are generated that determine what test files 542*404b540aSrobert are run. These files are: 543*404b540aSrobert </p> 544*404b540aSrobert <ul> 545*404b540aSrobert <li> 546*404b540aSrobert <em>testsuite_files </em> 547*404b540aSrobert <p> This is a list of all the test cases that will be run. Each 548*404b540aSrobert test case is on a separate line, given with an absolute path 549*404b540aSrobert from the <em>libsrcdir/testsuite</em> directory. 550*404b540aSrobert </p> 551*404b540aSrobert </li> 552*404b540aSrobert 553*404b540aSrobert <li> 554*404b540aSrobert <em>testsuite_files_interactive </em> 555*404b540aSrobert <p> This is a list of all the interactive test cases, using the 556*404b540aSrobert same format as the file list above. These tests are not run by default. 557*404b540aSrobert </p> 558*404b540aSrobert </li> 559*404b540aSrobert 560*404b540aSrobert <li> 561*404b540aSrobert <em>testsuite_files_performance</em> 562*404b540aSrobert <p> This is a list of all the performance test cases, using the 563*404b540aSrobert same format as the file list above. These tests are not run by default. 564*404b540aSrobert </p> 565*404b540aSrobert </li> 566*404b540aSrobert 567*404b540aSrobert <li> 568*404b540aSrobert <em>testsuite_thread</em> 569*404b540aSrobert <p> This file indicates that the host system can run tests which 570*404b540aSrobert incolved multiple threads. 571*404b540aSrobert </p> 572*404b540aSrobert </li> 573*404b540aSrobert 574*404b540aSrobert <li> 575*404b540aSrobert <em>testsuite_wchar_t</em> 576*404b540aSrobert <p> This file indicates that the host system can run the wchar_t 577*404b540aSrobert tests, and corresponds to the macro definition <code> 578*404b540aSrobert _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h. 579*404b540aSrobert </p> 580*404b540aSrobert </li> 581*404b540aSrobert </ul> 582*404b540aSrobert 583*404b540aSrobert <pre> 584*404b540aSrobert make check-abi</pre> 585*404b540aSrobert <p>The library ABI can be tested. This involves testing the shared 586*404b540aSrobert library against an ABI-defining previous version of symbol exports. </p> 587*404b540aSrobert 588*404b540aSrobert <pre> 589*404b540aSrobert make check-compile</pre> 590*404b540aSrobert <p>This rule compiles, but does not link or execute, the 591*404b540aSrobert <em>testsuite_files</em> test cases and displays the output on stdout.</p> 592*404b540aSrobert 593*404b540aSrobert <pre> 594*404b540aSrobert make check-performance</pre> 595*404b540aSrobert <p>This rule runs through the <em>testsuite_files_performance</em> 596*404b540aSrobert test cases and collects information for performance analysis and 597*404b540aSrobert can be used to spot performance regressions. Various timing 598*404b540aSrobert information is collected, as well as number of hard page faults, 599*404b540aSrobert and memory used. This is not run by default, and the implementation 600*404b540aSrobert is in flux. 601*404b540aSrobert</p> 602*404b540aSrobert 603*404b540aSrobert <p> 604*404b540aSrobert We are interested in any strange failures of the 605*404b540aSrobert testsuite; please see <a href="faq/index.html#2_4">FAQ 2.4</a> 606*404b540aSrobert for which files to examine. 607*404b540aSrobert </p> 608*404b540aSrobert 609*404b540aSrobert<hr/> 610*404b540aSrobert<h2><a name="debug">Running debug-mode tests</a></h2> 611*404b540aSrobert<p>To run the libstdc++ test suite under the <a 612*404b540aSrobert href="debug.html#safe">debug mode</a>, 613*404b540aSrobert edit <code>libstdc++/scripts/testsuite_flags</code> to add the 614*404b540aSrobert compile-time flag <code>-D_GLIBCXX_DEBUG</code> to the result 615*404b540aSrobert printed by the <code>--build-cxx</code> option. Additionally, add 616*404b540aSrobert the <code>-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on pedantic 617*404b540aSrobert checking. The libstdc++ test suite should produce precisely the same 618*404b540aSrobert results under debug mode that it does under release mode: any 619*404b540aSrobert deviation indicates an error in either the library or the test 620*404b540aSrobert suite.</p> 621*404b540aSrobert 622*404b540aSrobert<hr /> 623*404b540aSrobert<h2><a name="future">Future</a></h2> 624*404b540aSrobert 625*404b540aSrobert<p> 626*404b540aSrobertShared runs need to be implemented, for targets that support shared libraries. 627*404b540aSrobert</p> 628*404b540aSrobert 629*404b540aSrobert<p> 630*404b540aSrobertDiffing of expected output to standard streams needs to be finished off. 631*404b540aSrobert</p> 632*404b540aSrobert 633*404b540aSrobert<p> 634*404b540aSrobertThe V3 testing framework supports, or will eventually support, 635*404b540aSrobertadditional keywords for the purpose of easing the job of writing 636*404b540aSroberttest cases. All V3-keywords are of the form <code>@xxx@</code>. 637*404b540aSrobertCurrently plans for supported keywords include: 638*404b540aSrobert</p> 639*404b540aSrobert 640*404b540aSrobert<dl> 641*404b540aSrobert<dt> <code> @require@ <files> </code> </dt> 642*404b540aSrobert<dd> 643*404b540aSrobert <p> 644*404b540aSrobert The existence of <files> is essential for the test to complete 645*404b540aSrobert successfully. For example, a test case foo.C using bar.baz as 646*404b540aSrobert input file could say 647*404b540aSrobert </p> 648*404b540aSrobert <pre> 649*404b540aSrobert // @require@ bar.baz</pre> 650*404b540aSrobert <p> 651*404b540aSrobert The special variable % stands for the rootname, e.g. the 652*404b540aSrobert file-name without its `.C' extension. Example of use (taken 653*404b540aSrobert verbatim from 27_io/filebuf.cc) 654*404b540aSrobert </p> 655*404b540aSrobert <pre> 656*404b540aSrobert // @require@ %-*.tst %-*.txt</pre> 657*404b540aSrobert</dd> 658*404b540aSrobert<dt> <code> @diff@ <first-list> <second-list> </code> </dt> 659*404b540aSrobert<dd> 660*404b540aSrobert <p> 661*404b540aSrobert After the test case compiles and ran successfully, diff 662*404b540aSrobert <first-list> against <second-list>, these lists should 663*404b540aSrobert have the same length. The test fails if diff returns non-zero a 664*404b540aSrobert pair of files. 665*404b540aSrobert </p> 666*404b540aSrobert</dd> 667*404b540aSrobert</dl> 668*404b540aSrobert 669*404b540aSrobert<hr /> 670*404b540aSrobert<h2><a name="internals">DejaGNU internals</a></h2> 671*404b540aSrobert 672*404b540aSrobert<p>This is information for those looking at making changes to the testsuite 673*404b540aSrobertstructure, and/or needing to trace dejagnu's actions with --verbose. This 674*404b540aSrobertwill not be useful to people who are "merely" adding new tests to the existing 675*404b540aSrobertstructure. 676*404b540aSrobert</p> 677*404b540aSrobert 678*404b540aSrobert<p>The first key point when working with dejagnu is the idea of a "tool". 679*404b540aSrobertFiles, directories, and functions are all implicitly used when they are 680*404b540aSrobertnamed after the tool in use. Here, the tool will always be "libstdc++". 681*404b540aSrobert</p> 682*404b540aSrobert 683*404b540aSrobert<p>The <code>lib</code> subdir contains support routines. The 684*404b540aSrobert<code>lib/libstdc++.exp</code> file ("support library") is loaded 685*404b540aSrobertautomagically, and must explicitly load the others. For example, files can 686*404b540aSrobertbe copied from the core compiler's support directory into <code>lib</code>. 687*404b540aSrobert</p> 688*404b540aSrobert 689*404b540aSrobert<p>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are 690*404b540aSrobertour own. Callbacks must be prefixed with the name of the tool. To easily 691*404b540aSrobertdistinguish the others, by convention our own routines are named "v3-*". 692*404b540aSrobert</p> 693*404b540aSrobert 694*404b540aSrobert<p>The next key point when working with dejagnu is "test files". Any 695*404b540aSrobertdirectory whose name starts with the tool name will be searched for test files. 696*404b540aSrobert(We have only one.) In those directories, any <code>.exp</code> file is 697*404b540aSrobertconsidered a test file, and will be run in turn. Our main test file is called 698*404b540aSrobert<code>normal.exp</code>; it runs all the tests in testsuite_files using the 699*404b540aSrobertcallbacks loaded from the support library. 700*404b540aSrobert</p> 701*404b540aSrobert 702*404b540aSrobert<p>The <code>config</code> directory is searched for any particular "target 703*404b540aSrobertboard" information unique to this library. This is currently unused and sets 704*404b540aSrobertonly default variables. 705*404b540aSrobert</p> 706*404b540aSrobert 707*404b540aSrobert 708*404b540aSrobert<!-- ####################################################### --> 709*404b540aSrobert 710*404b540aSrobert<hr /> 711*404b540aSrobert<p class="fineprint"><em> 712*404b540aSrobertSee <a href="17_intro/license.html">license.html</a> for copying conditions. 713*404b540aSrobertComments and suggestions are welcome, and may be sent to 714*404b540aSrobert<a href="mailto:libstdc++@gcc.gnu.org">the libstdc++ mailing list</a>. 715*404b540aSrobert</em></p> 716*404b540aSrobert 717*404b540aSrobert 718*404b540aSrobert</body> 719*404b540aSrobert</html> 720