1Major changes between releases Automated Testing Framework 2=========================================================================== 3 4 5Changes in version 0.13 6*********************** 7 8Experimental version released on March 31st, 2011. 9 10This is the first release after the creation of the Kyua project, a more 11modular and reliable replacement for ATF. From now on, ATF will change to 12accomodate the transition to this new codebase, but ATF will still continue 13to see development in the short/medium term. Check out the project page at 14http://code.google.com/p/kyua/ for more details. 15 16The changes in this release are: 17 18* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 19 new package that aims to replace atf-run and atf-report. The ATF tests 20 can be run with the new system by issuing a 'make installcheck-kyua' from 21 the top-level directory of the project (assuming the 'kyua' binary is 22 available during the configuration stage of ATF). 23 24* atf-run and atf-report are now in maintenance mode (but *not* deprecated 25 yet!). Kyua already implements a new, much more reliable runtime engine 26 that provides similar features to these tools. That said, it is not 27 complete yet so all development efforts should go towards it. 28 29* If GDB is installed, atf-run dumps the stack trace of crashing test 30 programs in an attempt to aid debugging. Contributed by Antti Kantee. 31 32* Reverted default timeout change in previous release and reset its value 33 to 5 minutes. This was causing several issues, specially when running 34 the existing NetBSD test suite in qemu. 35 36* Fixed the 'match' output checker in atf-check to properly validate the 37 last line of a file even if it does not have a newline. 38 39* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 40 check for the presence (or lack thereof) of an element in a collection. 41 42* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 43 when the cleanup of a test case triggered asynchronous modifications to 44 its work directory (e.g. killing a daemon process that cleans up a pid 45 file in the work directory). 46 47* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 48 instead of just listing them as having 0 test cases. 49 50 51Changes in version 0.12 52*********************** 53 54Experimental version released on November 7th, 2010. 55 56* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 57 ATF_REQUIRE_THROW but allows checking for the validity of the exception's 58 error message by means of a regular expression. 59 60* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 61 regular expression match in a string. 62 63* Changed the default timeout for test cases from 5 minutes to 30 seconds. 64 30 seconds is long enough for virtually all tests to complete, and 5 65 minutes is a way too long pause in a test suite where a single test case 66 stalls. 67 68* Deprecated the use.fs property. While this seemed like a good idea in 69 the first place to impose more control on what test cases can do, it 70 turns out to be bad. First, use.fs=false prevents bogus test cases 71 from dumping core so after-the-fact debugging is harder. Second, 72 supporting use.fs adds a lot of unnecessary complexity. atf-run will 73 now ignore any value provided to use.fs and will allow test cases to 74 freely access the file system if they wish to. 75 76* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 77 library. The 'text' module became private in 0.11 but was being used 78 externally to simplify the parsing of configuration variables. 79 80* Made atf-run recognize the 'unprivileged-user' configuration variable 81 and automatically drop root privileges when a test case sets 82 require.user=unprivileged. Note that this is, by no means, done for 83 security purposes; this is just for user convenience; tests should, in 84 general, not be blindly run as root in the first place. 85 86 87Changes in version 0.11 88*********************** 89 90Experimental version released on October 20th, 2010. 91 92* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 93 their counterparts in atf-c. 94 95* Clearly separated the modules in atf-c that are supposed to be public 96 from those that are implementation details. The header files for the 97 internal modules are not installed any more. 98 99* Made the atf-check tool private. It is only required by atf-sh and being 100 public has the danger of causing confusion. Also, making it private 101 simplifies the public API of atf. 102 103* Changed atf-sh to enable per-command error checking (set -e) by default. 104 This catches many cases in which a test case is broken but it is not 105 reported as such because execution continues. 106 107* Fixed the XSTL and CSS stylesheets to support expected failures. 108 109 110Changes in version 0.10 111*********************** 112 113Experimental version released on July 2nd, 2010. 114 115Miscellaneous features 116 117* Added expected failures support to test cases and atf-run. These 118 include, for example, expected clean exits, expected reception of fatal 119 signals, expected timeouts and expected errors in condition checks. 120 These statuses can be used to denote test cases that are known to fail 121 due to a bug in the code they are testing. atf-report reports these 122 tests separately but they do not count towards the failed test cases 123 amount. 124 125* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 126 allow easy checking of call failures that update errno. 127 128* Added the has.cleanup meta-data property to test caes that specifies 129 whether the test case has a cleanup routine or not; its value is 130 automatically set. This property is read by atf-run to know if it has to 131 run the cleanup routine; skipping this run for every test case 132 significantly speeds up the run time of test suites. 133 134* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 135 take the expected exception as the first argument and the statement to 136 execute as the second argument. 137 138Changes in atf-check 139 140* Changed atf-check to support negating the status and output checks by 141 prefixing them with not- and added support to specify multiple checkers 142 for stdout and stderr, not only one. 143 144* Added the match output checker to atf-check to look for regular 145 expressions in the stdout and stderr of commands. 146 147* Modified the exit checks in atf-check to support checking for the 148 reception of signals. 149 150Code simplifications and cleanups 151 152* Removed usage messages from test programs to simplify the 153 implementation of every binding by a significant amount. They just now 154 refer the user to the appropriate manual page and do not attempt to wrap 155 lines on terminal boundaries. Test programs are not supposed to be run 156 by users directly so this minor interface regression is not important. 157 158* Removed the atf-format internal utility, which is unused after the 159 change documented above. 160 161* Removed the atf-cleanup internal utility. It has been unused since the 162 test case isolation was moved to atf-run in 0.8 163 164* Splitted the Makefile.am into smaller files for easier maintenance and 165 dropped the use of M4. Only affects users building from the repository 166 sources. 167 168* Intermixed tests with the source files in the source tree to provide 169 them more visibility and easier access. The tests directory is gone from 170 the source tree and tests are now suffixed by _test, not prefixed by t_. 171 172* Simplifications to the atf-c library: removed the io, tcr and ui 173 modules as they had become unnecessary after all simplifications 174 introduced since the 0.8 release. 175 176* Removed the application/X-atf-tcr format introduced in 0.8 release. 177 Tests now print a much simplified format that is easy to parse and nicer 178 to read by end users. As a side effect, the default for test cases is 179 now to print their results to stdout unless otherwise stated by providing 180 the -r flag. 181 182* Removed XML distribution documents and replaced them with plain-text 183 documents. They provided little value and introduced a lot of complexity 184 to the build system. 185 186* Simplified the output of atf-version by not attempting to print a 187 revision number when building form a distfile. Makes the build system 188 easier to maintain. 189 190 191Changes in version 0.9 192********************** 193 194Experimental version released on June 3rd, 2010. 195 196* Added atf-sh, an interpreter to process test programs written using 197 the shell API. This is not really a shell interpreter by itself though: 198 it is just a wrapper around the system shell that eases the loading of 199 the necessary ATF libraries. 200 201* Removed atf-compile in favour of atf-sh. 202 203* Added the use.fs metadata property to test case, which is used to 204 specify which test cases require file system access. This is to 205 highlight dependencies on external resources more clearly and to speed up 206 the execution of test suites by skipping the creation of many unnecessary 207 work directories. 208 209* Fixed test programs to get a sane default value for their source 210 directory. This means that it should not be necessary any more to pass 211 -s when running test programs that do not live in the current directory. 212 213* Defining test case headers became optional. This is trivial to achieve 214 in shell-based tests but a bit ugly in C and C++. In C, use the new 215 ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 216 ATF_TEST_CASE_WITHOUT_HEAD. 217 218 219Changes in version 0.8 220********************** 221 222Experimental version released on May 7th, 2010. 223 224* Test programs no longer run several test cases in a row. The execution 225 of a test program now requires a test case name, and that single test 226 case is executed. To execute several test cases, use the atf-run utility 227 as usual. 228 229* Test programs no longer fork a subprocess to isolate the execution of 230 test cases. They run the test case code in-process, and a crash of the 231 test case will result in a crash of the test program. This is to ease 232 debugging of faulty test cases. 233 234* Test programs no longer isolate their test cases. This means that they 235 will not create temporary directories nor sanitize the environment any 236 more. Yes: running a test case that depends on system state by hand will 237 most likely yield different results depending on where (machine, 238 directory, user environment, etc.) it is run. Isolation has been moved 239 to atf-run. 240 241* Test programs no longer print a cryptic format (application/X-atf-tcs) 242 on a special file channel. They can now print whatever they want on the 243 screen. Because test programs can now only run one test case every time, 244 providing controlled output is not necessary any more. 245 246* Test programs no longer write their status into a special file 247 descriptor. Instead, they create a file with the results, which is later 248 parsed by atf-run. This changes the semantics of the -r flag. 249 250* atf-run has been adjusted to perform the test case isolation. As a 251 result, there is now a single canonical place that implements the 252 isolation of test caes. In previous releases, the three language 253 bindings (C, C++ and shell) had to be kept in sync with each other (read: 254 not a nice thing to do at all). As a side effect of this change, writing 255 bindings for other languages will be much, much easier from now on. 256 257* atf-run forks test programs on a test case basis, instead of on a test 258 program basis as it did before. This is to provide the test case 259 isolation that was before implemented by the test programs themselves. 260 261* Removed the atf-exec tool. This was used to implement test case 262 isolation in atf-sh, but it is now unnecessary. 263 264* It is now optional to define the descr meta-data property. It has been 265 proven to be mostly useless, because test cases often carry a descriptive 266 name of their own. 267 268 269Changes in version 0.7 270********************** 271 272Experimental version released on December 22nd, 2009. 273 274* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 275 will come later. 276 277* Migrated all build-time checks for header files to proper ATF tests. 278 This demonstrates the use of the new feature described above. 279 280* Added an internal API for child process management. 281 282* Converted all plain-text distribution documents to a Docbook canonical 283 version, and include pre-generated plain text and HTML copies in the 284 distribution file. 285 286* Simplified the contents of the Makefile.am by regenerating it from a 287 canonical Makefile.am.m4 source. As a side-effect, some dependency 288 specifications were fixed. 289 290* Migrated all checks from the check target to installcheck, as these 291 require ATF to be installed. 292 293* Fixed sign comparison mismatches triggered by the now-enabled 294 -Wsign-compare. 295 296* Fixed many memory and object leaks. 297 298 299Changes in version 0.6 300********************** 301 302Experimental version released on January 18th, 2009. 303 304* Make atf-exec be able to kill its child process after a certain period 305 of time; this is controlled through the new -t option. 306 307* Change atf-sh to use atf-exec's -t option to control the test case's 308 timeouts, instead of doing it internally. Same behavior as before, but 309 noticeably faster. 310 311* atf-exec's -g option and atf-killpg are gone due to the previous 312 change. 313 314* Added the atf-check(1) tool, a program that executes a given command 315 and checks its exit code against a known value and allows the management 316 of stdout and stderr in multiple ways. This replaces the previous 317 atf_check function in the atf-sh library and exposes this functionality 318 to both atf-c and atf-c++. 319 320* Added the ATF_REQUIRE family of macros to the C interface. These help 321 in checking for fatal test conditions. The old ATF_CHECK macros now 322 perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 323 can now continue its execution and the failures will not be reported 324 until the end of the whole run. 325 326* Extended the amount of ATF_CHECK_* C macros with new ones to provide 327 more features to the developer. These also have their corresponding 328 counterparts in the ATF_REQUIRE_* family. The new macros (listing the 329 suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 330 _STREQ_MSG. 331 332 333Changes in version 0.5 334********************** 335 336Experimental version released on May 1st, 2008. 337 338* Clauses 3 and 4 of the BSD license used by the project were dropped. 339 All the code is now under a 2-clause BSD license compatible with the GNU 340 General Public License (GPL). 341 342* Added a C-only binding so that binary test programs do not need to be 343 tied to C++ at all. This binding is now known as the atf-c library. 344 345* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 346 347* Renamed the POSIX shell binding to atf-sh for consistency with the new 348 atf-c and atf-c++. 349 350* Added a -w flag to test programs through which it is possible to 351 specify the work directory to be used. This was possible in prior 352 releases by defining the workdir configuration variable (-v workdir=...), 353 but was a conceptually incorrect mechanism. 354 355* Test programs now preserve the execution order of test cases when they 356 are given in the command line. Even those mentioned more than once are 357 executed multiple times to comply with the user's requests. 358 359 360Changes in version 0.4 361********************** 362 363Experimental version released on February 4th, 2008. 364 365* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 366 C++ and POSIX shell interfaces used to write test programs. 367 368* Added a pkg-config file, useful to get the flags to build against the 369 C++ library or to easily detect the presence of ATF. 370 371* Added a way for test cases to require a specific architecture and/or 372 machine type through the new 'require.arch' and 'require.machine' 373 meta-data properties, respectively. 374 375* Added the 'timeout' property to test cases, useful to set an 376 upper-bound limit for the test's run time and thus prevent global test 377 program stalls due to the test case's misbehavior. 378 379* Added the atf-exec(1) internal utility, used to execute a command 380 after changing the process group it belongs to. 381 382* Added the atf-killpg(1) internal utility, used to kill process groups. 383 384* Multiple portability fixes. Of special interest, full support for 385 SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 386 C++ compiler. 387 388* Fixed a serious bug that prevented atf-run(1) from working at all 389 under Fedora 8 x86_64. Due to the nature of the bug, other platforms 390 were likely affected too. 391 392 393Changes in version 0.3 394********************** 395 396Experimental version released on November 11th, 2007. 397 398* Added XML output support to atf-report. This is accompanied by a DTD 399 for the format's structure and sample XSLT/CSS files to post-process this 400 output and convert it to a plain HTML report. 401 402* Changed atf-run to add system information to the report it generates. 403 This is currently used by atf-report's XML output only, and is later 404 printed in the HTML reports in a nice and useful summary table. The user 405 and system administrator are allowed to tune this feature by means of 406 hooks. 407 408* Removed the test cases' 'isolated' property. This was intended to 409 avoid touching the file system at all when running the related test case, 410 but this has not been true for a long while: some control files are 411 unconditionally required for several purposes, and we cannot easily get 412 rid of them. This way we remove several critical and delicate pieces of 413 code. 414 415* Improved atf-report's CSV output format to include information about 416 test programs too. 417 418* Fixed the tests that used atf-compile to not require this tool as a 419 helper. Avoids systems without build-time utilities to skip many tests 420 that could otherwise be run. (E.g. NetBSD without the comp.tgz set 421 installed.) 422 423* Many general cleanups: Fixed many pieces of code marked as ugly and/or 424 incomplete. 425 426 427Changes in version 0.2 428********************** 429 430Experimental version released on September 20th, 2007. 431 432* Test cases now get a known umask on entry. 433 434* atf-run now detects many unexpected failures caused by test programs and 435 reports them as bogus tests. atf-report is able to handle these new 436 errors and nicely reports them to the user. 437 438* All the data formats read and written by the tools have been 439 documented and cleaned up. These include those grammars that define how 440 the different components communicate with each other as well as the 441 format of files written by the developers and users: the Atffiles and the 442 configuration files. 443 444* Added the atf-version tool, a utility that displays information about 445 the currently installed version of ATF. 446 447* Test cases can now define an optional cleanup routine to undo their 448 actions regardless of their exit status. 449 450* atf-report now summarizes the list of failed (bogus) test programs 451 when using the ticker output format. 452 453* Test programs now capture some termination signals and clean up any 454 temporary files before exiting the program. 455 456* Multiple bug fixes and improvements all around. 457 458 459Changes in version 0.1 460********************** 461 462Experimental version released on August 20th, 2007. 463 464* First public version. This was released coinciding with the end of the 465 Google Summer of Code 2007 program. 466 467 468=========================================================================== 469vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 470