1.\" Copyright 2012 Google Inc. 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions are 6.\" met: 7.\" 8.\" * Redistributions of source code must retain the above copyright 9.\" notice, this list of conditions and the following disclaimer. 10.\" * Redistributions in binary form must reproduce the above copyright 11.\" notice, this list of conditions and the following disclaimer in the 12.\" documentation and/or other materials provided with the distribution. 13.\" * Neither the name of Google Inc. nor the names of its contributors 14.\" may be used to endorse or promote products derived from this software 15.\" without specific prior written permission. 16.\" 17.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 18.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 19.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 20.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 21.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 22.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 23.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 24.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 25.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28.Dd September 9, 2012 29.Dt KYUA-ATF-INTERFACE 1 30.Os 31.Sh NAME 32.Nm atf-interface 33.Nd Description of the ATF test program interface 34.Sh DESCRIPTION 35The interface of ATF test programs is the interface of the test 36programs linked to the 37.Nm atf-c , 38.Nm atf-c++ 39and 40.Nm atf-sh 41libraries provided by ATF. 42.Pp 43The ATF interface can be understood as the mechanisms used by test programs 44to communicate with the runtime engine as well as the assumptions that test 45programs and test cases can make while running. 46.Pp 47A test case is the most basic part of a test suite. A test case is 48supposed to reproduce one, and only one, scenario. For example: if the 49item under test was a function, the test case would provide a single set of 50input parameters to the function and check its output; If the item under 51test was a binary, the test case would provide a single set of arguments to 52the program and check its behavior. 53.Ss Test case parts 54Test cases have three parts: 55.Bl -tag -width cleanupXX 56.It Head 57Programmatically defines metadata properties. The head must not perform 58any other thing than defining such properties. In particular, no testing 59whatsoever can happen in the head. (Ideally the definition of metadata 60properties would not happen programmatically.) 61.It Body 62The actual test case which performs any desired testing and reports a 63result. The body is executed by the runtime engine in a deterministic way; 64see the isolation section below. 65.It Cleanup 66An optional cleanup routine. Note that the runtime engine will attempt to 67clean up the work directory automatically, so this routine should only be 68provided in cases where the test modifies system-wide state not known by 69the runtime engine. The cleanup part is executed in the same directory as 70the body. However, the body and the cleanup parts 71.Em do not share the same process space ; 72the only way to pass data around from the body to the cleanup is by means 73of files in the work directory. 74.El 75.Ss Metadata properties 76The following test case metadata properties must be exported in the test 77case list for every test case: 78.Bl -tag -width XX 79.It Va ident 80Single-word string. The name of the test case. Must be unique within the 81test program. 82.El 83.Pp 84The following test case metadata properties may be exported in the 85test case list for every test case: 86.Bl -tag -width XX 87.It Va descr 88Multi-word string. A textual description for the test case. Usually, 89providing a descriptive identifier is better than providing a textual 90description. 91.It Va has.cleanup 92Boolean. Whether the test case defines a cleanup routine or not. 93.It Va require.arch 94Whitespace separated list of the architectures required by the test case. 95If defined, the test case is skipped unless the host architecture matches 96any of the values defined in this property. 97.It Va require.config 98Whitespace separated list of configuration variable names. The list of 99configuration variables that must be defined. The test is skipped if any 100of these is missing. 101.It Va require.files 102Whitespace separated list of absolute paths to installed files. If any of 103these files is not found, the test case is skipped. 104.It Va require.machine 105Whitespace separated list of the machine types required by the test case. 106If defined, the test case is skipped unless the host machine type matches 107any of the values defined in this property. 108.It Va require.progs 109Whitespace separated list of program names (either absolute names or base 110names). If any of these programs is not found, the test case is skipped. 111.It Va require.user 112Either 113.Sq root 114or 115.Sq unprivileged . 116If 117.Sq root , 118the test case must be run as the superuser or otherwise it is skipped. If 119.Sq unprivileged , 120the test case must be run as an unprivileged user or else it is skipped. 121.It Va timeout 122Integer. The amount of seconds the test case can run for before it is 123killed by the runtime engine. 124.El 125.Ss Configuration properties 126The following properties may be defined by the runtime engine and are 127propagated to the test cases: 128.Bl -tag -width XX 129.It Va unprivileged-user 130String, optional. Specifies the name of the user under which tests that 131set 132.Sq require.user=unprivileged 133are executed. 134.El 135.Ss Results 136A test case must always report a result by creating the results file 137specified through the 138.Fl r 139flag. For convenience when running test cases without the runtime engine, 140this file may point to 141.Pa /dev/stdout 142or 143.Pa /dev/stderr 144in which case the file must not be created (because the creation will 145fail). 146.Pp 147Aside from creating the results file, the process in which the test case 148runs must terminate in particular ways for the test result to be considered 149valid. 150.Pp 151If the test case fails to create the test result, if the test result is 152created but contains an invalid syntax, or if the termination status of the 153process does not match the requirements of the test result, the runtime 154engine marks the test case as 155.Sq broken . 156Note that the 157.Sq broken 158state is decided by the runtime engine; a test case cannot report itself as 159.Sq broken . 160.Pp 161The general syntax for the results file is as follows: 162.Bd -literal -offset indent 163<status>[[(int)]: reason] 164.Ed 165.Pp 166The following results are allowed: 167.Bl -tag -width XX 168.It expected_death 169The process is expected to terminate either due to a clean call to 170.Xr exit 3 171or due to the reception of a signal. The contents of the file are 172.Sq expected_death: <reason>\\n . 173Example: 174.Sq expected_death: Calling libdofoo breaks due to bug xyz . 175.It expected_exit 176The process is expected to terminate cleanly. The contents of the file are 177.Sq expected_exit: <reason> 178if the exit code is irrelevant or 179.Sq expected_exit(<exitcode>): <reason> 180if the process must terminate with a given exit code. Examples: 181.Sq expected_exit: Calling bar exits but it should not 182or 183.Sq expected_exit(123): Calling bar exits with an unexpected code . 184.It expected_failure 185The process must exit cleanly with an 186.Va EXIT_SUCCESS 187exit code. The contents of the file are 188.Sq expected_failure: <reason>\\n 189Example: 190.Sq expected_failure: 2 + 2 = 3 . 191.It expected_signal 192The process is expected to terminate due to the reception of a signal. The 193contents of the file are 194.Sq expected_signal: <reason> 195if the signal number is irrelevant or 196.Sq expected_signal(<signalno>): <reason> 197if the process must terminate due to a particular signal. Examples: 198.Sq expected_signal: Calling bar crashes 199or 200.Sq expected_signal(1): Calling bar kills ourselves due to unhandled SIGHUP . 201.It expected_timeout 202The process is expected to hang for longer than its 203.Va timeout 204metadata property. Only the runtime engine can control this situation 205because the runtime engine is the one implementing the timeout 206functionality. 207.It failed 208The process must exit cleanly with an 209.Va EXIT_FAILURE 210exit code. The contents of the file are 211.Sq failed: <reason>\\n . 212Example: 213.Sq failed: Failed on purpose\\n . 214.It passed 215The process must exit cleanly with an 216.Va EXIT_SUCCESS 217exit code. The contents of the file are 218.Sq passed\\n . 219.It skipped 220The process must exit cleanly with an 221.Va EXIT_SUCCESS 222exit code. The contents of the file are 223.Sq skipped: <reason>\\n . 224Example: 225.Sq skipped: Skipped because the foo is not present\\n . 226.El 227.Ss Isolation 228The runtime engine attempts to isolate test cases from other test cases in 229the same test program and from the rest of the system by performing what is 230called 231.Em test case isolation . 232.Pp 233Whenever the user runs a test program binary by hand (i.e. not through 234.Xr kyua 1 ) , 235the test program will print a warning message stating that test case 236isolation does not work and therefore the program may cause side-effects 237and/or report invalid values. 238.Pp 239The runtime engine must set the 240.Va __RUNNING_INSIDE_ATF_RUN 241environment variable to the magic value 242.Sq internal-yes-value 243to tell the test programs that they are being run with isolation enabled. 244.Pp 245The test case isolation performs the following: 246.Bl -tag -width XX 247.It Process space 248Each test case body and cleanup routines are executed in independent 249processes. Corollary: the test case can do whatever it wants to the 250current process (such as modifying global variables) without having to undo 251such changes. 252.It Process group 253The test case body and cleanup are executed in their own process groups. 254Should they spawn any children, such children should maintain the same 255process group. This is done to allow the runtime engine to kill the whole 256process subtree once the test case finishes (or if the test case hangs). 257.It Work directory 258The test case body and its cleanup (if any) are executed in a temporary 259directory automatically created by the runtime engine. This temporary 260directory is shared among the body and cleanup parts of a single test case 261but is completely separate from the temporary directories of other tests. 262Corollary: the test case body and cleanup routines can write to their 263current directory without bothering to clean any files and/or directories 264they create. The runtime engine takes care to recursively delete the 265temporary directories after the execution of a test case. Any file systems 266mounted within the temporary directory will be unmounted if possible. 267.It Home directory 268The 269.Va HOME 270environment variable is set to the absolute path of the work directory. 271.It Umask 272The value of the umask is set to 0022. 273.It Environment 274The 275.Va LANG , 276.Va LC_ALL , 277.Va LC_COLLATE , 278.Va LC_CTYPE , 279.Va LC_MESSAGES , 280.Va LC_MONETARY , 281.Va LC_NUMERIC 282and 283.Va LC_TIME 284variables are unset. The 285.Va TZ 286variable is set to 287.Sq UTC . 288.It Process limits 289The maximum soft core size limit is raised to its corresponding hard limit. 290This is a simple, best-effort attempt at allowing test cases to dump core 291for further diagnostic purposes. 292.El 293.Ss Test programs 294A test program is, simply put, a collection of related test cases. The 295test program can be seen as a command-line dispatcher for the test cases. 296A test program must provide one or more test cases. If it does not contain 297any test case, the runtime system will report it as invalid. 298.Pp 299Test programs expose their list of test cases in a machine parseable 300format. The runtime engine obtains the list of test cases to know what 301tests to run and to know how to set up the environment of each test prior 302execution. The test program must not do any test when asked to dump its 303test case list. 304.Pp 305The generic syntax to obtain the list of test cases included in a test 306program is: 307.Bd -literal -offset indent 308<test-program> -l 309.Ed 310.Pp 311The list of test cases follows the following format: 312.Bd -literal -offset indent 313LIST ::= HEADER NEWLINE TEST_CASES 314 315HEADER ::= 'Content-Type: application/X-atf-tp; version="1"' 316NEWLINE ::= '\\n' 317TEST_CASES ::= TEST_CASE | TEST_CASE NEWLINE TEST_CASES 318 319TEST_CASE ::= IDENT_PROPERTY PROPERTIES 320IDENT_PROPERTY ::= 'ident' DELIM STRING NEWLINE 321DELIM ::= ': ' 322 323PROPERTIES ::= PROPERTY | PROPERTY PROPERTIES 324PROPERTY ::= PROPERTY_NAME DELIM STRING NEWLINE 325PROPERTY_NAME ::= (see below) 326.Ed 327.Pp 328An example: 329.Bd -literal -offset indent 330Content-Type: application/X-atf-tp; version="1" 331 332ident: addition 333descr: Tests that the addition function works 334 335ident: subtraction 336descr: Tests that the subtraction function works 337 338ident: remove 339descr: Tests removing files 340require.root: true 341timeout: 50 342has.cleanup: true 343.Ed 344.Pp 345The syntax to run a test case body part is: 346.Bd -literal -offset indent 347<test-program> [-r resfile] [-s srcdir] [-v var=value]* <test-case>[:body] 348.Ed 349.Pp 350This must run the test case body 351.Dq as is , 352without any attempt of isolating it from the rest of the system. It is the 353responsibility of the runtime engine to do such isolation. 354.Pp 355The runtime engine always passes the path of a nonexistent file to 356.Fl r , 357which must be created by the test case; and always passes an absolute path 358to the 359.Fl s 360flag pointing to the directory containing the test program executable. 361.Pp 362The runtime engine shall pass any configuration variables it wants through 363the 364.Fl v 365flag, and these can be later inspected by the test case at will. 366.Pp 367A note to users: if you run the test case by hand (not through 368.Xr kyua 1 nor 369.Xr atf-run 1 ) 370from the command line, none of the isolation features described in the 371isolation section apply. This means that the test case can (and probably 372will) write to the current directory and leave garbage behind. Also, given 373that the test case is executed without e.g. clearing the environment, the 374results of the test case may differ from those obtained when running the 375test case inside the runtime engine. 376.Em Only use this for debugging purposes 377(i.e. to run the test case code under GDB). 378.Pp 379The syntax to run a test case cleanup part is: 380.Bd -literal -offset indent 381<test-program> [-s srcdir] [-v var=value]* <test-case>:cleanup 382.Ed 383.Pp 384This can only be performed if and only if the test case sets the 385.Va has.cleanup 386property to true. Otherwise the behavior of executing the cleanup part is 387undefined. 388.Pp 389The same rules for 390.Fl s 391and 392.Fl v 393apply as to when running the body. 394.Pp 395The cleanup part must be executed in the same directory as the body but in 396a separate process space. The only way for test cases to transfer state 397(if any) from the body to the cleanup routine is by means of files in the 398current directory. 399.Pp 400The cleanup part does not have to worry about deleting temporary files 401created in the current directory. The runtime engine does this 402automatically. 403.Sh SEE ALSO 404.Xr kyua-test 1 , 405.Xr kyuafile 5 406