1.\" Copyright 2012 Google Inc. 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions are 6.\" met: 7.\" 8.\" * Redistributions of source code must retain the above copyright 9.\" notice, this list of conditions and the following disclaimer. 10.\" * Redistributions in binary form must reproduce the above copyright 11.\" notice, this list of conditions and the following disclaimer in the 12.\" documentation and/or other materials provided with the distribution. 13.\" * Neither the name of Google Inc. nor the names of its contributors 14.\" may be used to endorse or promote products derived from this software 15.\" without specific prior written permission. 16.\" 17.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 18.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 19.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 20.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 21.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 22.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 23.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 24.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 25.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28.Dd February 9, 2013 29.Dt KYUAFILE 5 30.Os 31.Sh NAME 32.Nm Kyuafile 33.Nd Test suite description files 34.Sh SYNOPSIS 35.Fn atf_test_program "string name" "[string test_suite]" "[string metadata]" 36.Fn include "string path" 37.Fn plain_test_program "string name" "[string test_suite]" "[string metadata]" 38.Fn syntax "int version" 39.Fn test_suite "string name" 40.Sh DESCRIPTION 41A test suite is a collection of test programs and is represented by a 42hierarchical layout of test binaries on the file system. Any subtree of 43the file system can represent a test suite, provided that it includes one 44or more 45.Nm Ns s , 46which are the test suite definition files. 47.Pp 48A 49.Nm 50is a Lua script whose purpose is to describe the structure of the test 51suite it belongs to. To do so, the script has access to a collection of 52special functions provided by 53.Xr kyua 1 . 54.Pp 55A typical 56.Nm 57will look similar to this: 58.Bd -literal -offset indent 59syntax(2) 60 61test_suite('first') 62 63-- Declare the test programs that are in this directory. 64atf_test_program{name='foo_test'} 65atf_test_program{name='bar_test', test_suite='second'} 66plain_test_program{name='legacy_test'} 67plain_test_program{name='legacy2_test', allowed_architectures='amd64 i386', 68 required_files='/bin/ls', timeout=30} 69 70-- Recurse into any subdirectories that may have other tests. 71include('dir1/Kyuafile') 72include('dir2/Kyuafile') 73.Ed 74.Ss File versioning 75Every 76.Nm 77file starts with a call to 78.Fn syntax "int version" . 79This call determines the specific schema used by the file so that future 80backwards-incompatible modifications to the file can be introduced. 81.Pp 82Any new 83.Nm 84file should set 85.Fa version 86to 87.Sq 2 . 88.Ss Test suite definition 89Every 90.Nm 91should define the name of the test suite it belongs to by calling the 92.Fn test_suite 93function at the very beginning. 94Individual test programs can override this value in their definition, but 95the most common style is to list a single test suite name for the whole 96file. 97.Pp 98The purpose of the test suite name definition is to tell 99.Xr kyua 1 100scoping for the run-time configuration variables that these programs 101accept. 102.Ss Test program registration 103A 104.Nm 105can register test programs by means of a variety of 106.Fn *_test_program 107functions, all of which take the name of a test program and a set of 108optional metadata properties that describe such test program. 109.Pp 110The test programs to be registered must live in the current directory; in 111other words, the various 112.Fn *_test_program 113calls cannot reference test programs in other directories. 114The rationale for this is to force all 115.Nm 116files to be self-contained, and to simplify their internal representation. 117.Pp 118ATF-based test programs (those that implement the 119.Xr kyua-atf-interface 7 120interface) can be registered with the 121.Fn atf_test_program 122table constructor. This function takes the 123.Fa name 124of the test program, an optional 125.Fa test_suite 126name that overrides the global test suite name and a collection of optional 127metadata settings for all the test cases in the test program. Any metadata 128properties defined by the test cases themselves override the metadata values 129defined here. 130.Pp 131Plain test programs (those that implement the 132.Xr kyua-plain-interface 7 133interface) can be registered with the 134.Fn plain_test_program 135table constructor. This function takes the 136.Fa name 137of the test program an optional 138.Fa test_suite 139name that overrides the global test suite name and a collection of optional 140metadata settings for the test program. 141.Pp 142Please see 143.Xr kyua-tester-list 5 144for the list of metadata properties that can be given to test programs. 145All the properties that can be given to a test case can also be given to a test 146program. 147.Ss Recursion 148To reference test programs in another subdirectory, a different 149.Nm 150must be created in that directory and it must be included into the original 151.Nm 152by means of the 153.Fn include 154function. 155.Pp 156Note that each file is processed in its own Lua environment: there is no 157mechanism to pass state from one file to the other. The reason for this is 158that there is no such thing as a 159.Dq top-level 160.Nm 161in a test suite: the user has to be able to run the test suite from any 162directory in a given hierarchy, and this execution must not depend on files 163that live in parent directories. 164.Ss Top-level Kyuafile 165Every system has a top directory into which test suites get installed. The 166default is 167.Pa __TESTSDIR__ . 168Within this directory live test suites, each of which is in an independent 169subdirectory. Each subdirectory can be provided separately by independent 170third-party packages. 171.Pp 172Kyua allows running all the installed test suites at once in order to 173provide comprehensive cross-component reports. In order to do this, there 174is a special file in the top directory that knows how to inspect the 175subdirectories in search for other Kyuafiles and include them. 176.Pp 177The 178.Sx FILES 179section includes more details on where this file lives. 180.Sh FILES 181.Bl -tag -width XX 182.It Pa __TESTSDIR__/Kyuafile . 183Top-level 184.Nm 185for the current system. 186.It Pa __EGDIR__/Kyuafile.top . 187Sample file to serve as a top-level 188.Nm . 189.El 190.Sh SEE ALSO 191.Xr kyua 1 192