1.\" $OpenBSD: bsd.regress.mk.5,v 1.17 2018/10/05 11:15:29 bluhm Exp $ 2.\" 3.\" Copyright (c) 2002 Anil Madhavapeddy 4.\" Copyright (c) 2000 Marc Espie 5.\" 6.\" All rights reserved. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer. 13.\" 2. Redistributions in binary form must reproduce the above copyright 14.\" notice, this list of conditions and the following disclaimer in the 15.\" documentation and/or other materials provided with the distribution. 16.\" 17.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 18.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 19.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 20.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 21.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 22.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 23.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 24.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 25.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 26.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27.\" 28.Dd $Mdocdate: October 5 2018 $ 29.Dt BSD.REGRESS.MK 5 30.Os 31.Sh NAME 32.Nm bsd.regress.mk 33.Nd regression test master Makefile fragment 34.Sh SYNOPSIS 35.Fd .include <bsd.regress.mk> 36.Sh DESCRIPTION 37.Nm 38holds the standard routines used by the source regression tests. 39Some variables and targets are for its internal use only. 40The rest are documented here. 41.Pp 42Since this file also includes 43.Nm bsd.prog.mk , 44all of the usual 45.Ox 46Makefile variables may be used to build the regression 47test programs. 48.Sh TARGETS 49.Bl -tag -width regress 50.It Cm depend 51Build any dependencies required to carry out the current set 52of regression tests. 53.It Cm regress 54Executes all of the regression targets defined in the Makefile. 55If one of the tests fails, the line 56.Qq FAILED 57is printed to the standard output. 58By default, execution continues with the next test and, after running 59all the tests, 60.Sy make Cm regress 61exits with an exit status of 0 even if some of the tests failed. 62.It Cm run-regress-* 63Runs an individual regression test. 64If the exit status of the program indicates an error or timeout, 65then a failure is logged, otherwise the test is marked as a success. 66.El 67.Sh VARIABLES 68.Bl -tag -width Ds 69.It Ev REGRESS_CLEANUP 70If this variable is set, it contains a target that is executed 71after all regress targets. 72.It Ev REGRESS_FAIL_EARLY 73If this variable is set to anything but 74.Dq no , 75the 76.Cm regress 77target will abort as soon as a test fails. 78.It Ev REGRESS_LOG 79Points to the fully-qualified path of a file to which regression 80results are appended. 81Defaults to 82.Pa /dev/null . 83.It Ev REGRESS_MAXTIME 84Maximum limit of CPU seconds to spend on the regression test. 85Exceeding this time will result in a failure being logged. 86.It Ev REGRESS_ROOT_TARGETS 87Targets for which root access is required to run the test. 88The 89.Ev SUDO 90variable is invoked for these targets. 91See also 92.Ev SUDO . 93.It Ev REGRESS_SETUP 94If this variable is set, it contains a target that is executed 95before each regress target. 96Regress targets depend on this one to enforce setup. 97.It Ev REGRESS_SETUP_ONCE 98If this variable is set, it contains a target that is executed 99once before all regress targets. 100Regress targets depend on a stamp file generated during setup. 101.It Ev REGRESS_SKIP_SLOW 102If this variable is not empty, skip over all the regression 103tests which have been marked as being 'slow' using the 104.Ev REGRESS_SLOW_TARGETS 105variable. 106.It Ev REGRESS_SLOW_TARGETS 107Targets which are defined as 'slow'. 108All of these tests can be skipped by setting the 109.Ev REGRESS_SKIP_SLOW 110variable. 111.It Ev REGRESS_TARGETS 112Targets which are invoked to run the set of regression tests 113for this Makefile. 114Defaults to 115.Cm run-regress-${PROG} . 116.It Ev SUDO 117Location of a command used to switch to root for certain 118test targets which require it. 119See 120.Xr doas 1 . 121.El 122.Pp 123Some variables are intended to be set at runtime in the environment 124or in 125.Xr mk.conf 5 , 126but not in the regress Makefile itself. 127.Sh GUIDELINES 128If an individual test passes, 129.Sy make Ar testname 130should return with an exit status of 0. 131If it fails, it should return with a non-zero exit status. 132.Pp 133If a test cannot be executed because a package is not installed or 134some environment variable is not set, 135.Sy make Ar testname 136should print 137.Qq SKIPPED 138to stdout and exit with status 0. 139To skip everything, implement the 140.Cm regress 141target with a command that prints 142.Qq SKIPPED . 143.Pp 144Some tests may require a special setup on the test machine that has 145to be done manually before testing. 146This requirement has to be documented in the Makefile or in a 147.Pa README 148file. 149The test should find out whether the setup exists before running 150and print 151.Qq SKIPPED 152and exit if it is missing. 153.Pp 154Tests should not fail because an intended feature has not been 155implemented yet. 156To avoid such false failures, a test should show the reason, print 157.Qq DISABLED 158to stdout and exit with status 0. 159.Pp 160Tests must be able to run with an 161.Pa obj 162directory. 163In case the test is not linked to the build or the tester forgot 164to run 165.Sy make Cm obj 166before, this directory or symlink may not exist. 167Then the test should run anyway. 168.Pp 169Tests are executed with 170.Sy make Cm regress , 171but running 172.Sy make Cm all 173or 174.Sy make 175should have the same effect. 176Tests must be runnable by root, and may also succeed when run as a 177regular user. 178Tests must not assume they have a controlling tty, 179to allow them to be run by 180.Xr cron 8 . 181An individual regress test may create a pseudo tty if it needs one. 182.Pp 183Tests should use the binaries installed and the kernel running on 184the local system. 185They may use environment variables to test alternative binaries or 186remote kernels running on other machines. 187In some cases a test may need binaries or libraries or object files 188to be present in 189.Pa /usr/obj/ 190that exist only after 191.Sy make Cm build 192was run in 193.Pa /usr/src/ . 194The test must not assume that they have already been built, but 195should run 196.Sy make 197in the appropriate source directory as a dependency. 198For missing generated source or header files a target called 199.Sy make Cm generated 200is common. 201The 202.Pa /usr/src/ 203tree can be found with a relative path or with the 204.Ev BSDSRCDIR 205variable. 206.Sh SEE ALSO 207.Xr bsd.port.mk 5 208.Sh HISTORY 209The regression system originally came from 210.Nx , 211with many tests added by 212.Ox 213since. 214The current Makefile framework was written by Artur Grabowski 215and Marc Espie for 216.Ox 3.1 . 217.Sh BUGS AND LIMITATIONS 218The build system is unable to distinguish between timeouts due to 219.Ev REGRESS_MAXTIME 220being exceeded, or a genuine failure occurring. 221