1.\" $OpenBSD: bsd.regress.mk.5,v 1.25 2023/09/25 05:43:22 claudio Exp $ 2.\" 3.\" Copyright (c) 2002 Anil Madhavapeddy 4.\" Copyright (c) 2000 Marc Espie 5.\" 6.\" All rights reserved. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer. 13.\" 2. Redistributions in binary form must reproduce the above copyright 14.\" notice, this list of conditions and the following disclaimer in the 15.\" documentation and/or other materials provided with the distribution. 16.\" 17.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 18.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 19.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 20.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 21.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 22.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 23.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 24.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 25.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 26.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27.\" 28.Dd $Mdocdate: September 25 2023 $ 29.Dt BSD.REGRESS.MK 5 30.Os 31.Sh NAME 32.Nm bsd.regress.mk 33.Nd regression test master Makefile fragment 34.Sh SYNOPSIS 35.Fd .include <bsd.regress.mk> 36.Sh DESCRIPTION 37.Nm 38holds the standard routines used by the source regression tests. 39Some variables and targets are for its internal use only. 40The rest are documented here. 41.Pp 42Since this file also includes 43.Nm bsd.prog.mk , 44all of the usual 45.Ox 46Makefile variables may be used to build the regression 47test programs. 48.Sh TARGETS 49.Bl -tag -width regress 50.It Cm depend 51Build any dependencies required to carry out the current set 52of regression tests. 53.It Cm regress 54Executes all of the regression targets defined in the Makefile. 55If one of the tests fails, the line 56.Qq FAILED 57is printed to the standard output. 58By default, execution continues with the next test and, after running 59all the tests, 60.Sy make Cm regress 61exits with an exit status of 0 even if some of the tests failed. 62.It Cm run-regress-* 63Runs an individual regression test. 64If the exit status of the program indicates an error or timeout, 65then a failure is logged, otherwise the test is marked as a success. 66.El 67.Sh VARIABLES 68.Bl -tag -width Ds 69.It Ev REGRESS_CLEANUP 70If this variable is set, it contains a target that is executed 71after all regress targets. 72.It Ev REGRESS_EXPECTED_FAILURES 73Targets which are expected to fail. 74Specifically: sometimes tests are added for features that should be 75supported eventually, but are somewhat complicated to fix right away. 76.It Ev REGRESS_FAIL_EARLY 77If this variable is set to anything but 78.Dq no , 79the 80.Cm regress 81target will abort as soon as a test fails. 82Defaults to 83.Dq yes 84unless 85.Ev REGRESS_LOG 86is set. 87.It Ev REGRESS_LOG 88Points to the fully-qualified path of a file to which regression 89results are appended. 90Defaults to 91.Pa /dev/null . 92If set to any other path, 93.Ev REGRESS_FAIL_EARLY 94defaults to 95.Dq no . 96.It Ev REGRESS_ROOT_TARGETS 97Targets for which root access is required to run the test. 98The 99.Ev SUDO 100variable should be used in the test parts that require root. 101See also 102.Ev SUDO . 103.It Ev REGRESS_SETUP 104If this variable is set, it contains a target that is executed 105before each regress target. 106Regress targets depend on this one to enforce setup. 107.It Ev REGRESS_SETUP_ONCE 108If this variable is set, it contains a target that is executed 109once before all regress targets. 110Regress targets depend on a stamp file generated during setup. 111.It Ev REGRESS_SKIP_SLOW 112If this variable is not empty, skip over all the regression 113tests which have been marked as being 'slow' using the 114.Ev REGRESS_SLOW_TARGETS 115variable. 116.It Ev REGRESS_SKIP_TARGETS 117Targets which are unconditionally skipped. 118.It Ev REGRESS_SLOW_TARGETS 119Targets which are defined as 'slow'. 120All of these tests can be skipped by setting the 121.Ev REGRESS_SKIP_SLOW 122variable. 123.It Ev REGRESS_TARGETS 124Targets which are invoked to run the set of regression tests 125for this Makefile. 126Defaults to 127.Cm run-regress-${PROG} . 128.It Ev SUDO 129Location of a command used to switch to root for certain 130test targets which require it. 131See 132.Xr doas 1 . 133.El 134.Pp 135Some variables are intended to be set at runtime in the environment 136or in 137.Xr mk.conf 5 , 138but not in the regress Makefile itself. 139.Sh GUIDELINES 140If an individual test passes, 141.Sy make Ar testname 142should return with an exit status of 0. 143If it fails, it should return with a non-zero exit status. 144.Pp 145If a test cannot be executed because a package is not installed or 146some environment variable is not set, 147.Sy make Ar testname 148should print 149.Qq SKIPPED 150to stdout and exit with status 0. 151To skip everything, implement the 152.Cm regress 153target with a command that prints 154.Qq SKIPPED . 155.Pp 156Some tests may require a special setup on the test machine that has 157to be done manually before testing. 158This requirement has to be documented in the Makefile or in a 159.Pa README 160file. 161The test should find out whether the setup exists before running 162and print 163.Qq SKIPPED 164and exit if it is missing. 165.Pp 166Tests should not fail because an intended feature has not been 167implemented yet. 168To avoid such false failures, a test should show the reason, print 169.Qq DISABLED 170to stdout and exit with status 0. 171.Pp 172Tests must be able to run with an 173.Pa obj 174directory. 175In case the test is not linked to the build or the tester forgot 176to run 177.Sy make Cm obj 178before, this directory or symlink may not exist. 179Then the test should run anyway. 180.Pp 181Tests are executed with 182.Sy make Cm regress , 183but running 184.Sy make Cm all 185or 186.Sy make 187should have the same effect. 188Tests must be runnable by root, and may also succeed when run as a 189regular user. 190Tests must not assume they have a controlling tty, 191to allow them to be run by 192.Xr cron 8 . 193An individual regress test may create a pseudo tty if it needs one. 194.Pp 195Tests should use the binaries installed and the kernel running on 196the local system. 197They may use environment variables to test alternative binaries or 198remote kernels running on other machines. 199In some cases a test may need binaries or libraries or object files 200to be present in 201.Pa /usr/obj/ 202that exist only after 203.Sy make Cm build 204was run in 205.Pa /usr/src/ . 206The test must not assume that they have already been built, but 207should run 208.Sy make 209in the appropriate source directory as a dependency. 210For missing generated source or header files a target called 211.Sy make Cm generated 212is common. 213The 214.Pa /usr/src/ 215tree can be found with a relative path or with the 216.Ev BSDSRCDIR 217variable. 218.Pp 219Tests should generally not set or modify 220.Ev MALLOC_OPTIONS 221unless it is testing specific behaviour that depends on a particular 222flag or combination of flags. 223When running tests, the appropriate sysctl 224.Pq vm.malloc_conf 225could be used instead. 226.Sh SEE ALSO 227.Xr bsd.port.mk 5 228.Sh HISTORY 229The regression system originally came from 230.Nx , 231with many tests added by 232.Ox 233since. 234The current Makefile framework was written by Artur Grabowski 235and Marc Espie for 236.Ox 3.1 . 237