1*86d7f5d3SJohn Marino# $FreeBSD: src/tools/regression/usr.bin/regress.m4,v 1.5 2004/11/11 19:47:54 nik Exp $ 2*86d7f5d3SJohn Marino 3*86d7f5d3SJohn Marinodnl A library of routines for doing regression tests for userland utilities. 4*86d7f5d3SJohn Marino 5*86d7f5d3SJohn Marinodnl Start up. We initialise the exit status to 0 (no failure) and change 6*86d7f5d3SJohn Marinodnl into the directory specified by our first argument, which is the 7*86d7f5d3SJohn Marinodnl directory to run the tests inside. 8*86d7f5d3SJohn Marinodefine(`REGRESSION_START', 9*86d7f5d3SJohn MarinoTESTDIR=$1 10*86d7f5d3SJohn Marinoif [ -z "$TESTDIR" ]; then 11*86d7f5d3SJohn Marino TESTDIR=. 12*86d7f5d3SJohn Marinofi 13*86d7f5d3SJohn Marinocd $TESTDIR 14*86d7f5d3SJohn Marino 15*86d7f5d3SJohn MarinoSTATUS=0) 16*86d7f5d3SJohn Marino 17*86d7f5d3SJohn Marinodnl Check $? to see if we passed or failed. The first parameter is the test 18*86d7f5d3SJohn Marinodnl which passed or failed. It may be nil. 19*86d7f5d3SJohn Marinodefine(`REGRESSION_PASSFAIL', 20*86d7f5d3SJohn Marinoif [ $? -eq 0 ]; then 21*86d7f5d3SJohn Marino echo "ok - $1 # Test detected no regression. (in $TESTDIR)" 22*86d7f5d3SJohn Marinoelse 23*86d7f5d3SJohn Marino STATUS=$? 24*86d7f5d3SJohn Marino echo "not ok - $1 # Test failed: regression detected. See above. (in $TESTDIR)" 25*86d7f5d3SJohn Marinofi) 26*86d7f5d3SJohn Marino 27*86d7f5d3SJohn Marinodnl An actual test. The first parameter is the test name. The second is the 28*86d7f5d3SJohn Marinodnl command/commands to execute for the actual test. Their exit status is 29*86d7f5d3SJohn Marinodnl checked. It is assumed that the test will output to stdout, and that the 30*86d7f5d3SJohn Marinodnl output to be used to check for regression will be in regress.TESTNAME.out. 31*86d7f5d3SJohn Marinodefine(`REGRESSION_TEST', 32*86d7f5d3SJohn Marino$2 | diff -u regress.$1.out - 33*86d7f5d3SJohn MarinoREGRESSION_PASSFAIL($1)) 34*86d7f5d3SJohn Marino 35*86d7f5d3SJohn Marinodnl A freeform regression test. Only exit status is checked. 36*86d7f5d3SJohn Marinodefine(`REGRESSION_TEST_FREEFORM', 37*86d7f5d3SJohn Marino$2 38*86d7f5d3SJohn MarinoREGRESSION_PASSFAIL($1)) 39*86d7f5d3SJohn Marino 40*86d7f5d3SJohn Marinodnl A regression test like REGRESSION_TEST, except only regress.out is used 41*86d7f5d3SJohn Marinodnl for checking output differences. The first argument is the command, the 42*86d7f5d3SJohn Marinodnl second argument (which may be empty) is the test name. 43*86d7f5d3SJohn Marinodefine(`REGRESSION_TEST_ONE', 44*86d7f5d3SJohn Marino$1 | diff -u regress.out - 45*86d7f5d3SJohn MarinoREGRESSION_PASSFAIL($2)) 46*86d7f5d3SJohn Marino 47*86d7f5d3SJohn Marinodnl A fatal error. This will exit with the given status (first argument) and 48*86d7f5d3SJohn Marinodnl print the message (second argument) prefixed with the string "FATAL :" to 49*86d7f5d3SJohn Marinodnl the error stream. 50*86d7f5d3SJohn Marinodefine(`REGRESSION_FATAL', 51*86d7f5d3SJohn Marinoecho "Bail out! $2 (in $TESTDIR)" > /dev/stderr 52*86d7f5d3SJohn Marinoexit $1) 53*86d7f5d3SJohn Marino 54*86d7f5d3SJohn Marinodnl Cleanup. Exit with the status code of the last failure. Should probably 55*86d7f5d3SJohn Marinodnl be the number of failed tests, but hey presto, this is what it does. This 56*86d7f5d3SJohn Marinodnl could also clean up potential droppings, if some forms of regression tests 57*86d7f5d3SJohn Marinodnl end up using mktemp(1) or such. 58*86d7f5d3SJohn Marinodefine(`REGRESSION_END', 59*86d7f5d3SJohn Marinoexit $STATUS) 60