186d7f5d3SJohn Marino# $FreeBSD: src/tools/regression/usr.bin/regress.m4,v 1.5 2004/11/11 19:47:54 nik Exp $ 286d7f5d3SJohn Marino 386d7f5d3SJohn Marinodnl A library of routines for doing regression tests for userland utilities. 486d7f5d3SJohn Marino 586d7f5d3SJohn Marinodnl Start up. We initialise the exit status to 0 (no failure) and change 686d7f5d3SJohn Marinodnl into the directory specified by our first argument, which is the 786d7f5d3SJohn Marinodnl directory to run the tests inside. 886d7f5d3SJohn Marinodefine(`REGRESSION_START', 986d7f5d3SJohn MarinoTESTDIR=$1 1086d7f5d3SJohn Marinoif [ -z "$TESTDIR" ]; then 1186d7f5d3SJohn Marino TESTDIR=. 1286d7f5d3SJohn Marinofi 1386d7f5d3SJohn Marinocd $TESTDIR 1486d7f5d3SJohn Marino 1586d7f5d3SJohn MarinoSTATUS=0) 1686d7f5d3SJohn Marino 1786d7f5d3SJohn Marinodnl Check $? to see if we passed or failed. The first parameter is the test 1886d7f5d3SJohn Marinodnl which passed or failed. It may be nil. 1986d7f5d3SJohn Marinodefine(`REGRESSION_PASSFAIL', 2086d7f5d3SJohn Marinoif [ $? -eq 0 ]; then 2186d7f5d3SJohn Marino echo "ok - $1 # Test detected no regression. (in $TESTDIR)" 2286d7f5d3SJohn Marinoelse 2386d7f5d3SJohn Marino STATUS=$? 2486d7f5d3SJohn Marino echo "not ok - $1 # Test failed: regression detected. See above. (in $TESTDIR)" 2586d7f5d3SJohn Marinofi) 2686d7f5d3SJohn Marino 2786d7f5d3SJohn Marinodnl An actual test. The first parameter is the test name. The second is the 2886d7f5d3SJohn Marinodnl command/commands to execute for the actual test. Their exit status is 2986d7f5d3SJohn Marinodnl checked. It is assumed that the test will output to stdout, and that the 3086d7f5d3SJohn Marinodnl output to be used to check for regression will be in regress.TESTNAME.out. 3186d7f5d3SJohn Marinodefine(`REGRESSION_TEST', 3286d7f5d3SJohn Marino$2 | diff -u regress.$1.out - 3386d7f5d3SJohn MarinoREGRESSION_PASSFAIL($1)) 3486d7f5d3SJohn Marino 3586d7f5d3SJohn Marinodnl A freeform regression test. Only exit status is checked. 3686d7f5d3SJohn Marinodefine(`REGRESSION_TEST_FREEFORM', 3786d7f5d3SJohn Marino$2 3886d7f5d3SJohn MarinoREGRESSION_PASSFAIL($1)) 3986d7f5d3SJohn Marino 4086d7f5d3SJohn Marinodnl A regression test like REGRESSION_TEST, except only regress.out is used 4186d7f5d3SJohn Marinodnl for checking output differences. The first argument is the command, the 4286d7f5d3SJohn Marinodnl second argument (which may be empty) is the test name. 4386d7f5d3SJohn Marinodefine(`REGRESSION_TEST_ONE', 4486d7f5d3SJohn Marino$1 | diff -u regress.out - 4586d7f5d3SJohn MarinoREGRESSION_PASSFAIL($2)) 4686d7f5d3SJohn Marino 4786d7f5d3SJohn Marinodnl A fatal error. This will exit with the given status (first argument) and 4886d7f5d3SJohn Marinodnl print the message (second argument) prefixed with the string "FATAL :" to 4986d7f5d3SJohn Marinodnl the error stream. 5086d7f5d3SJohn Marinodefine(`REGRESSION_FATAL', 5186d7f5d3SJohn Marinoecho "Bail out! $2 (in $TESTDIR)" > /dev/stderr 5286d7f5d3SJohn Marinoexit $1) 5386d7f5d3SJohn Marino 5486d7f5d3SJohn Marinodnl Cleanup. Exit with the status code of the last failure. Should probably 5586d7f5d3SJohn Marinodnl be the number of failed tests, but hey presto, this is what it does. This 5686d7f5d3SJohn Marinodnl could also clean up potential droppings, if some forms of regression tests 5786d7f5d3SJohn Marinodnl end up using mktemp(1) or such. 5886d7f5d3SJohn Marinodefine(`REGRESSION_END', 5986d7f5d3SJohn Marinoexit $STATUS) 60