1/** 2@page tests Testing 3 4@section testsOverview Testing Overview 5 6In DHCP, a unit test exercises a particular piece of code in 7isolation. There is a separate unit test per module or API. Each unit 8test lives in a directory beneath the code it is designed to exercise. 9So, we (will eventually) have: 10 11@verbatim 12server/tests/ 13client/tests/ 14common/tests/ 15dhcpctl/tests/ 16... 17@endverbatim 18 19And so on. 20 21Ideally each function would be invoked with every possible type of input, and 22each branch of every function would be checked. In practice we try to be a bit 23more pragmatic, and target the most basic operations, as well tricky code, and 24areas we have seen bugs in the past. 25 26We are using <a href="http://code.google.com/p/kyua/wiki/ATF">ATF (Automated 27Test Framework)</a> as a framework to run our unittests. 28 29@section testsAtf ATF unit-tests 30 31ATF stands for Automated Test Framework, and is the framework used for unit 32tests in ISC DHCP and BIND9. ATF sources can be downloaded from 33http://code.google.com/p/kyua/wiki/ATF . ATF itself must be configured, compiled 34and then installed to be available during the DHCP configure procedure. Please 35follow INSTALL file supplied with ATF sources (it's essentially the typical 36./configure && make && make install procedure). 37 38The ATF successor, called Kyua, is being developed. As of August 2012, the 39latest available release of Kyua is 0.5. It claims to offer feature parity with 40ATF. Migration to Kyua may be planned some time in the future, but DHCP uses ATF 41for now. Such an upgrade should be done in coordination with BIND. The latest 42tested version of ATF that DHCP's unittests were run against is 0.15. 43 44To build the unit-tests, use the following: 45 46@verbatim 47$ ./configure --with-atf 48$ make 49$ make check 50@endverbatim 51 52The following syntax is supported as well: 53@verbatim 54$ ./configure --with-atf=/path/to/your/atf/install 55@endverbatim 56 57but it seems to have troubles sometimes detecting ATF installation, at least 58with ATF 0.14 and Mac OS X 10.6.8. 59 60Each code directory (e.g. server/) that has unit-tests has a sub-directory 61named tests (e.g. server/tests). You can execute "make check" in that 62directory to run specific subset of tests. 63 64Unit-tests are grouped into suites, each suite being a separate 65executable. The typical way to run tests is: 66 67@verbatim 68$ atf-run | atf-report 69@endverbatim 70 71atf-run will read the Atffile in the current directory and execute all the tests 72specified in it. Using atf-run - rather than calling the test binary directly - 73has several major benefits. The main one is that atf-run is able to recover from 74test segfault and continue execution from the next case onwards. Another is that 75it is possible to specify a timeout for a test. atf-run will kill the test in 76case of any infinite loops and will continue running next tests. 77 78It is possible to run atf-run without passing its output to atf-report, but its 79output is somewhat convoluted. That is useful in some situations, e.g. when one 80wants to see test output. 81 82It is possible to run test binary directly. The only required parameter is the 83test case name. The binary will print out a warning that direct binary execution 84is not recommended as it won't be able to recover from crash. However, such an 85approach is convenient for running the test under the debugger. 86 87@section testsAtfAdding Adding new unit-tests 88 89There are a small number of unit-tests that are not ATF based. They will be 90converted to ATF soon. Please do not use any other frameworks. 91 92Sadly, the DHCP code was not written with unit-testing in mind: often a 93non-standard approach is required for writing unit-tests. The existing code 94often has many dependencies that make testing a single piece of code awkward to 95unit test. For example, to test hash tables, one needs to also include the 96OMAPI code. Rather than significantly refactoring the code (a huge task that 97could take months), we decided to link whatever is needed in the tests. If 98developing new test suite, it is recommended that you take a look at existing 99tests and just copy them as a starting point. 100 101 102In particular, the following 103things should be done for adding new tests: 104 105<b>1. Tests directory.</b> For each code component (server, client, common, 106etc.) there should be a tests subdirectory. If it isn't there yet, then it must 107be created. This can be done by: 108 109a). Creating the directory: 110 111@verbatim 112 $ mkdir $subdir/tests 113 $ cvs add tests 114@endverbatim 115 116b). Adding the subdirectory to the build system: 117 118 Add to $subdir/Makefile.am: 119 120@verbatim 121 SUBDIRS = tests 122@endverbatim 123 124 Add to the AC_OUTPUT macro in configure.ac: 125 126@verbatim 127 subdir/tests/Makefile 128@endverbatim 129 130c. Create a Makefile.am in the new directory, something similar to this: 131 132@verbatim 133 AM_CPPFLAGS = -I../.. 134 135 check_PROGRAMS = test_foo 136 137 TESTS = test_foo 138 139 test_foo_SOURCES = test_foo.c 140 test_foo_LDADD = ../../tests/libt_api.a # plus others... 141@endverbatim 142 143See existing Makefile.am for examples, and the Automake documentation: 144 145 http://www.gnu.org/software/automake/manual/html_node/Tests.html 146 147<b>2. Implement the test.</b> That typically means that you create a new file that will 148hold test code. It is recommended you name it (tested_feature_name)_unittest.c 149and put the file in specified tests directory. For example tests related to 150hash tables used on the server side should be named 151server/tests/hash_unittest.c. If in doubt, it is convenient to name the test 152code after the file that holds tested code, e.g. server/mdb6.c is tested in 153server/tests/mdb6_unittest.c. 154 155The file server/tests/simple_unittest.c holds a template explaining the basic 156layout of the ATF tests. There may be many test cases in a single *_unittest.c 157file. Make sure that you register all your test cases using ATF_TP_ADD_TC() 158macro, and try to minimize modifications to the tested code if possible. Keep in 159mind that we are using modernized \ref codingGuidelines for test 160development. You are advised to also look at atf-c-api(3) man page. 161 162To add a new test, such as when a new module is added or when you want to start 163testing existing code, you can copy the server/tests/simple_unittest.c as a new 164new file, add the new file as a target in Makefile.am, and begin adding 165tests. Reviewing that file is a good idea, even if you decide to write your test 166from scratch, as it give you quick overview of the essential capabilities of the 167ATF framework (how to write test, how to make checks, pass or fail test 168etc.). Do not forget to add your new file to git via "git add 169yourtest_unittest.c". 170 171<b>3. Extend Makefile.am</b> to build your test. In particular, add your binary 172name to ATF_TESTS. The tests directory will be built only in case where 173ATF is enabled, using --with-atf during configure phase. 174 175<b>4. Modify Atffile to include your new test</b>, if needed. Tests in the 176specified directory must be registered in Atffile. See server/tests/Atffile for 177an example. Currently every executable with name of the form *_unittest will be 178executed automatically. If you followed naming convention proposed in a previous 179step, your test will be included and will be included automatically. 180 181<b>5. Enjoy your improved confidence in the code</b>, as you can run the tests after 182any change you may want to do: 183 184@verbatim 185$ make check 186@endverbatim 187 188to run all tests for all components. See \ref atfTests section for more details 189on running tests. 190 191@section testsAtfCoding ATF Coding Guidelines 192 193As the unit-test code creates an evironment that works under a different 194regime than the production code, there are slight differences to standard 195coding guidelines. In particular: 196 197- The code is written using C99. Double slash comments are allowed. 198- Please do not use tabs. Use 4 spaces for each indent level. 199 200*/ 201