1/** 2@page tests Testing 3 4@section testsOverview Testing Overview 5 6In DHCP, a unit test exercises a particular piece of code in 7isolation. There is a separate unit test per module or API. Each unit 8test lives in a directory beneath the code it is designed to exercise. 9So, we (will eventually) have: 10 11@verbatim 12server/tests/ 13client/tests/ 14common/tests/ 15dhcpctl/tests/ 16... 17@endverbatim 18 19And so on. 20 21Ideally each function would be invoked with every possible type of input, and 22each branch of every function would be checked. In practice we try to be a bit 23more pragmatic, and target the most basic operations, as well tricky code, and 24areas we have seen bugs in the past. 25 26We are using <a href="http://code.google.com/p/kyua/wiki/ATF">ATF (Automated 27Test Framework)</a> as a framework to run our unittests. 28 29@section testsAtf ATF unit-tests 30 31ATF stands for Automated Test Framework, and is the framework used for unit 32tests in ISC DHCP. Since ATF was created, a next generation QA toolchain 33called, Kyua, has beend developed that can execute ATF unit tests. There are 34two approaches one can take to install the necessary software for executing 35ISC DHCP unit tests: 36 371) Install pre-Kyua release (0.19 or earlier) of ATF 38 39Using this approach is the simplest as it involves installing only ATF. However 40it requires using a version of ATF that is no longer maintained and which may 41eventually may be unavailable. 42 43ATF may be found here: https://github.com/jmmv/atf 44 45 462) Install post-Kyua release of ATF (0.20 or later) and Kyua 47 48Using this approach requires the installation of more software as Kyua has 49dependencies that ATF lacks. The advantage is that the version(s) that can 50be used are all still supported. 51 52Kyua may be found here: https://github.com/jmmv/kyua 53 54@subsection preKyuaATF Using pre-Kyua ATF 55 56This section describes how to install and use pre-Kyua ATF release to run 57ISC DHCP unit tests. The most recent release of ATF that can be used without 58also needing Kyua, is ATF version 0.19. While this version is still available 59and using it is least software to install to run ISC DHCP unit tests, there 60may come a point in time when this version is no longer available. It may be 61downloaded here: https://github.com/jmmv/atf/releases/tag/atf-0.19 62 63After downloading and unpacking the source, you'll need to configure 64it, build it, and then install it. Note you'll need to include the 65"--enable-tools" option when configuring ATF. The following would build 66and install it to /opt/atf/atf-0.19/local: 67 68@verbatim 69 70$ cd /opt/atf 71$ tar -xvf $HOME/Downloads/atf-0.19.tar.gz 72 73# While the --prefix is optional, you must specify --enable-tools 74# as a configure option 75 76$ ./configure --prefix=/opt/atf/atf-0.19/local --enable-tools 77$ make 78$ sudo make install 79 80@endverbatim 81 82To build and run the ISC DHCP unit-tests, use the following: 83 84@verbatim 85$ git clone https://gitlab.isc.org/isc-projects/dhcp.git 86$ cd dhcp 87 88# Note because we used --prefix when configuraing ATF above, we 89# need to tell ISC DHCP configure where it is located. 90 91$ ./configure --with-atf=/opt/atf/atf-0.19/local 92 93# Run all of the unit tests via "make check" 94$ make check 95@endverbatim 96 97This will traverse the source tree running the unit tests in each unit test 98subdirectory. Note that if one or more tests in a unit test subdirectory fail 99the make process will stop. To run all of the tests regardless of outcome, 100use "make -k check" 101 102Note, If you installed ATF to its default location rather than using --prefix 103or you install the pkg-config .pc files for ATF, you should be able to 104simply specify "--with-atf" without a path when configuring ISC DHCP. 105 106Each dhcp source sub-directory (e.g. server, common) that has unit-tests has a 107sub-directory named tests (e.g. server/tests). You can execute "make check" in 108that sub-directory to run its specific subset of tests. 109 110Unit-tests are grouped into suites, each suite being a separate 111executable. The typical way to run tests is: 112 113@verbatim 114$ atf-run | atf-report 115 116# (This assumes atf-run and atf-report are in your path) 117# or 118 119$ sh ../../tests/unittests.sh 120@endverbatim 121 122atf-run will read the Atffile in the current directory and execute all 123the tests specified in it. Using atf-run - rather than calling the 124test binary directly - has several major benefits. The main one is 125that atf-run is able to recover from test segfault and continue 126execution from the next case onwards. Another is that it is possible 127to specify a timeout for a test. atf-run will kill the test in case of 128any infinite loops and will continue running next tests. 129 130It is possible to run atf-run without passing its output to atf-report, but its 131output is somewhat convoluted. That is useful in some situations, e.g. when one 132wants to see test output. 133 134It is possible to run test binary directly. The only required 135parameter is the test case name. The binary will print out a warning 136that direct binary execution is not recommended as it won't be able to 137recover from crash. However, such an approach is convenient for 138running the test under the debugger. 139 140@subsection KyuaPlusATF Using Kyua with ATF 141 142Kyua is a QA toolchain that has been layered on top of ATF. The tools 143necessary to run ATF unit tests migrated from ATF to within Kyua beginning 144with ATF 0.20. This section includes instructions for building and running 145ISC DHCP with Kyua. Instructions are included for both Ubuntu 18.04 and MacOS. 146 147@subsubsection KyuaLinux Kyua Ubuntu 18.04 Instructions 148 149The following instructions are based on Ubuntu 18.04, but 150are typical of a linux distribution/installation. It installs all software 151into default locations for simplicity. If you want to use 152custom installation paths, you may have to set additional environment 153variables, library paths, and the like. 154 155Before we install kyua, we must install its dependencies: lua, sqlite3, 156ATF, and lutok 157 158@verbatim 159 160# lua and sqlite3 have packages, we'll use them 161 162$ apt-get install liblua5.1-0-dev 163$ apt-get install libsqlite3-dev 164 165# Fetch, build, and install the latest version of ATF 166 167$ git clone https://github.com/jmmv/atf.git 168$ cd atf 169$ autoreconf --install --force 170$ ./configure 171$ make 172$ sudo make install 173$ cd .. 174 175# Fetch, build, and install the latest version of lutok 176 177$ git clone https://github.com/jmmv/lutok.git 178$ cd lutok 179 180# We must use -I argument to tell autoreconf where to find the ATF .m4 files 181$ autoreconf --install -s -I/usr/local/share/aclocal 182 183$ ./configure 184$ make 185$ sudo make install 186$ cd .. 187 188# Finally, we fetch, build and install the latest version of kyua 189 190$ git clone https://github.com/jmmv/kyua.git 191$ cd kyua 192 193# We must use -I argument to tell autoreconf where to find the ATF .m4 files 194$ autoreconf --install -s -I/usr/local/share/aclocal 195 196$ ./configure 197$ make 198$ sudo make install 199$ cd .. 200@endverbatim 201 202To build ISC DHCP with Kyua and run all the unit tests is the same: 203 204@verbatim 205$ git clone https://gitlab.isc.org/isc-projects/dhcp.git 206$ cd dhcp 207 208# Because we used default installations we simply enable unit tests with 209# --with-atf with no path value. 210 211$ ./configure --with-atf 212$ make 213 214# Run the ISC DHCP unit tests 215$ make check 216@endverbatim 217 218@subsubsection KyuaMacOS Kyua macOS Instructions 219 220The following instructions are for based on macOS Mojave (10.1.4.4). 221Before we install kyua, we must install its dependencies: pkgconfig, lutok, 222and ATF: 223 224@verbatim 225 226# install dependencies 227$ brew install pkgconfig 228$ brew install lutok 229 230# install atf so we can build kyua and dhcp with unit tests 231$ git clone https://github.com/jmmv/atf.git 232$ cd atf 233$ autoreconf --install --force 234$ ./configure --prefix=/opt/atf 235$ make 236$ sudo make install 237$ cd .. 238 239# configure kyua with atf 240$ git clone https://github.com/jmmv/kyua.git 241$ cd kyua 242$ autoreconf --install -s -I/opt/atf/share/aclocal 243$ ./configure --prefix=/opt/kyua 244$ make 245$ sudo make install 246$ cd .. 247@endverbatim 248 249To build ISC DHCP with Kyua and run all the unit tests is the same: 250 251@verbatim 252$ git clone https://gitlab.isc.org/isc-projects/dhcp.git 253$ cd dhcp 254 255# Because we used the prefix "/opt/atf" to install ATF we need to 256# add that path to --with-atf when we configure ISC DHCP. 257 258$ ./configure --with-atf=/opt/atf 259$ make 260 261# Add kyua to the path and then run the ISC DHCP unit tests 262# via "make check" 263$ export PATH=$PATH:/opt/kyua/bin 264$ make check 265@endverbatim 266 267@section testsAtfAdding Adding new unit-tests 268 269There are a small number of unit-tests that are not ATF based. They will be 270converted to ATF soon. Please do not use any other frameworks. 271 272Sadly, the DHCP code was not written with unit-testing in mind: often a 273non-standard approach is required for writing unit-tests. The existing code 274often has many dependencies that make testing a single piece of code awkward to 275unit test. For example, to test hash tables, one needs to also include the 276OMAPI code. Rather than significantly refactoring the code (a huge task that 277could take months), we decided to link whatever is needed in the tests. If 278developing new test suite, it is recommended that you take a look at existing 279tests and just copy them as a starting point. 280 281 282In particular, the following 283things should be done for adding new tests: 284 285<b>1. Tests directory.</b> For each code component (server, client, common, 286etc.) there should be a tests subdirectory. If it isn't there yet, then it must 287be created. This can be done by: 288 289a). Creating the directory: 290 291@verbatim 292 $ mkdir $subdir/tests 293 $ cvs add tests 294@endverbatim 295 296b). Adding the subdirectory to the build system: 297 298 Add to $subdir/Makefile.am: 299 300@verbatim 301 SUBDIRS = tests 302@endverbatim 303 304 Add to the AC_OUTPUT macro in configure.ac: 305 306@verbatim 307 subdir/tests/Makefile 308@endverbatim 309 310c. Create a Makefile.am in the new directory, something similar to this: 311 312@verbatim 313 AM_CPPFLAGS = -I../.. 314 315 check_PROGRAMS = test_foo 316 317 TESTS = test_foo 318 319 test_foo_SOURCES = test_foo.c 320 test_foo_LDADD = ../../tests/libt_api.a # plus others... 321@endverbatim 322 323See existing Makefile.am for examples, and the Automake documentation: 324 325 http://www.gnu.org/software/automake/manual/html_node/Tests.html 326 327<b>2. Implement the test.</b> That typically means that you create a 328new file that will hold test code. It is recommended you name it 329(tested_feature_name)_unittest.c and put the file in specified tests 330directory. For example tests related to hash tables used on the 331server side should be named server/tests/hash_unittest.c. If in doubt, 332it is convenient to name the test code after the file that holds 333tested code, e.g. server/mdb6.c is tested in 334server/tests/mdb6_unittest.c. 335 336The file server/tests/simple_unittest.c holds a template explaining 337the basic layout of the ATF tests. There may be many test cases in a 338single *_unittest.c file. Make sure that you register all your test 339cases using ATF_TP_ADD_TC() macro, and try to minimize modifications 340to the tested code if possible. Keep in mind that we are using 341modernized \ref codingGuidelines for test development. You are advised 342to also look at atf-c-api(3) man page. 343 344To add a new test, such as when a new module is added or when you want 345to start testing existing code, you can copy the 346server/tests/simple_unittest.c as a new new file, add the new file as 347a target in Makefile.am, and begin adding tests. Reviewing that file 348is a good idea, even if you decide to write your test from scratch, as 349it give you quick overview of the essential capabilities of the ATF 350framework (how to write test, how to make checks, pass or fail test 351etc.). Do not forget to add your new file to git via "git add 352yourtest_unittest.c". 353 354<b>3. Extend Makefile.am</b> to build your test. In particular, add your binary 355name to ATF_TESTS. The tests directory will be built only in case where 356ATF is enabled, using --with-atf during configure phase. 357 358<b>4. Modify Atffile to include your new test</b>, if needed. Tests in 359the specified directory must be registered in Atffile. See 360server/tests/Atffile for an example. Currently every executable with 361name of the form *_unittest will be executed automatically. If you 362followed naming convention proposed in a previous step, your test will 363be included and will be included automatically. 364 365<b>5. Enjoy your improved confidence in the code</b>, as you can run 366the tests after any change you may want to do: 367 368@verbatim 369$ make check 370@endverbatim 371 372to run all tests for all components. See \ref atfTests section for more details 373on running tests. 374 375@section testsAtfCoding ATF Coding Guidelines 376 377As the unit-test code creates an evironment that works under a different 378regime than the production code, there are slight differences to standard 379coding guidelines. In particular: 380 381- The code is written using C99. Double slash comments are allowed. 382- Please do not use tabs. Use 4 spaces for each indent level. 383 384*/ 385