diff options
-rw-r--r-- | doc/devel/atf.dox | 191 | ||||
-rw-r--r-- | doc/devel/mainpage.dox | 1 | ||||
-rw-r--r-- | doc/devel/qa.dox | 9 | ||||
-rw-r--r-- | tests/HOWTO-unit-test | 87 |
4 files changed, 165 insertions, 123 deletions
diff --git a/doc/devel/atf.dox b/doc/devel/atf.dox index 29615c7e..ff4e182b 100644 --- a/doc/devel/atf.dox +++ b/doc/devel/atf.dox @@ -3,6 +3,29 @@ @section testsOverview Testing Overview +In DHCP, a unit test exercises a particular piece of code in +isolation. There is a separate unit test per module or API. Each unit +test lives in a directory beneath the code it is designed to exercise. +So, we (will eventually) have: + +@verbatim +server/tests/ +client/tests/ +common/tests/ +dhcpctl/tests/ +... +@endverbatim + +And so on. + +Ideally each function would be invoked with every possible type of input, and +each branch of every function would be checked. In practice we try to be a bit +more pragmatic, and target the most basic operations, as well tricky code, and +areas we have seen bugs in the past. + +We are using <a href="http://code.google.com/p/kyua/wiki/ATF">ATF (Automated +Test Framework)</a> as a framework to run our unittests. + @section testsAtf ATF unit-tests ATF stands for Automated Test Framework, and is the framework used for unit @@ -21,17 +44,17 @@ tested version of ATF that DHCP's unittests were run against is 0.15. To build the unit-tests, use the following: @verbatim -./configure --with-atf -make -make check +$ ./configure --with-atf +$ make +$ make check @endverbatim The following syntax is supported as well: @verbatim -./configure --with-atf=/path/to/your/atf/install +$ ./configure --with-atf=/path/to/your/atf/install @endverbatim -but it seems to have troubles detecting ATF installation, at least +but it seems to have troubles sometimes detecting ATF installation, at least with ATF 0.14 and Mac OS X 10.6.8. Each code directory (e.g. server/) that has unit-tests has a sub-directory @@ -42,69 +65,131 @@ Unit-tests are grouped into suites, each suite being a separate executable. The typical way to run tests is: @verbatim -atf-run | atf-report +$ atf-run | atf-report @endverbatim -atf-run will read the Atffile in the current directory and execute -all the tests specified in it. Using atf-run - rather than calling the -test binary directly - has several major benefits. The main one is that -atf-run is able to recover from test segfault and continue execution -from the next case onwards. +atf-run will read the Atffile in the current directory and execute all the tests +specified in it. Using atf-run - rather than calling the test binary directly - +has several major benefits. The main one is that atf-run is able to recover from +test segfault and continue execution from the next case onwards. Another is that +it is possible to specify a timeout for a test. atf-run will kill the test in +case of any infinite loops and will continue running next tests. -It is possible to run atf-run without passing its output to atf-report, -but its output is somewhat convoluted. That is useful in some situations, -e.g. when one wants to see test output. +It is possible to run atf-run without passing its output to atf-report, but its +output is somewhat convoluted. That is useful in some situations, e.g. when one +wants to see test output. -It is possible to run test binary directly. The only required parameter -is the test case name. The binary will print out a warning that direct -binary execution is not recommended as it won't be able to recover -from crash. However, such an approach is convenient for running the -test under the debugger. +It is possible to run test binary directly. The only required parameter is the +test case name. The binary will print out a warning that direct binary execution +is not recommended as it won't be able to recover from crash. However, such an +approach is convenient for running the test under the debugger. @section testsAtfAdding Adding new unit-tests -There are a small number of unit-tests that are not ATF based. They will -be converted to ATF soon. Please do not use any other frameworks. +There are a small number of unit-tests that are not ATF based. They will be +converted to ATF soon. Please do not use any other frameworks. Sadly, the DHCP code was not written with unit-testing in mind: often a -non-standard approach is required for writing unit-tests. The existing -code often has many dependencies that make testing a single piece of code -awkward to unit test. For example, to test hash tables, one needs to -also include the OMAPI code. Rather than significantly refactoring the -code (a huge task that could take months), we decided to link whatever -is needed in the tests. If developing new test suite, it is recommended -that you take a look at existing tests and just copy them. In particular, -the following things should be done for adding new tests: - -1. Create new file that will hold test code. It is recommended you -name it (tested_feature_name)_unittest.c and put the file in specified -tests directory. For example tests related to hash tables used on the -server side should be named server/tests/hash_unittest.c. If in doubt, -it is convenient to name the test code after the file that holds tested -code, e.g. server/mdb6.c is tested in server/tests/mdb6_unittest.c. - -2. Implement the test. The file server/tests/simple_unittest.c holds a -template explaining the basic layout of the ATF tests. There may be many -test cases in a single *_unittest.c file. Make sure that you register -all your test cases using ATF_TP_ADD_TC() macro, and try to minimize -modifications to the tested code if possible. Keep in mind that we are -using modernized \ref codingGuidelines for test development. You are -advised to also look at atf-c-api(3). - -3. Extend Makefile.am to build your test. In particular, add your binary +non-standard approach is required for writing unit-tests. The existing code +often has many dependencies that make testing a single piece of code awkward to +unit test. For example, to test hash tables, one needs to also include the +OMAPI code. Rather than significantly refactoring the code (a huge task that +could take months), we decided to link whatever is needed in the tests. If +developing new test suite, it is recommended that you take a look at existing +tests and just copy them as a starting point. + + +In particular, the following +things should be done for adding new tests: + +<b>1. Tests directory.</b> For each code component (server, client, common, +etc.) there should be a tests subdirectory. If it isn't there yet, then it must +be created. This can be done by: + +a). Creating the directory: + +@verbatim + $ mkdir $subdir/tests + $ cvs add tests +@endverbatim + +b). Adding the subdirectory to the build system: + + Add to $subdir/Makefile.am: + +@verbatim + SUBDIRS = tests +@endverbatim + + Add to the AC_OUTPUT macro in configure.ac: + +@verbatim + subdir/tests/Makefile +@endverbatim + +c. Create a Makefile.am in the new directory, something similar to this: + +@verbatim + AM_CPPFLAGS = -I../.. + + check_PROGRAMS = test_foo + + TESTS = test_foo + + test_foo_SOURCES = test_foo.c + test_foo_LDADD = ../../tests/libt_api.a # plus others... +@endverbatim + +See existing Makefile.am for examples, and the Automake documentation: + + http://www.gnu.org/software/automake/manual/html_node/Tests.html + +<b>2. Implement the test.</b> That typically means that you create a new file that will +hold test code. It is recommended you name it (tested_feature_name)_unittest.c +and put the file in specified tests directory. For example tests related to +hash tables used on the server side should be named +server/tests/hash_unittest.c. If in doubt, it is convenient to name the test +code after the file that holds tested code, e.g. server/mdb6.c is tested in +server/tests/mdb6_unittest.c. + +The file server/tests/simple_unittest.c holds a template explaining the basic +layout of the ATF tests. There may be many test cases in a single *_unittest.c +file. Make sure that you register all your test cases using ATF_TP_ADD_TC() +macro, and try to minimize modifications to the tested code if possible. Keep in +mind that we are using modernized \ref codingGuidelines for test +development. You are advised to also look at atf-c-api(3) man page. + +To add a new test, such as when a new module is added or when you want to start +testing existing code, you can copy the server/tests/simple_unittest.c as a new +new file, add the new file as a target in Makefile.am, and begin adding +tests. Reviewing that file is a good idea, even if you decide to write your test +from scratch, as it give you quick overview of the essential capabilities of the +ATF framework (how to write test, how to make checks, pass or fail test +etc.). Do not forget to add your new file to git via "git add +yourtest_unittest.c". + +<b>3. Extend Makefile.am</b> to build your test. In particular, add your binary name to ATF_TESTS. The tests directory will be built only in case where -ATF is enabled, using --enable-atf during configure phase. +ATF is enabled, using --with-atf during configure phase. -4. Modify Atffile to include your new test binary, if needed. If you -followed naming convention proposed in step 2, your test will be included -and will be included automatically. +<b>4. Modify Atffile to include your new test</b>, if needed. Tests in specified +directory must be registered in Atffile. See server/tests/Atffile for +example. Currently every executable with name that falls into *_unittest pattern +will be executed automatically. If you followed naming convention proposed in a +previous step, your test will be included and will be included automatically. -5. Enjoy your improved confidence in the code, as you can run the tests after +<b>5. Enjoy your improved confidence in the code</b>, as you can run the tests after any change you may want to do: @verbatim -make check -atf-run | atf-report +$ make check +@endverbatim + +or run them manually + +@verbatim +$ cd server/tests +$ atf-run | atf-report @endverbatim @section testsAtfCoding ATF Coding Guidelines diff --git a/doc/devel/mainpage.dox b/doc/devel/mainpage.dox index d96e4d2e..e67f290b 100644 --- a/doc/devel/mainpage.dox +++ b/doc/devel/mainpage.dox @@ -19,6 +19,7 @@ - @subpage qaTests - @subpage cppcheck - @subpage doxygen + - @subpage valgrind - @subpage debug - @subpage omapi - @subpage omapiIntro diff --git a/doc/devel/qa.dox b/doc/devel/qa.dox index 32404957..7505330e 100644 --- a/doc/devel/qa.dox +++ b/doc/devel/qa.dox @@ -81,4 +81,13 @@ See tests/tools/perfdhcp directory in BIND10 source code. href="http://tahi.org/logo/dhcpv6/">DHCPv6 conformance tests</a>. ISC plans to deploy and run them periodically in a near future. + @section valgrind Memory correctness using valgrind + +<a href="http://valgrind.org/">Valgrind</a> is a powerful tool for dynamic code +analysis. It allows running existing code (often even without recompiling) in a +special environment that tracks memory operations. In particular, it is able to +detect: memory leaks, buffer overflows, usage of uninitialized memory, double +frees and similar errors. We currently do not use valgrind in ISC DHCP testing, +but there are plans for start using it. + */ diff --git a/tests/HOWTO-unit-test b/tests/HOWTO-unit-test index 1b6cf521..b84beba3 100644 --- a/tests/HOWTO-unit-test +++ b/tests/HOWTO-unit-test @@ -1,6 +1,18 @@ Introduction ------------ +That is only a brief overview of tests in ISC DHCP. For more thorough +description, see ISC DHCP Developer's Guide. You can generate it, by +having Doxygen installed and doing: + + cd doc + make devel + +and then opening doc/html/index.html + +Tests Overview +-------------- + In DHCP, a unit test exercises a particular piece of code in isolation. There is a separate unit test per module or API. Each unit test lives in a directory beneath the code it is designed to exercise. @@ -13,21 +25,16 @@ So, we (will eventually) have: And so on. -Ideally each function would be invoked with every possible type of -input, and each branch of every function would be checked. In practice -we try to be a bit more pragmatic, and target the most basic -operations, as well tricky code, and areas we have seen bugs in the -past. - We are using ATF (Automated Test Framework) as a framework to run our -unittests. +unittests. See ISC DHCP Developer's Guide for much more thorough +description of unit-test and ATF framework in general. Running Unit Tests ------------------ In order to run the unit tests for DHCP, enable ATF support during configure: -$ ./configure --enable-atf +$ ./configure --with-atf And then use: @@ -48,69 +55,9 @@ test. See atf-run and atf-report documentation. Adding a New Unit Test ---------------------- -To add an additional test to an existing test program, you must create -a function for the new test in the C source file. You can extend existing -test suite or create a new one. For introductory example, see -server/tests/simple_unittest.c. It is a skeleton example of ATF test. - -Tests in specified directory must be registered in Atffile. See -server/tests/Atffile for example. Currently every executable with name -that falls into *_unittest pattern will be executed automatically. +See ISC DHCP Developer's Guide. Adding a New Unit Test Program ------------------------------ -To add a new program, such as when a new module is added, you can copy -the "simple_unittest.c" file (in this directory) to a new name, add -the new file as a target in Makefile.am, and begin adding tests. Do -not forget to add it to CVS via "cvs add". - -If there is no "tests" directory for a given subdirectory, then one -must be created. This can be done by: - -1. Creating the directory: - - $ mkdir $subdir/tests - $ cvs add tests - -2. Adding the subdirectory to the build system: - - Add to $subdir/Makefile.am: - - SUBDIRS = tests - - Add to the AC_OUTPUT macro in configure.ac: - - $subdir/tests/Makefile - -3. Create a Makefile.am in the new directory, something like this: - - AM_CPPFLAGS = -I../.. - - check_PROGRAMS = test_foo - - TESTS = test_foo - - test_foo_SOURCES = test_foo.c - test_foo_LDADD = ../../tests/libt_api.a # plus others... - - -See existing Makefile.am for examples, and the Automake documentation: - - http://www.gnu.org/software/automake/manual/html_node/Tests.html - - -Support Functions ------------------ - -See http://www.netbsd.org/~jmmv/atf/index.html for details about ATF. Note -that although ATF was originally used for NetBSD validation, it is used -on many different platforms and is not NetBSD specific. - -Additional Testing ------------------- - -Other static or runtime testing is always an option. For instance, you -can use valgrind to check for memory leaks. - -$Id: HOWTO-unit-test,v 1.2.570.1 2012/01/31 15:50:15 tomasz Exp $ +See ISC DHCP Developer's Guide. |