1Overview 2================================================== 3This directory contains a random test generator for the gmp 4compatibility layer of imath. The tests cases are randomly generated 5and run with both gmp and imath. The results are compared and any 6mismatching results are flagged as failures. 7 8You should not see any failures when running these tests. 9 10Requirements 11================================================== 12These tests use the python ffi to run the imath and gmp functions. To 13run these tests you will need the following items 14 15 * libimath.so 16 * python 3 17 * gmp library and header files 18 19 20Running 21================================================== 22All the tests cases will be generated and run automatically by the 23makefile. First, make sure you have built libimath.so in the top level 24imath directory. 25 26Use the following command to generate and run the tests 27 28 $ make TESTS=random.tests 29 30This should generate all the needed wrapper files along with the 31randomized unit tests. By default the unit tests are output to the 32random.tests file. The tests can be run by hand using the following 33command line 34 35 $ ./runtest random.tests 36 37You can also write your own unit tests and run them by passing the 38file name to the runtest script. 39 40 41Testing Methodology 42================================================== 43The goal of our testing is to ensure that we are compatible with the 44gmp api. To test our compatibility layer we generate inputs for each 45gmp api that we wrap and then call the gmp version of the api and 46compare the results to the imath version. The results should be 47identical. Any output mismatch is considered an error. 48 49 50Testcase Generation 51-------------------- 52The test data generation is inspired by the QuickCheck testing 53methodolgy. We want to test both random data and important values such 54as 0,1,-1, etc. We generate input data based on the type of the input 55parameter. Most apis take either an mpz_t or a signed/unsigned long. 56 57For each parameter of the given type we generate the following values: 58 59 mpz_t: 0,1,-1, all min/max values, all min/max values +/- 1 60 small numbers: 1-4 digits 61 medium numbers: 5-20 digits 62 large numbers: 20-100 digits 63 long : 0,1,-1 short,int,long min/max values 64 random long values 65 unsigned long: 0,1,-1, unsigned short, unsigned int, unsigned long min/max values 66 random unsigned long values 67 68The generated data for each paramater is combined to produce a series 69of inputs to the function. The input data format looks like: 70 71mpz_add|0,1,2 72 73Which represents the call mpz_add(0, 1, 2). Additional test cases can be written by hand. 74 75Test Structure 76-------------------- 77The tests are run using the python ffi (the ctypes module). We have a 78single description of each api that is used to generate the following: 79 80 * intput data 81 * c function wrapper to call the libgmp or libimath function 82 * python wrapper to call both libgmp and libimath wrappers and compare results 83 84The test_gmp.so and test_imath.so libraries are loaded at runtime by 85the python runner scrip and used to run each test input and compare 86the results. 87 88The code generation pattern looks something like this: 89 90genctest.py ~~generates~~> gmp_test.c <--includes-- gmp_custom_test.c 91gmp_test.c ~~generates~~> gmp_tests.so <--links-- libgmp.so 92genpytest.py ~~generates~~> wrappers.py <--calls-- gmp_test.so 93gmp_test.so --loads--> runtest.py <--includes-- wrappers.py 94 ^ 95 | 96 reads 97 | 98gendata.py ~~generates~~> random.tests 99 100Adding tests for new APIs 101================================================== 102New apis can be tested by modifying the gmpapy.py file. This file 103contains a list of all apis that will be tested. Adding an api to the 104apis list will cause code and input data to be generated for that api. 105Rerunning make will generate the test data and code to run the test. 106 107