1# DExTer (Debugging Experience Tester) 2 3## Introduction 4 5DExTer is a suite of tools used to evaluate the "User Debugging Experience". DExTer drives an external debugger, running on small test programs, and collects information on the behavior at each debugger step to provide quantitative values that indicate the quality of the debugging experience. 6 7## Supported Debuggers 8 9DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html). GDB is not currently supported. 10 11The following command evaluates your environment, listing the available and compatible debuggers: 12 13 dexter.py list-debuggers 14 15## Dependencies 16[TODO] Add a requirements.txt or an install.py and document it here. 17 18### Python 3.6 19 20DExTer requires python version 3.6 or greater. 21 22### pywin32 python package 23 24This is required to access the DTE interface for the Visual Studio debuggers. 25 26 <python-executable> -m pip install pywin32 27 28### clang 29 30DExTer is current compatible with 'clang' and 'clang-cl' compiler drivers. The compiler must be available for DExTer, for example the following command should successfully build a runnable executable. 31 32 <compiler-executable> tests/nostdlib/fibonacci/test.cpp 33 34## Running a test case 35 36The following commands build fibonacci.cpp from the tests/nostdlib directory and run it in LLDB, reporting the debug experience heuristic. The first pair of commands build with no optimizations (-O0) and score 1.0000. The second pair of commands build with optimizations (-O2) and score 0.2832 which suggests a worse debugging experience. 37 38 clang -O0 -g tests/nostdlib/fibonacci.cpp -o tests/nostdlib/fibonacci/test 39 dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp 40 test.cpp = (1.0000) 41 42 clang -O2 -g tests/nostdlib/fibonacci/test.cpp -o tests/nostdlib/fibonacci/test 43 dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp 44 test.cpp = (0.2832) 45 46## An example test case 47 48The sample test case (tests/nostdlib/fibonacci) looks like this: 49 50 1. #ifdef _MSC_VER 51 2. # define DEX_NOINLINE __declspec(noinline) 52 3. #else 53 4. # define DEX_NOINLINE __attribute__((__noinline__)) 54 5. #endif 55 6. 56 7. DEX_NOINLINE 57 8. void Fibonacci(int terms, int& total) 58 9. { 59 0. int first = 0; 60 11. int second = 1; 61 12. for (int i = 0; i < terms; ++i) 62 13. { 63 14. int next = first + second; // DexLabel('start') 64 15. total += first; 65 16. first = second; 66 17. second = next; // DexLabel('end') 67 18. } 68 19. } 69 20. 70 21. int main() 71 22. { 72 23. int total = 0; 73 24. Fibonacci(5, total); 74 25. return total; 75 26. } 76 27. 77 28. /* 78 29. DexExpectWatchValue('i', '0', '1', '2', '3', '4', 79 30. from_line='start', to_line='end') 80 31. DexExpectWatchValue('first', '0', '1', '2', '3', '5', 81 32. from_line='start', to_line='end') 82 33. DexExpectWatchValue('second', '1', '2', '3', '5', 83 34 from_line='start', to_line='end') 84 35. DexExpectWatchValue('total', '0', '1', '2', '4', '7', 85 36. from_line='start', to_line='end') 86 37. DexExpectWatchValue('next', '1', '2', '3', '5', '8', 87 38. from_line='start', to_line='end') 88 39. DexExpectWatchValue('total', '7', on_line=25) 89 40. DexExpectStepKind('FUNC_EXTERNAL', 0) 90 41. */ 91 92[DexLabel][1] is used to give a name to a line number. 93 94The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should 95have particular values, `'0', '1', '2', '3','4'`, sequentially over the program 96lifetime on particular lines. You can refer to a named line or simply the line 97number (See line 39). 98 99At the end of the test is the following line: 100 101 DexExpectStepKind('FUNC_EXTERNAL', 0) 102 103This [DexExpectStepKind][3] command indicates that we do not expect the debugger 104to step into a file outside of the test directory. 105 106[1]: Commands.md#DexLabel 107[2]: Commands.md#DexExpectWatchValue 108[3]: Commands.md#DexExpectStepKind 109 110## Detailed DExTer reports 111 112Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using LLDB as the debugger and producing a detailed report: 113 114 $ dexter.py test --vs-solution clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci 115 116The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example: 117 118 fibonacci = (0.2832) 119 120 ## BEGIN ## 121 [1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}] 122 [2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] 123 [3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] 124 . [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}] 125 . [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 126 . [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] 127 . [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 128 . [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 129 . [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}] 130 . [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] 131 . [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 132 . [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}] 133 . [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] 134 . [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 135 . [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}] 136 . [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] 137 . [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 138 . [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}] 139 . [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] 140 . [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 141 . [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}] 142 . [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] 143 . [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] 144 . [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] 145 [25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}] 146 ## END (25 steps) ## 147 148 149 step kind differences [0/1] 150 FUNC_EXTERNAL: 151 0 152 153 test.cpp:15-18 [first] [9/21] 154 expected encountered values: 155 0 156 1 157 2 158 3 159 160 missing values: 161 5 [-6] 162 163 result optimized away: 164 step 5 (Variable is optimized away and not available.) [-3] 165 step 7 (Variable is optimized away and not available.) 166 step 8 (Variable is optimized away and not available.) 167 step 11 (Variable is optimized away and not available.) 168 step 14 (Variable is optimized away and not available.) 169 step 17 (Variable is optimized away and not available.) 170 step 20 (Variable is optimized away and not available.) 171 step 23 (Variable is optimized away and not available.) 172 173 test.cpp:15-18 [i] [15/21] 174 result optimized away: 175 step 5 (Variable is optimized away and not available.) [-3] 176 step 7 (Variable is optimized away and not available.) [-3] 177 step 8 (Variable is optimized away and not available.) [-3] 178 step 9 (Variable is optimized away and not available.) [-3] 179 step 11 (Variable is optimized away and not available.) [-3] 180 step 12 (Variable is optimized away and not available.) 181 step 14 (Variable is optimized away and not available.) 182 step 15 (Variable is optimized away and not available.) 183 step 17 (Variable is optimized away and not available.) 184 step 18 (Variable is optimized away and not available.) 185 step 20 (Variable is optimized away and not available.) 186 step 21 (Variable is optimized away and not available.) 187 step 23 (Variable is optimized away and not available.) 188 189 test.cpp:15-18 [second] [21/21] 190 expected encountered values: 191 1 192 2 193 3 194 5 195 196 result optimized away: 197 step 5 (Variable is optimized away and not available.) [-3] 198 step 7 (Variable is optimized away and not available.) [-3] 199 step 8 (Variable is optimized away and not available.) [-3] 200 step 11 (Variable is optimized away and not available.) [-3] 201 step 14 (Variable is optimized away and not available.) [-3] 202 step 17 (Variable is optimized away and not available.) [-3] 203 step 20 (Variable is optimized away and not available.) [-3] 204 step 23 (Variable is optimized away and not available.) 205 206 test.cpp:15-18 [total] [21/21] 207 expected encountered values: 208 0 209 210 missing values: 211 1 [-6] 212 2 [-6] 213 4 [-6] 214 7 [-3] 215 216 test.cpp:16-18 [next] [15/21] 217 result optimized away: 218 step 5 (Variable is optimized away and not available.) [-3] 219 step 8 (Variable is optimized away and not available.) [-3] 220 step 11 (Variable is optimized away and not available.) [-3] 221 step 14 (Variable is optimized away and not available.) [-3] 222 step 17 (Variable is optimized away and not available.) [-3] 223 step 20 (Variable is optimized away and not available.) 224 step 23 (Variable is optimized away and not available.) 225 226 test.cpp:26 [total] [0/7] 227 expected encountered values: 228 7 229 230The first line 231 232 fibonacci = (0.2832) 233 234shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example: 235 236 test.cpp:15-18 [first] [9/21] 237 expected encountered values: 238 0 239 1 240 2 241 3 242 243 missing values: 244 5 [-6] 245 246 result optimized away: 247 step 5 (Variable is optimized away and not available.) [-3] 248 step 7 (Variable is optimized away and not available.) 249 step 8 (Variable is optimized away and not available.) 250 step 11 (Variable is optimized away and not available.) 251 step 14 (Variable is optimized away and not available.) 252 step 17 (Variable is optimized away and not available.) 253 step 20 (Variable is optimized away and not available.) 254 step 23 (Variable is optimized away and not available.) 255 256shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away. 257 258## Writing new test cases 259 260Each test can be either embedded within the source file using comments or included as a separate file with the .dex extension. Dexter does not include support for building test cases, although if a Visual Studio Solution (.sln) is used as the test file, VS will build the program as part of launching a debugger session if it has not already been built. 261