README.md
1# DExTer (Debugging Experience Tester)
2
3## Introduction
4
5DExTer is a suite of tools used to evaluate the "User Debugging Experience". DExTer drives an external debugger, running on small test programs, and collects information on the behavior at each debugger step to provide quantitative values that indicate the quality of the debugging experience.
6
7## Supported Debuggers
8
9DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html). GDB is not currently supported.
10
11The following command evaluates your environment, listing the available and compatible debuggers:
12
13 dexter.py list-debuggers
14
15## Dependencies
16[TODO] Add a requirements.txt or an install.py and document it here.
17
18### Python 3.6
19
20DExTer requires python version 3.6 or greater.
21
22### pywin32 python package
23
24This is required to access the DTE interface for the Visual Studio debuggers.
25
26 <python-executable> -m pip install pywin32
27
28### clang
29
30DExTer is current compatible with 'clang' and 'clang-cl' compiler drivers. The compiler must be available for DExTer, for example the following command should successfully build a runnable executable.
31
32 <compiler-executable> tests/nostdlib/fibonacci/test.cpp
33
34## Running a test case
35
36The following commands build fibonacci.cpp from the tests/nostdlib directory and run it in LLDB, reporting the debug experience heuristic. The first pair of commands build with no optimizations (-O0) and score 1.0000. The second pair of commands build with optimizations (-O2) and score 0.2832 which suggests a worse debugging experience.
37
38 clang -O0 -g tests/nostdlib/fibonacci.cpp -o tests/nostdlib/fibonacci/test
39 dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp
40 test.cpp = (1.0000)
41
42 clang -O2 -g tests/nostdlib/fibonacci/test.cpp -o tests/nostdlib/fibonacci/test
43 dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp
44 test.cpp = (0.2832)
45
46## An example test case
47
48The sample test case (tests/nostdlib/fibonacci) looks like this:
49
50 1. #ifdef _MSC_VER
51 2. # define DEX_NOINLINE __declspec(noinline)
52 3. #else
53 4. # define DEX_NOINLINE __attribute__((__noinline__))
54 5. #endif
55 6.
56 7. DEX_NOINLINE
57 8. void Fibonacci(int terms, int& total)
58 9. {
59 0. int first = 0;
60 11. int second = 1;
61 12. for (int i = 0; i < terms; ++i)
62 13. {
63 14. int next = first + second; // DexLabel('start')
64 15. total += first;
65 16. first = second;
66 17. second = next; // DexLabel('end')
67 18. }
68 19. }
69 20.
70 21. int main()
71 22. {
72 23. int total = 0;
73 24. Fibonacci(5, total);
74 25. return total;
75 26. }
76 27.
77 28. /*
78 29. DexExpectWatchValue('i', '0', '1', '2', '3', '4',
79 30. from_line='start', to_line='end')
80 31. DexExpectWatchValue('first', '0', '1', '2', '3', '5',
81 32. from_line='start', to_line='end')
82 33. DexExpectWatchValue('second', '1', '2', '3', '5',
83 34 from_line='start', to_line='end')
84 35. DexExpectWatchValue('total', '0', '1', '2', '4', '7',
85 36. from_line='start', to_line='end')
86 37. DexExpectWatchValue('next', '1', '2', '3', '5', '8',
87 38. from_line='start', to_line='end')
88 39. DexExpectWatchValue('total', '7', on_line=25)
89 40. DexExpectStepKind('FUNC_EXTERNAL', 0)
90 41. */
91
92[DexLabel][1] is used to give a name to a line number.
93
94The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should
95have particular values, `'0', '1', '2', '3','4'`, sequentially over the program
96lifetime on particular lines. You can refer to a named line or simply the line
97number (See line 39).
98
99At the end of the test is the following line:
100
101 DexExpectStepKind('FUNC_EXTERNAL', 0)
102
103This [DexExpectStepKind][3] command indicates that we do not expect the debugger
104to step into a file outside of the test directory.
105
106[1]: Commands.md#DexLabel
107[2]: Commands.md#DexExpectWatchValue
108[3]: Commands.md#DexExpectStepKind
109
110## Detailed DExTer reports
111
112Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using LLDB as the debugger and producing a detailed report:
113
114 $ dexter.py test --vs-solution clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci
115
116The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example:
117
118 fibonacci = (0.2832)
119
120 ## BEGIN ##
121 [1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}]
122 [2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
123 [3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
124 . [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}]
125 . [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
126 . [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
127 . [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
128 . [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
129 . [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}]
130 . [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
131 . [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
132 . [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}]
133 . [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
134 . [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
135 . [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}]
136 . [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
137 . [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
138 . [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}]
139 . [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
140 . [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
141 . [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}]
142 . [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
143 . [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
144 . [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
145 [25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}]
146 ## END (25 steps) ##
147
148
149 step kind differences [0/1]
150 FUNC_EXTERNAL:
151 0
152
153 test.cpp:15-18 [first] [9/21]
154 expected encountered values:
155 0
156 1
157 2
158 3
159
160 missing values:
161 5 [-6]
162
163 result optimized away:
164 step 5 (Variable is optimized away and not available.) [-3]
165 step 7 (Variable is optimized away and not available.)
166 step 8 (Variable is optimized away and not available.)
167 step 11 (Variable is optimized away and not available.)
168 step 14 (Variable is optimized away and not available.)
169 step 17 (Variable is optimized away and not available.)
170 step 20 (Variable is optimized away and not available.)
171 step 23 (Variable is optimized away and not available.)
172
173 test.cpp:15-18 [i] [15/21]
174 result optimized away:
175 step 5 (Variable is optimized away and not available.) [-3]
176 step 7 (Variable is optimized away and not available.) [-3]
177 step 8 (Variable is optimized away and not available.) [-3]
178 step 9 (Variable is optimized away and not available.) [-3]
179 step 11 (Variable is optimized away and not available.) [-3]
180 step 12 (Variable is optimized away and not available.)
181 step 14 (Variable is optimized away and not available.)
182 step 15 (Variable is optimized away and not available.)
183 step 17 (Variable is optimized away and not available.)
184 step 18 (Variable is optimized away and not available.)
185 step 20 (Variable is optimized away and not available.)
186 step 21 (Variable is optimized away and not available.)
187 step 23 (Variable is optimized away and not available.)
188
189 test.cpp:15-18 [second] [21/21]
190 expected encountered values:
191 1
192 2
193 3
194 5
195
196 result optimized away:
197 step 5 (Variable is optimized away and not available.) [-3]
198 step 7 (Variable is optimized away and not available.) [-3]
199 step 8 (Variable is optimized away and not available.) [-3]
200 step 11 (Variable is optimized away and not available.) [-3]
201 step 14 (Variable is optimized away and not available.) [-3]
202 step 17 (Variable is optimized away and not available.) [-3]
203 step 20 (Variable is optimized away and not available.) [-3]
204 step 23 (Variable is optimized away and not available.)
205
206 test.cpp:15-18 [total] [21/21]
207 expected encountered values:
208 0
209
210 missing values:
211 1 [-6]
212 2 [-6]
213 4 [-6]
214 7 [-3]
215
216 test.cpp:16-18 [next] [15/21]
217 result optimized away:
218 step 5 (Variable is optimized away and not available.) [-3]
219 step 8 (Variable is optimized away and not available.) [-3]
220 step 11 (Variable is optimized away and not available.) [-3]
221 step 14 (Variable is optimized away and not available.) [-3]
222 step 17 (Variable is optimized away and not available.) [-3]
223 step 20 (Variable is optimized away and not available.)
224 step 23 (Variable is optimized away and not available.)
225
226 test.cpp:26 [total] [0/7]
227 expected encountered values:
228 7
229
230The first line
231
232 fibonacci = (0.2832)
233
234shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example:
235
236 test.cpp:15-18 [first] [9/21]
237 expected encountered values:
238 0
239 1
240 2
241 3
242
243 missing values:
244 5 [-6]
245
246 result optimized away:
247 step 5 (Variable is optimized away and not available.) [-3]
248 step 7 (Variable is optimized away and not available.)
249 step 8 (Variable is optimized away and not available.)
250 step 11 (Variable is optimized away and not available.)
251 step 14 (Variable is optimized away and not available.)
252 step 17 (Variable is optimized away and not available.)
253 step 20 (Variable is optimized away and not available.)
254 step 23 (Variable is optimized away and not available.)
255
256shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away.
257
258## Writing new test cases
259
260Each test can be either embedded within the source file using comments or included as a separate file with the .dex extension. Dexter does not include support for building test cases, although if a Visual Studio Solution (.sln) is used as the test file, VS will build the program as part of launching a debugger session if it has not already been built.
261