1========================== 2Source-based Code Coverage 3========================== 4 5.. contents:: 6 :local: 7 8Introduction 9============ 10 11This document explains how to use clang's source-based code coverage feature. 12It's called "source-based" because it operates on AST and preprocessor 13information directly. This allows it to generate very precise coverage data. 14 15Clang ships two other code coverage implementations: 16 17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the 18 various sanitizers. It can provide up to edge-level coverage. 19 20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo. 21 This is enabled by ``-ftest-coverage`` or ``--coverage``. 22 23From this point onwards "code coverage" will refer to the source-based kind. 24 25The code coverage workflow 26========================== 27 28The code coverage workflow consists of three main steps: 29 30* Compiling with coverage enabled. 31 32* Running the instrumented program. 33 34* Creating coverage reports. 35 36The next few sections work through a complete, copy-'n-paste friendly example 37based on this program: 38 39.. code-block:: cpp 40 41 % cat <<EOF > foo.cc 42 #define BAR(x) ((x) || (x)) 43 template <typename T> void foo(T x) { 44 for (unsigned I = 0; I < 10; ++I) { BAR(I); } 45 } 46 int main() { 47 foo<int>(0); 48 foo<float>(0); 49 return 0; 50 } 51 EOF 52 53Compiling with coverage enabled 54=============================== 55 56To compile code with coverage enabled, pass ``-fprofile-instr-generate 57-fcoverage-mapping`` to the compiler: 58 59.. code-block:: console 60 61 # Step 1: Compile with coverage enabled. 62 % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo 63 64Note that linking together code with and without coverage instrumentation is 65supported. Uninstrumented code simply won't be accounted for in reports. 66 67Running the instrumented program 68================================ 69 70The next step is to run the instrumented program. When the program exits it 71will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE`` 72environment variable. If that variable does not exist, the profile is written 73to ``default.profraw`` in the current directory of the program. If 74``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing 75directory structure will be created. Additionally, the following special 76**pattern strings** are rewritten: 77 78* "%p" expands out to the process ID. 79 80* "%h" expands out to the hostname of the machine running the program. 81 82* "%t" expands out to the value of the ``TMPDIR`` environment variable. On 83 Darwin, this is typically set to a temporary scratch directory. 84 85* "%Nm" expands out to the instrumented binary's signature. When this pattern 86 is specified, the runtime creates a pool of N raw profiles which are used for 87 on-line profile merging. The runtime takes care of selecting a raw profile 88 from the pool, locking it, and updating it before the program exits. If N is 89 not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must 90 be between 1 and 9. The merge pool specifier can only occur once per filename 91 pattern. 92 93* "%c" expands out to nothing, but enables a mode in which profile counter 94 updates are continuously synced to a file. This means that if the 95 instrumented program crashes, or is killed by a signal, perfect coverage 96 information can still be recovered. Continuous mode does not support value 97 profiling for PGO, and is only supported on Darwin at the moment. Support for 98 Linux may be mostly complete but requires testing, and support for Windows 99 may require more extensive changes: please get involved if you are interested 100 in porting this feature. 101 102.. code-block:: console 103 104 # Step 2: Run the program. 105 % LLVM_PROFILE_FILE="foo.profraw" ./foo 106 107Note that continuous mode is also used on Fuchsia where it's the only supported 108mode, but the implementation is different. The Darwin and Linux implementation 109relies on padding and the ability to map a file over the existing memory 110mapping which is generally only available on POSIX systems and isn't suitable 111for other platforms. 112 113On Fuchsia, we rely on the ability to relocate counters at runtime using a 114level of indirection. On every counter access, we add a bias to the counter 115address. This bias is stored in ``__llvm_profile_counter_bias`` symbol that's 116provided by the profile runtime and is initially set to zero, meaning no 117relocation. The runtime can map the profile into memory at arbitrary locations, 118and set bias to the offset between the original and the new counter location, 119at which point every subsequent counter access will be to the new location, 120which allows updating profile directly akin to the continuous mode. 121 122The advantage of this approach is that doesn't require any special OS support. 123The disadvantage is the extra overhead due to additional instructions required 124for each counter access (overhead both in terms of binary size and performance) 125plus duplication of counters (i.e. one copy in the binary itself and another 126copy that's mapped into memory). This implementation can be also enabled for 127other platforms by passing the ``-runtime-counter-relocation`` option to the 128backend during compilation. 129 130.. code-block:: console 131 132 % clang++ -fprofile-instr-generate -fcoverage-mapping -mllvm -runtime-counter-relocation foo.cc -o foo 133 134Creating coverage reports 135========================= 136 137Raw profiles have to be **indexed** before they can be used to generate 138coverage reports. This is done using the "merge" tool in ``llvm-profdata`` 139(which can combine multiple raw profiles and index them at the same time): 140 141.. code-block:: console 142 143 # Step 3(a): Index the raw profile. 144 % llvm-profdata merge -sparse foo.profraw -o foo.profdata 145 146There are multiple different ways to render coverage reports. The simplest 147option is to generate a line-oriented report: 148 149.. code-block:: console 150 151 # Step 3(b): Create a line-oriented coverage report. 152 % llvm-cov show ./foo -instr-profile=foo.profdata 153 154This report includes a summary view as well as dedicated sub-views for 155templated functions and their instantiations. For our example program, we get 156distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If 157``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line 158region counts (even in macro expansions): 159 160.. code-block:: none 161 162 1| 20|#define BAR(x) ((x) || (x)) 163 ^20 ^2 164 2| 2|template <typename T> void foo(T x) { 165 3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 166 ^22 ^20 ^20^20 167 4| 2|} 168 ------------------ 169 | void foo<int>(int): 170 | 2| 1|template <typename T> void foo(T x) { 171 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 172 | ^11 ^10 ^10^10 173 | 4| 1|} 174 ------------------ 175 | void foo<float>(int): 176 | 2| 1|template <typename T> void foo(T x) { 177 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 178 | ^11 ^10 ^10^10 179 | 4| 1|} 180 ------------------ 181 182If ``--show-branches=count`` and ``--show-expansions`` are also enabled, the 183sub-views will show detailed branch coverage information in addition to the 184region counts: 185 186.. code-block:: none 187 188 ------------------ 189 | void foo<float>(int): 190 | 2| 1|template <typename T> void foo(T x) { 191 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 192 | ^11 ^10 ^10^10 193 | ------------------ 194 | | | 1| 10|#define BAR(x) ((x) || (x)) 195 | | | ^10 ^1 196 | | | ------------------ 197 | | | | Branch (1:17): [True: 9, False: 1] 198 | | | | Branch (1:24): [True: 0, False: 1] 199 | | | ------------------ 200 | ------------------ 201 | | Branch (3:23): [True: 10, False: 1] 202 | ------------------ 203 | 4| 1|} 204 ------------------ 205 206 207To generate a file-level summary of coverage statistics instead of a 208line-oriented report, try: 209 210.. code-block:: console 211 212 # Step 3(c): Create a coverage summary. 213 % llvm-cov report ./foo -instr-profile=foo.profdata 214 Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover 215 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 216 /tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 217 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 218 TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 219 220The ``llvm-cov`` tool supports specifying a custom demangler, writing out 221reports in a directory structure, and generating html reports. For the full 222list of options, please refer to the `command guide 223<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_. 224 225A few final notes: 226 227* The ``-sparse`` flag is optional but can result in dramatically smaller 228 indexed profiles. This option should not be used if the indexed profile will 229 be reused for PGO. 230 231* Raw profiles can be discarded after they are indexed. Advanced use of the 232 profile runtime library allows an instrumented program to merge profiling 233 information directly into an existing raw profile on disk. The details are 234 out of scope. 235 236* The ``llvm-profdata`` tool can be used to merge together multiple raw or 237 indexed profiles. To combine profiling data from multiple runs of a program, 238 try e.g: 239 240 .. code-block:: console 241 242 % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata 243 244Exporting coverage data 245======================= 246 247Coverage data can be exported into JSON using the ``llvm-cov export`` 248sub-command. There is a comprehensive reference which defines the structure of 249the exported data at a high level in the llvm-cov source code. 250 251Interpreting reports 252==================== 253 254There are five statistics tracked in a coverage summary: 255 256* Function coverage is the percentage of functions which have been executed at 257 least once. A function is considered to be executed if any of its 258 instantiations are executed. 259 260* Instantiation coverage is the percentage of function instantiations which 261 have been executed at least once. Template functions and static inline 262 functions from headers are two kinds of functions which may have multiple 263 instantiations. This statistic is hidden by default in reports, but can be 264 enabled via the ``-show-instantiation-summary`` option. 265 266* Line coverage is the percentage of code lines which have been executed at 267 least once. Only executable lines within function bodies are considered to be 268 code lines. 269 270* Region coverage is the percentage of code regions which have been executed at 271 least once. A code region may span multiple lines (e.g in a large function 272 body with no control flow). However, it's also possible for a single line to 273 contain multiple code regions (e.g in "return x || y && z"). 274 275* Branch coverage is the percentage of "true" and "false" branches that have 276 been taken at least once. Each branch is tied to individual conditions in the 277 source code that may each evaluate to either "true" or "false". These 278 conditions may comprise larger boolean expressions linked by boolean logical 279 operators. For example, "x = (y == 2) || (z < 10)" is a boolean expression 280 that is comprised of two individual conditions, each of which evaluates to 281 either true or false, producing four total branch outcomes. 282 283Of these five statistics, function coverage is usually the least granular while 284branch coverage is the most granular. 100% branch coverage for a function 285implies 100% region coverage for a function. The project-wide totals for each 286statistic are listed in the summary. 287 288Format compatibility guarantees 289=============================== 290 291* There are no backwards or forwards compatibility guarantees for the raw 292 profile format. Raw profiles may be dependent on the specific compiler 293 revision used to generate them. It's inadvisable to store raw profiles for 294 long periods of time. 295 296* Tools must retain **backwards** compatibility with indexed profile formats. 297 These formats are not forwards-compatible: i.e, a tool which uses format 298 version X will not be able to understand format version (X+k). 299 300* Tools must also retain **backwards** compatibility with the format of the 301 coverage mappings emitted into instrumented binaries. These formats are not 302 forwards-compatible. 303 304* The JSON coverage export format has a (major, minor, patch) version triple. 305 Only a major version increment indicates a backwards-incompatible change. A 306 minor version increment is for added functionality, and patch version 307 increments are for bugfixes. 308 309Impact of llvm optimizations on coverage reports 310================================================ 311 312llvm optimizations (such as inlining or CFG simplification) should have no 313impact on coverage report quality. This is due to the fact that the mapping 314from source regions to profile counters is immutable, and is generated before 315the llvm optimizer kicks in. The optimizer can't prove that profile counter 316instrumentation is safe to delete (because it's not: it affects the profile the 317program emits), and so leaves it alone. 318 319Note that this coverage feature does not rely on information that can degrade 320during the course of optimization, such as debug info line tables. 321 322Using the profiling runtime without static initializers 323======================================================= 324 325By default the compiler runtime uses a static initializer to determine the 326profile output path and to register a writer function. To collect profiles 327without using static initializers, do this manually: 328 329* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared 330 library and executable. When the linker finds a definition of this symbol, it 331 knows to skip loading the object which contains the profiling runtime's 332 static initializer. 333 334* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it 335 once from each instrumented executable. This function parses 336 ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files 337 at that path. To get the same behavior without truncating existing files, 338 pass a filename pattern string to ``void __llvm_profile_set_filename(char 339 *)``. These calls can be placed anywhere so long as they precede all calls 340 to ``__llvm_profile_write_file``. 341 342* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write 343 out a profile. This function returns 0 when it succeeds, and a non-zero value 344 otherwise. Calling this function multiple times appends profile data to an 345 existing on-disk raw profile. 346 347In C++ files, declare these as ``extern "C"``. 348 349Using the profiling runtime without a filesystem 350------------------------------------------------ 351 352The profiling runtime also supports freestanding environments that lack a 353filesystem. The runtime ships as a static archive that's structured to make 354dependencies on a hosted environment optional, depending on what features 355the client application uses. 356 357The first step is to export ``__llvm_profile_runtime``, as above, to disable 358the default static initializers. Instead of calling the ``*_file()`` APIs 359described above, use the following to save the profile directly to a buffer 360under your control: 361 362* Forward-declare ``uint64_t __llvm_profile_get_size_for_buffer(void)`` and 363 call it to determine the size of the profile. You'll need to allocate a 364 buffer of this size. 365 366* Forward-declare ``int __llvm_profile_write_buffer(char *Buffer)`` and call it 367 to copy the current counters to ``Buffer``, which is expected to already be 368 allocated and big enough for the profile. 369 370* Optionally, forward-declare ``void __llvm_profile_reset_counters(void)`` and 371 call it to reset the counters before entering a specific section to be 372 profiled. This is only useful if there is some setup that should be excluded 373 from the profile. 374 375In C++ files, declare these as ``extern "C"``. 376 377Collecting coverage reports for the llvm project 378================================================ 379 380To prepare a coverage report for llvm (and any of its sub-projects), add 381``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw 382profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html 383report, run ``llvm/utils/prepare-code-coverage-artifact.py``. 384 385To specify an alternate directory for raw profiles, use 386``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use 387``-DLLVM_PROFILE_MERGE_POOL_SIZE``. 388 389Drawbacks and limitations 390========================= 391 392* Prior to version 2.26, the GNU binutils BFD linker is not able link programs 393 compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode. Possible 394 workarounds include disabling ``--gc-sections``, upgrading to a newer version 395 of BFD, or using the Gold linker. 396 397* Code coverage does not handle unpredictable changes in control flow or stack 398 unwinding in the presence of exceptions precisely. Consider the following 399 function: 400 401 .. code-block:: cpp 402 403 int f() { 404 may_throw(); 405 return 0; 406 } 407 408 If the call to ``may_throw()`` propagates an exception into ``f``, the code 409 coverage tool may mark the ``return`` statement as executed even though it is 410 not. A call to ``longjmp()`` can have similar effects. 411 412Clang implementation details 413============================ 414 415This section may be of interest to those wishing to understand or improve 416the clang code coverage implementation. 417 418Gap regions 419----------- 420 421Gap regions are source regions with counts. A reporting tool cannot set a line 422execution count to the count from a gap region unless that region is the only 423one on a line. 424 425Gap regions are used to eliminate unnatural artifacts in coverage reports, such 426as red "unexecuted" highlights present at the end of an otherwise covered line, 427or blue "executed" highlights present at the start of a line that is otherwise 428not executed. 429 430Branch regions 431-------------- 432When viewing branch coverage details in source-based file-level sub-views using 433``--show-branches``, it is recommended that users show all macro expansions 434(using option ``--show-expansions``) since macros may contain hidden branch 435conditions. The coverage summary report will always include these macro-based 436boolean expressions in the overall branch coverage count for a function or 437source file. 438 439Branch coverage is not tracked for constant folded branch conditions since 440branches are not generated for these cases. In the source-based file-level 441sub-view, these branches will simply be shown as ``[Folded - Ignored]`` so that 442users are informed about what happened. 443 444Branch coverage is tied directly to branch-generating conditions in the source 445code. Users should not see hidden branches that aren't actually tied to the 446source code. 447 448 449Switch statements 450----------------- 451 452The region mapping for a switch body consists of a gap region that covers the 453entire body (starting from the '{' in 'switch (...) {', and terminating where the 454last case ends). This gap region has a zero count: this causes "gap" areas in 455between case statements, which contain no executable code, to appear uncovered. 456 457When a switch case is visited, the parent region is extended: if the parent 458region has no start location, its start location becomes the start of the case. 459This is used to support switch statements without a ``CompoundStmt`` body, in 460which the switch body and the single case share a count. 461 462For switches with ``CompoundStmt`` bodies, a new region is created at the start 463of each switch case. 464 465Branch regions are also generated for each switch case, including the default 466case. If there is no explicitly defined default case in the source code, a 467branch region is generated to correspond to the implicit default case that is 468generated by the compiler. The implicit branch region is tied to the line and 469column number of the switch statement condition since no source code for the 470implicit case exists. 471