1==================== 2XRay Instrumentation 3==================== 4 5:Version: 1 as of 2016-11-08 6 7.. contents:: 8 :local: 9 10 11Introduction 12============ 13 14XRay is a function call tracing system which combines compiler-inserted 15instrumentation points and a runtime library that can dynamically enable and 16disable the instrumentation. 17 18More high level information about XRay can be found in the `XRay whitepaper`_. 19 20This document describes how to use XRay as implemented in LLVM. 21 22XRay in LLVM 23============ 24 25XRay consists of three main parts: 26 27- Compiler-inserted instrumentation points. 28- A runtime library for enabling/disabling tracing at runtime. 29- A suite of tools for analysing the traces. 30 31 **NOTE:** As of July 25, 2018 , XRay is only available for the following 32 architectures running Linux: x86_64, arm7 (no thumb), aarch64, powerpc64le, 33 mips, mipsel, mips64, mips64el, NetBSD: x86_64, FreeBSD: x86_64 and 34 OpenBSD: x86_64. 35 36The compiler-inserted instrumentation points come in the form of nop-sleds in 37the final generated binary, and an ELF section named ``xray_instr_map`` which 38contains entries pointing to these instrumentation points. The runtime library 39relies on being able to access the entries of the ``xray_instr_map``, and 40overwrite the instrumentation points at runtime. 41 42Using XRay 43========== 44 45You can use XRay in a couple of ways: 46 47- Instrumenting your C/C++/Objective-C/Objective-C++ application. 48- Generating LLVM IR with the correct function attributes. 49 50The rest of this section covers these main ways and later on how to customize 51what XRay does in an XRay-instrumented binary. 52 53Instrumenting your C/C++/Objective-C Application 54------------------------------------------------ 55 56The easiest way of getting XRay instrumentation for your application is by 57enabling the ``-fxray-instrument`` flag in your clang invocation. 58 59For example: 60 61:: 62 63 clang -fxray-instrument ... 64 65By default, functions that have at least 200 instructions (or contain a loop) will 66get XRay instrumentation points. You can tweak that number through the 67``-fxray-instruction-threshold=`` flag: 68 69:: 70 71 clang -fxray-instrument -fxray-instruction-threshold=1 ... 72 73The loop detection can be disabled with ``-fxray-ignore-loops`` to use only the 74instruction threshold. You can also specifically instrument functions in your 75binary to either always or never be instrumented using source-level attributes. 76You can do it using the GCC-style attributes or C++11-style attributes. 77 78.. code-block:: c++ 79 80 [[clang::xray_always_instrument]] void always_instrumented(); 81 82 [[clang::xray_never_instrument]] void never_instrumented(); 83 84 void alt_always_instrumented() __attribute__((xray_always_instrument)); 85 86 void alt_never_instrumented() __attribute__((xray_never_instrument)); 87 88When linking a binary, you can either manually link in the `XRay Runtime 89Library`_ or use ``clang`` to link it in automatically with the 90``-fxray-instrument`` flag. Alternatively, you can statically link-in the XRay 91runtime library from compiler-rt -- those archive files will take the name of 92`libclang_rt.xray-{arch}` where `{arch}` is the mnemonic supported by clang 93(x86_64, arm7, etc.). 94 95LLVM Function Attribute 96----------------------- 97 98If you're using LLVM IR directly, you can add the ``function-instrument`` 99string attribute to your functions, to get the similar effect that the 100C/C++/Objective-C source-level attributes would get: 101 102.. code-block:: llvm 103 104 define i32 @always_instrument() uwtable "function-instrument"="xray-always" { 105 ; ... 106 } 107 108 define i32 @never_instrument() uwtable "function-instrument"="xray-never" { 109 ; ... 110 } 111 112You can also set the ``xray-instruction-threshold`` attribute and provide a 113numeric string value for how many instructions should be in the function before 114it gets instrumented. 115 116.. code-block:: llvm 117 118 define i32 @maybe_instrument() uwtable "xray-instruction-threshold"="2" { 119 ; ... 120 } 121 122Special Case File 123----------------- 124 125Attributes can be imbued through the use of special case files instead of 126adding them to the original source files. You can use this to mark certain 127functions and classes to be never, always, or instrumented with first-argument 128logging from a file. The file's format is described below: 129 130.. code-block:: bash 131 132 # Comments are supported 133 [always] 134 fun:always_instrument 135 fun:log_arg1=arg1 # Log the first argument for the function 136 137 [never] 138 fun:never_instrument 139 140These files can be provided through the ``-fxray-attr-list=`` flag to clang. 141You may have multiple files loaded through multiple instances of the flag. 142 143XRay Runtime Library 144-------------------- 145 146The XRay Runtime Library is part of the compiler-rt project, which implements 147the runtime components that perform the patching and unpatching of inserted 148instrumentation points. When you use ``clang`` to link your binaries and the 149``-fxray-instrument`` flag, it will automatically link in the XRay runtime. 150 151The default implementation of the XRay runtime will enable XRay instrumentation 152before ``main`` starts, which works for applications that have a short 153lifetime. This implementation also records all function entry and exit events 154which may result in a lot of records in the resulting trace. 155 156Also by default the filename of the XRay trace is ``xray-log.XXXXXX`` where the 157``XXXXXX`` part is randomly generated. 158 159These options can be controlled through the ``XRAY_OPTIONS`` environment 160variable during program run-time, where we list down the options and their 161defaults below. 162 163+-------------------+-----------------+---------------+------------------------+ 164| Option | Type | Default | Description | 165+===================+=================+===============+========================+ 166| patch_premain | ``bool`` | ``false`` | Whether to patch | 167| | | | instrumentation points | 168| | | | before main. | 169+-------------------+-----------------+---------------+------------------------+ 170| xray_mode | ``const char*`` | ``""`` | Default mode to | 171| | | | install and initialize | 172| | | | before ``main``. | 173+-------------------+-----------------+---------------+------------------------+ 174| xray_logfile_base | ``const char*`` | ``xray-log.`` | Filename base for the | 175| | | | XRay logfile. | 176+-------------------+-----------------+---------------+------------------------+ 177| verbosity | ``int`` | ``0`` | Runtime verbosity | 178| | | | level. | 179+-------------------+-----------------+---------------+------------------------+ 180 181 182In addition to environment variable, you can also provide your own definition of 183``const char *__xray_default_options(void)`` function, which returns the option 184strings. This method effectively provides default options during program build 185time. For example, you can create an additional source file (e.g. ``xray-opt.c`` 186) with the following ``__xray_default_options`` definition: 187 188.. code-block:: c 189 190 __attribute__((xray_never_instrument)) 191 const char *__xray_default_options() { 192 return "patch_premain=true,xray_mode=xray-basic"; 193 } 194 195And link it with the program you want to instrument: 196 197:: 198 199 clang -fxray-instrument prog.c xray-opt.c ... 200 201The instrumented binary will use 'patch_premain=true,xray_mode=xray-basic' by 202default even without setting ``XRAY_OPTIONS``. 203 204Note that you still can override options designated by ``__xray_default_options`` 205using ``XRAY_OPTIONS`` during run-time. 206 207If you choose to not use the default logging implementation that comes with the 208XRay runtime and/or control when/how the XRay instrumentation runs, you may use 209the XRay APIs directly for doing so. To do this, you'll need to include the 210``xray_log_interface.h`` from the compiler-rt ``xray`` directory. The important API 211functions we list below: 212 213- ``__xray_log_register_mode(...)``: Register a logging implementation against 214 a string Mode identifier. The implementation is an instance of 215 ``XRayLogImpl`` defined in ``xray/xray_log_interface.h``. 216- ``__xray_log_select_mode(...)``: Select the mode to install, associated with 217 a string Mode identifier. Only implementations registered with 218 ``__xray_log_register_mode(...)`` can be chosen with this function. 219- ``__xray_log_init_mode(...)``: This function allows for initializing and 220 re-initializing an installed logging implementation. See 221 ``xray/xray_log_interface.h`` for details, part of the XRay compiler-rt 222 installation. 223 224Once a logging implementation has been initialized, it can be "stopped" by 225finalizing the implementation through the ``__xray_log_finalize()`` function. 226The finalization routine is the opposite of the initialization. When finalized, 227an implementation's data can be cleared out through the 228``__xray_log_flushLog()`` function. For implementations that support in-memory 229processing, these should register an iterator function to provide access to the 230data via the ``__xray_log_set_buffer_iterator(...)`` which allows code calling 231the ``__xray_log_process_buffers(...)`` function to deal with the data in 232memory. 233 234All of this is better explained in the ``xray/xray_log_interface.h`` header. 235 236Basic Mode 237---------- 238 239XRay supports a basic logging mode which will trace the application's 240execution, and periodically append to a single log. This mode can be 241installed/enabled by setting ``xray_mode=xray-basic`` in the ``XRAY_OPTIONS`` 242environment variable. Combined with ``patch_premain=true`` this can allow for 243tracing applications from start to end. 244 245Like all the other modes installed through ``__xray_log_select_mode(...)``, the 246implementation can be configured through the ``__xray_log_init_mode(...)`` 247function, providing the mode string and the flag options. Basic-mode specific 248defaults can be provided in the ``XRAY_BASIC_OPTIONS`` environment variable. 249 250Flight Data Recorder Mode 251------------------------- 252 253XRay supports a logging mode which allows the application to only capture a 254fixed amount of memory's worth of events. Flight Data Recorder (FDR) mode works 255very much like a plane's "black box" which keeps recording data to memory in a 256fixed-size circular queue of buffers, and have the data available 257programmatically until the buffers are finalized and flushed. To use FDR mode 258on your application, you may set the ``xray_mode`` variable to ``xray-fdr`` in 259the ``XRAY_OPTIONS`` environment variable. Additional options to the FDR mode 260implementation can be provided in the ``XRAY_FDR_OPTIONS`` environment 261variable. Programmatic configuration can be done by calling 262``__xray_log_init_mode("xray-fdr", <configuration string>)`` once it has been 263selected/installed. 264 265When the buffers are flushed to disk, the result is a binary trace format 266described by `XRay FDR format <XRayFDRFormat.html>`_ 267 268When FDR mode is on, it will keep writing and recycling memory buffers until 269the logging implementation is finalized -- at which point it can be flushed and 270re-initialised later. To do this programmatically, we follow the workflow 271provided below: 272 273.. code-block:: c++ 274 275 // Patch the sleds, if we haven't yet. 276 auto patch_status = __xray_patch(); 277 278 // Maybe handle the patch_status errors. 279 280 // When we want to flush the log, we need to finalize it first, to give 281 // threads a chance to return buffers to the queue. 282 auto finalize_status = __xray_log_finalize(); 283 if (finalize_status != XRAY_LOG_FINALIZED) { 284 // maybe retry, or bail out. 285 } 286 287 // At this point, we are sure that the log is finalized, so we may try 288 // flushing the log. 289 auto flush_status = __xray_log_flushLog(); 290 if (flush_status != XRAY_LOG_FLUSHED) { 291 // maybe retry, or bail out. 292 } 293 294The default settings for the FDR mode implementation will create logs named 295similarly to the basic log implementation, but will have a different log 296format. All the trace analysis tools (and the trace reading library) will 297support all versions of the FDR mode format as we add more functionality and 298record types in the future. 299 300 **NOTE:** We do not promise perpetual support for when we update the log 301 versions we support going forward. Deprecation of the formats will be 302 announced and discussed on the developers mailing list. 303 304Trace Analysis Tools 305-------------------- 306 307We currently have the beginnings of a trace analysis tool in LLVM, which can be 308found in the ``tools/llvm-xray`` directory. The ``llvm-xray`` tool currently 309supports the following subcommands: 310 311- ``extract``: Extract the instrumentation map from a binary, and return it as 312 YAML. 313- ``account``: Performs basic function call accounting statistics with various 314 options for sorting, and output formats (supports CSV, YAML, and 315 console-friendly TEXT). 316- ``convert``: Converts an XRay log file from one format to another. We can 317 convert from binary XRay traces (both basic and FDR mode) to YAML, 318 `flame-graph <https://github.com/brendangregg/FlameGraph>`_ friendly text 319 formats, as well as `Chrome Trace Viewer (catapult) 320 <https://github.com/catapult-project/catapult>` formats. 321- ``graph``: Generates a DOT graph of the function call relationships between 322 functions found in an XRay trace. 323- ``stack``: Reconstructs function call stacks from a timeline of function 324 calls in an XRay trace. 325 326These subcommands use various library components found as part of the XRay 327libraries, distributed with the LLVM distribution. These are: 328 329- ``llvm/XRay/Trace.h`` : A trace reading library for conveniently loading 330 an XRay trace of supported forms, into a convenient in-memory representation. 331 All the analysis tools that deal with traces use this implementation. 332- ``llvm/XRay/Graph.h`` : A semi-generic graph type used by the graph 333 subcommand to conveniently represent a function call graph with statistics 334 associated with edges and vertices. 335- ``llvm/XRay/InstrumentationMap.h``: A convenient tool for analyzing the 336 instrumentation map in XRay-instrumented object files and binaries. The 337 ``extract`` and ``stack`` subcommands uses this particular library. 338 339 340Minimizing Binary Size 341---------------------- 342 343XRay supports several different instrumentation points including ``function-entry``, 344``function-exit``, ``custom``, and ``typed`` points. These can be enabled individually 345using the ``-fxray-instrumentation-bundle=`` flag. For example if you only wanted to 346instrument function entry and custom points you could specify: 347 348:: 349 350 clang -fxray-instrument -fxray-instrumentation-bundle=function-entry,custom ... 351 352This will omit the other sled types entirely, reducing the binary size. You can also 353instrument just a sampled subset of functions using instrumentation groups. 354For example, to instrument only a quarter of available functions invoke: 355 356:: 357 358 clang -fxray-instrument -fxray-function-groups=4 359 360A subset will be chosen arbitrarily based on a hash of the function name. To sample a 361different subset you can specify ``-fxray-selected-function-group=`` with a group number 362in the range of 0 to ``xray-function-groups`` - 1. Together these options could be used 363to produce multiple binaries with different instrumented subsets. If all you need is 364runtime control over which functions are being traced at any given time it is better 365to selectively patch and unpatch the individual functions you need using the XRay 366Runtime Library's ``__xray_patch_function()`` method. 367 368Future Work 369=========== 370 371There are a number of ongoing efforts for expanding the toolset building around 372the XRay instrumentation system. 373 374Trace Analysis Tools 375-------------------- 376 377- Work is in progress to integrate with or develop tools to visualize findings 378 from an XRay trace. Particularly, the ``stack`` tool is being expanded to 379 output formats that allow graphing and exploring the duration of time in each 380 call stack. 381- With a large instrumented binary, the size of generated XRay traces can 382 quickly become unwieldy. We are working on integrating pruning techniques and 383 heuristics for the analysis tools to sift through the traces and surface only 384 relevant information. 385 386More Platforms 387-------------- 388 389We're looking forward to contributions to port XRay to more architectures and 390operating systems. 391 392.. References... 393 394.. _`XRay whitepaper`: http://research.google.com/pubs/pub45287.html 395 396