1Tracing with Intel Processor Trace 2================================== 3 4Intel PT is a technology available in modern Intel CPUs that allows efficient 5tracing of all the instructions executed by a process. 6LLDB can collect traces and dump them using its symbolication stack. 7You can read more here 8https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace. 9 10Prerequisites 11------------- 12 13Confirm that your CPU supports Intel PT 14(see https://www.intel.com/content/www/us/en/support/articles/000056730/processors.html) 15and that your operating system is Linux. 16 17Check for the existence of this particular file on your Linux system 18 19:: 20 21 $ cat /sys/bus/event_source/devices/intel_pt/type 22 23The output should be a number. Otherwise, try upgrading your kernel. 24 25 26Build Instructions 27------------------ 28 29Clone and build the low level Intel PT 30decoder library `LibIPT library <https://github.com/intel/libipt>`_. 31:: 32 33 $ git clone git@github.com:intel/libipt.git 34 $ mkdir libipt-build 35 $ cmake -S libipt -B libipt-build 36 $ cd libipt-build 37 $ make 38 39This will generate a few files in the ``<libipt-build>/lib`` 40and ``<libipt-build>/libipt/include`` directories. 41 42Configure and build LLDB with Intel PT support 43 44:: 45 46 $ cmake \ 47 -DLLDB_BUILD_INTEL_PT=ON \ 48 -DLIBIPT_INCLUDE_PATH="<libipt-build>/libipt/include" \ 49 -DLIBIPT_LIBRARY_PATH="<libipt-build>/lib" \ 50 ... other common configuration parameters 51 52:: 53 54 $ cd <lldb-build> && ninja lldb lldb-server # if using Ninja 55 56 57How to Use 58---------- 59 60When you are debugging a process, you can turn on intel-pt tracing, 61which will “record” all the instructions that the process will execute. 62After turning it on, you can continue debugging, and at any breakpoint, 63you can inspect the instruction list. 64 65For example: 66 67:: 68 69 lldb <target> 70 > b main 71 > run 72 > process trace start # start tracing on all threads, including future ones 73 # keep debugging until you hit a breakpoint 74 75 > thread trace dump instructions 76 # this should output something like 77 78 thread #2: tid = 2861133, total instructions = 5305673 79 libc.so.6`__GI___libc_read + 45 at read.c:25:1 80 [4962255] 0x00007fffeb64c63d subq $0x10, %rsp 81 [4962256] 0x00007fffeb64c641 movq %rdi, -0x18(%rbp) 82 libc.so.6`__GI___libc_read + 53 [inlined] __libc_read at read.c:26:10 83 [4962257] 0x00007fffeb64c645 callq 0x7fffeb66b640 ; __libc_enable_asynccancel 84 libc.so.6`__libc_enable_asynccancel 85 [4962258] 0x00007fffeb66b640 movl %fs:0x308, %eax 86 libc.so.6`__libc_enable_asynccancel + 8 87 [4962259] 0x00007fffeb66b648 movl %eax, %r11d 88 89 # you can keep pressing ENTER to see more and more instructions 90 91The number between brackets is the instruction index, 92and by default the current thread will be picked. 93 94Configuring the trace size 95-------------------------- 96 97The CPU stores the instruction list in a compressed format in a ring buffer, 98which keeps the latest information. 99By default, LLDB uses a buffer of 4KB per thread, 100but you can change it by running. 101The size must be a power of 2 and at least 4KB. 102 103:: 104 105 thread trace start all -s <size_in_bytes> 106 107For reference, a 1MB trace buffer can easily store around 5M instructions. 108 109Printing more instructions 110-------------------------- 111 112If you want to dump more instructions at a time, you can run 113 114:: 115 116 thread trace dump instructions -c <count> 117 118Printing the instructions of another thread 119------------------------------------------- 120 121By default the current thread will be picked when dumping instructions, 122but you can do 123 124:: 125 126 thread trace dump instructions <#thread index> 127 #e.g. 128 thread trace dump instructions 8 129 130to select another thread. 131 132Crash Analysis 133-------------- 134 135What if you are debugging + tracing a process that crashes? 136Then you can just do 137 138:: 139 140 thread trace dump instructions 141 142To inspect how it crashed! There's nothing special that you need to do. 143For example 144 145:: 146 147 * thread #1, name = 'a.out', stop reason = signal SIGFPE: integer divide by zero 148 frame #0: 0x00000000004009f1 a.out`main at main.cpp:8:14 149 6 int x; 150 7 cin >> x; 151 -> 8 cout << 12 / x << endl; 152 9 return 0; 153 10 } 154 (lldb) thread trace dump instructions -c 5 155 thread #1: tid = 604302, total instructions = 8388 156 libstdc++.so.6`std::istream::operator>>(int&) + 181 157 [8383] 0x00007ffff7b41665 popq %rbp 158 [8384] 0x00007ffff7b41666 retq 159 a.out`main + 66 at main.cpp:8:14 160 [8385] 0x00000000004009e8 movl -0x4(%rbp), %ecx 161 [8386] 0x00000000004009eb movl $0xc, %eax 162 [8387] 0x00000000004009f0 cltd 163 164.. note:: 165 At this moment, we are not including the failed instruction in the trace, 166 but in the future we might do it for readability. 167 168 169Offline Trace Analysis 170---------------------- 171 172It's also possible to record a trace using a custom Intel PT collector 173and decode + symbolicate the trace using LLDB. 174For that, the command trace load is useful. 175In order to use trace load, you need to first create a JSON file with 176the definition of the trace session. 177For example 178 179:: 180 181 { 182 "type": "intel-pt", 183 "cpuInfo": { 184 "vendor": "GenuineIntel", 185 "family": 6, 186 "model": 79, 187 "stepping": 1 188 }, 189 "processes": [ 190 { 191 "pid": 815455, 192 "triple": "x86_64-*-linux", 193 "threads": [ 194 { 195 "tid": 815455, 196 "iptTrace": "trace.file" # raw thread-specific trace from the AUX buffer 197 } 198 ], 199 "modules": [ # this are all the shared libraries + the main executable 200 { 201 "file": "a.out", # optional if it's the same as systemPath 202 "systemPath": "a.out", 203 "loadAddress": 4194304, 204 }, 205 { 206 "file": "libfoo.so", 207 "systemPath": "/usr/lib/libfoo.so", 208 "loadAddress": "0x00007ffff7bd9000", 209 }, 210 { 211 "systemPath": "libbar.so", 212 "loadAddress": "0x00007ffff79d7000", 213 } 214 ] 215 } 216 ] 217 } 218 219You can see the full schema by typing 220 221:: 222 223 trace schema intel-pt 224 225The JSON file mainly contains all the shared libraries that 226were part of the traced process, along with their memory load address. 227If the analysis is done on the same computer where the traces were obtained, 228it's enough to use the “systemPath” field. 229If the analysis is done on a different machines, these files need to be 230copied over and the “file” field should point to the 231location of the file relative to the JSON file. 232Once you have the JSON file and the module files in place, you can simple run 233 234:: 235 236 lldb 237 > trace load /path/to/json 238 > thread trace dump instructions <optional thread index> 239 240Then it's like in the live session case 241 242References 243---------- 244 245- Original RFC document_ for this feature. 246- Some details about how Meta is using Intel Processor Trace can be found in this blog_ post. 247 248.. _document: https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI 249.. _blog: https://engineering.fb.com/2021/04/27/developer-tools/reverse-debugging/ 250