1Tracing with Intel Processor Trace 2================================== 3 4.. contents:: 5 :local: 6 7Intel PT is a technology available in modern Intel CPUs that allows efficient 8tracing of all the instructions executed by a process. 9LLDB can collect traces and dump them using its symbolication stack. 10You can read more here 11https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace. 12 13Prerequisites 14------------- 15 16Confirm that your CPU supports Intel PT 17(see https://www.intel.com/content/www/us/en/support/articles/000056730/processors.html) 18and that your operating system is Linux. 19 20Check for the existence of this particular file on your Linux system 21 22:: 23 24 $ cat /sys/bus/event_source/devices/intel_pt/type 25 26The output should be a number. Otherwise, try upgrading your kernel. 27 28 29Build Instructions 30------------------ 31 32Clone and build the low level Intel PT 33decoder library [LibIPT library](https://github.com/intel/libipt). 34:: 35 36 $ git clone git@github.com:intel/libipt.git 37 $ mkdir libipt-build 38 $ cmake -S libipt -B libipt-build 39 $ cd libipt-build 40 $ make 41 42This will generate a few files in the ``<libipt-build>/lib`` 43and ``<libipt-build>/libipt/include`` directories. 44 45Configure and build LLDB with Intel PT support 46 47:: 48 49 $ cmake \ 50 -DLLDB_BUILD_INTEL_PT=ON \ 51 -DLIBIPT_INCLUDE_PATH="<libipt-build>/libipt/include" \ 52 -DLIBIPT_LIBRARY_PATH="<libipt-build>/lib" \ 53 ... other common configuration parameters 54 55:: 56 57 $ cd <lldb-build> && ninja lldb lldb-server # if using Ninja 58 59 60How to Use 61---------- 62 63When you are debugging a process, you can turn on intel-pt tracing, 64which will “record” all the instructions that the process will execute. 65After turning it on, you can continue debugging, and at any breakpoint, 66you can inspect the instruction list. 67 68For example: 69 70:: 71 72 lldb <target> 73 > b main 74 > run 75 > process trace start # start tracing on all threads, including future ones 76 # keep debugging until you hit a breakpoint 77 78 > thread trace dump instructions 79 # this should output something like 80 81 thread #2: tid = 2861133, total instructions = 5305673 82 libc.so.6`__GI___libc_read + 45 at read.c:25:1 83 [4962255] 0x00007fffeb64c63d subq $0x10, %rsp 84 [4962256] 0x00007fffeb64c641 movq %rdi, -0x18(%rbp) 85 libc.so.6`__GI___libc_read + 53 [inlined] __libc_read at read.c:26:10 86 [4962257] 0x00007fffeb64c645 callq 0x7fffeb66b640 ; __libc_enable_asynccancel 87 libc.so.6`__libc_enable_asynccancel 88 [4962258] 0x00007fffeb66b640 movl %fs:0x308, %eax 89 libc.so.6`__libc_enable_asynccancel + 8 90 [4962259] 0x00007fffeb66b648 movl %eax, %r11d 91 92 # you can keep pressing ENTER to see more and more instructions 93 94The number between brackets is the instruction index, 95and by default the current thread will be picked. 96 97Configuring the trace size 98-------------------------- 99 100The CPU stores the instruction list in a compressed format in a ring buffer, 101which keeps the latest information. 102By default, LLDB uses a buffer of 4KB per thread, 103but you can change it by running. 104The size must be a power of 2 and at least 4KB. 105 106:: 107 108 thread trace start all -s <size_in_bytes> 109 110For reference, a 1MB trace buffer can easily store around 5M instructions. 111 112Printing more instructions 113-------------------------- 114 115If you want to dump more instructions at a time, you can run 116 117:: 118 119 thread trace dump instructions -c <count> 120 121Printing the instructions of another thread 122------------------------------------------- 123 124By default the current thread will be picked when dumping instructions, 125but you can do 126 127:: 128 129 thread trace dump instructions <#thread index> 130 #e.g. 131 thread trace dump instructions 8 132 133to select another thread. 134 135Crash Analysis 136-------------- 137 138What if you are debugging + tracing a process that crashes? 139Then you can just do 140 141:: 142 143 thread trace dump instructions 144 145To inspect how it crashed! There's nothing special that you need to do. 146For example 147 148:: 149 150 * thread #1, name = 'a.out', stop reason = signal SIGFPE: integer divide by zero 151 frame #0: 0x00000000004009f1 a.out`main at main.cpp:8:14 152 6 int x; 153 7 cin >> x; 154 -> 8 cout << 12 / x << endl; 155 9 return 0; 156 10 } 157 (lldb) thread trace dump instructions -c 5 158 thread #1: tid = 604302, total instructions = 8388 159 libstdc++.so.6`std::istream::operator>>(int&) + 181 160 [8383] 0x00007ffff7b41665 popq %rbp 161 [8384] 0x00007ffff7b41666 retq 162 a.out`main + 66 at main.cpp:8:14 163 [8385] 0x00000000004009e8 movl -0x4(%rbp), %ecx 164 [8386] 0x00000000004009eb movl $0xc, %eax 165 [8387] 0x00000000004009f0 cltd 166 167.. note:: 168 At this moment, we are not including the failed instruction in the trace, 169 but in the future we might do it for readability. 170 171 172Offline Trace Analysis 173---------------------- 174 175It's also possible to record a trace using a custom Intel PT collector 176and decode + symbolicate the trace using LLDB. 177For that, the command trace load is useful. 178In order to use trace load, you need to first create a JSON file with 179the definition of the trace session. 180For example 181 182:: 183 184 { 185 "type": "intel-pt", 186 "cpuInfo": { 187 "vendor": "GenuineIntel", 188 "family": 6, 189 "model": 79, 190 "stepping": 1 191 }, 192 "processes": [ 193 { 194 "pid": 815455, 195 "triple": "x86_64-*-linux", 196 "threads": [ 197 { 198 "tid": 815455, 199 "iptTrace": "trace.file" # raw thread-specific trace from the AUX buffer 200 } 201 ], 202 "modules": [ # this are all the shared libraries + the main executable 203 { 204 "file": "a.out", # optional if it's the same as systemPath 205 "systemPath": "a.out", 206 "loadAddress": 4194304, 207 }, 208 { 209 "file": "libfoo.so", 210 "systemPath": "/usr/lib/libfoo.so", 211 "loadAddress": "0x00007ffff7bd9000", 212 }, 213 { 214 "systemPath": "libbar.so", 215 "loadAddress": "0x00007ffff79d7000", 216 } 217 ] 218 } 219 ] 220 } 221 222You can see the full schema by typing 223 224:: 225 226 trace schema intel-pt 227 228The JSON file mainly contains all the shared libraries that 229were part of the traced process, along with their memory load address. 230If the analysis is done on the same computer where the traces were obtained, 231it's enough to use the “systemPath” field. 232If the analysis is done on a different machines, these files need to be 233copied over and the “file” field should point to the 234location of the file relative to the JSON file. 235Once you have the JSON file and the module files in place, you can simple run 236 237:: 238 239 lldb 240 > trace load /path/to/json 241 > thread trace dump instructions <optional thread index> 242 243Then it's like in the live session case 244 245References 246---------- 247 248- Original RFC document_ for this feature. 249- Some details about how Meta is using Intel Processor Trace can be found in this blog_ post. 250 251.. _document: https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI 252.. _blog: https://engineering.fb.com/2021/04/27/developer-tools/reverse-debugging/ 253