xref: /llvm-project/lldb/docs/use/intel_pt.rst (revision f47914a7cd733751d6ae4dcb9c240520c34ec492)
1Tracing with Intel Processor Trace
2==================================
3
4Intel PT is a technology available in modern Intel CPUs that allows efficient
5tracing of all the instructions executed by a process.
6LLDB can collect traces and dump them using its symbolication stack.
7You can read more here
8https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace.
9
10Prerequisites
11-------------
12
13Confirm that your CPU supports Intel PT
14(see https://www.intel.com/content/www/us/en/support/articles/000056730/processors.html)
15and that your operating system is Linux.
16
17Check for the existence of this particular file on your Linux system
18
19::
20
21  $ cat /sys/bus/event_source/devices/intel_pt/type
22
23The output should be a number. Otherwise, try upgrading your kernel.
24
25
26Build Instructions
27------------------
28
29Clone and build the low level Intel PT
30decoder library `LibIPT library <https://github.com/intel/libipt>`_.
31::
32
33  $ git clone git@github.com:intel/libipt.git
34  $ mkdir libipt-build
35  $ cmake -S libipt -B libipt-build
36  $ cd libipt-build
37  $ make
38
39This will generate a few files in the ``<libipt-build>/lib``
40and ``<libipt-build>/libipt/include`` directories.
41
42Configure and build LLDB with Intel PT support
43
44::
45
46  $ cmake \
47      -DLLDB_BUILD_INTEL_PT=ON \
48      -DLIBIPT_INCLUDE_PATH="<libipt-build>/libipt/include" \
49      -DLIBIPT_LIBRARY_PATH="<libipt-build>/lib" \
50      ... other common configuration parameters
51
52::
53
54  $ cd <lldb-build> && ninja lldb lldb-server # if using Ninja
55
56
57How to Use
58----------
59
60When you are debugging a process, you can turn on intel-pt tracing,
61which will “record” all the instructions that the process will execute.
62After turning it on, you can continue debugging, and at any breakpoint,
63you can inspect the instruction list.
64
65For example:
66
67::
68
69  lldb <target>
70  > b main
71  > run
72  > process trace start # start tracing on all threads, including future ones
73  # keep debugging until you hit a breakpoint
74
75  > thread trace dump instructions
76  # this should output something like
77
78  thread #2: tid = 2861133, total instructions = 5305673
79    libc.so.6`__GI___libc_read + 45 at read.c:25:1
80      [4962255] 0x00007fffeb64c63d    subq   $0x10, %rsp
81      [4962256] 0x00007fffeb64c641    movq   %rdi, -0x18(%rbp)
82    libc.so.6`__GI___libc_read + 53 [inlined] __libc_read at read.c:26:10
83      [4962257] 0x00007fffeb64c645    callq  0x7fffeb66b640            ; __libc_enable_asynccancel
84    libc.so.6`__libc_enable_asynccancel
85      [4962258] 0x00007fffeb66b640    movl   %fs:0x308, %eax
86    libc.so.6`__libc_enable_asynccancel + 8
87      [4962259] 0x00007fffeb66b648    movl   %eax, %r11d
88
89  # you can keep pressing ENTER to see more and more instructions
90
91The number between brackets is the instruction index,
92and by default the current thread will be picked.
93
94Configuring the trace size
95--------------------------
96
97The CPU stores the instruction list in a compressed format in a ring buffer,
98which keeps the latest information.
99By default, LLDB uses a buffer of 4KB per thread,
100but you can change it by running.
101The size must be a power of 2 and at least 4KB.
102
103::
104
105  thread trace start all -s <size_in_bytes>
106
107For reference, a 1MB trace buffer can easily store around 5M instructions.
108
109Printing more instructions
110--------------------------
111
112If you want to dump more instructions at a time, you can run
113
114::
115
116  thread trace dump instructions -c <count>
117
118Printing the instructions of another thread
119-------------------------------------------
120
121By default the current thread will be picked when dumping instructions,
122but you can do
123
124::
125
126  thread trace dump instructions <#thread index>
127  #e.g.
128  thread trace dump instructions 8
129
130to select another thread.
131
132Crash Analysis
133--------------
134
135What if you are debugging + tracing a process that crashes?
136Then you can just do
137
138::
139
140  thread trace dump instructions
141
142To inspect how it crashed! There's nothing special that you need to do.
143For example
144
145::
146
147    * thread #1, name = 'a.out', stop reason = signal SIGFPE: integer divide by zero
148        frame #0: 0x00000000004009f1 a.out`main at main.cpp:8:14
149      6       int x;
150      7       cin >> x;
151   -> 8       cout << 12 / x << endl;
152      9       return 0;
153      10  }
154    (lldb) thread trace dump instructions -c 5
155    thread #1: tid = 604302, total instructions = 8388
156      libstdc++.so.6`std::istream::operator>>(int&) + 181
157        [8383] 0x00007ffff7b41665    popq   %rbp
158        [8384] 0x00007ffff7b41666    retq
159      a.out`main + 66 at main.cpp:8:14
160        [8385] 0x00000000004009e8    movl   -0x4(%rbp), %ecx
161        [8386] 0x00000000004009eb    movl   $0xc, %eax
162        [8387] 0x00000000004009f0    cltd
163
164.. note::
165  At this moment, we are not including the failed instruction in the trace,
166  but in the future we might do it for readability.
167
168
169Offline Trace Analysis
170----------------------
171
172It's also possible to record a trace using a custom Intel PT collector
173and decode + symbolicate the trace using LLDB.
174For that, the command trace load is useful.
175In order to use trace load, you need to first create a JSON file with
176the definition of the trace session.
177For example
178
179::
180
181  {
182    "type": "intel-pt",
183    "cpuInfo": {
184      "vendor": "GenuineIntel",
185      "family": 6,
186      "model": 79,
187      "stepping": 1
188    },
189    "processes": [
190      {
191        "pid": 815455,
192        "triple": "x86_64-*-linux",
193        "threads": [
194          {
195            "tid": 815455,
196            "iptTrace": "trace.file" # raw thread-specific trace from the AUX buffer
197          }
198        ],
199        "modules": [ # this are all the shared libraries + the main executable
200          {
201            "file": "a.out", # optional if it's the same as systemPath
202            "systemPath": "a.out",
203            "loadAddress": 4194304,
204          },
205          {
206            "file": "libfoo.so",
207            "systemPath": "/usr/lib/libfoo.so",
208            "loadAddress": "0x00007ffff7bd9000",
209          },
210          {
211            "systemPath": "libbar.so",
212            "loadAddress": "0x00007ffff79d7000",
213          }
214        ]
215      }
216    ]
217  }
218
219You can see the full schema by typing
220
221::
222
223  trace schema intel-pt
224
225The JSON file mainly contains all the shared libraries that
226were part of the traced process, along with their memory load address.
227If the analysis is done on the same computer where the traces were obtained,
228it's enough to use the “systemPath” field.
229If the analysis is done on a different machines, these files need to be
230copied over and the “file” field should point to the
231location of the file relative to the JSON file.
232Once you have the JSON file and the module files in place, you can simple run
233
234::
235
236  lldb
237  > trace load /path/to/json
238  > thread trace dump instructions <optional thread index>
239
240Then it's like in the live session case
241
242References
243----------
244
245- Original RFC document_ for this feature.
246- Some details about how Meta is using Intel Processor Trace can be found in this blog_ post.
247
248.. _document: https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI
249.. _blog: https://engineering.fb.com/2021/04/27/developer-tools/reverse-debugging/
250