[BOLT] Detect Linux kernel version if the binary is a Linux kernel (#119088)

This makes it easier to handle differences (e.g. of exception table
entry size) between versions of Linux kernel

[BOLT][Linux] Refactor reading of PC-relative addresses. NFCI (#120491)

Fix evaluation order problem identified in
https://github.com/llvm/llvm-project/pull/119088.

[BOLT][Linux] Support ORC for alternative instructions (#96709)

Alternative instruction sequences in the Linux kernel can modify the
stack and thus they need their own ORC unwind entries. Since the

[BOLT][Linux] Support ORC for alternative instructions (#96709)

Alternative instruction sequences in the Linux kernel can modify the
stack and thus they need their own ORC unwind entries. Since there's
only one ORC table, it has to be "shared" among multiple instruction
sequences. The kernel achieves this by putting a restriction on
instruction boundaries. If ORC state changes at a given IP, only one of
the alternative sequences can have an instruction starting/ending at
this IP. Then, developers can insert NOPs to guarantee the above
requirement is met.

The most common use of ORC with alternatives is "pushf; pop %rax"
sequence used for paravirtualization. Note that newer kernel versions
no longer use .parainstructions; instead, they utilize alternatives for
the same purpose.

Before we implement a better support for alternatives, we can safely
skip ORC entries associated with them.

Fixes #87052.

show more ...

[BOLT] Skip optimization of functions with alt instructions (#95172)

Alternative instructions in the Linux kernel may modify control flow in
a function. As such, it is unsafe to optimize functions

[BOLT] Skip optimization of functions with alt instructions (#95172)

Alternative instructions in the Linux kernel may modify control flow in
a function. As such, it is unsafe to optimize functions with alternative
instructions until we properly support CFG alternatives.

Previously, we marked functions with alt instructions before the
emission, but that could be too late if we remove or replace
instructions with alternatives. We could have marked functions as
non-simple immediately after reading .altinstructions, but it's nice to
be able to view functions after CFG is built. Thus assign the non-simple
status after building CFG.

show more ...

[BOLT] Fix duplicate diagnostic message (#95167)

Print .altinstructions parsing stats only once.

[BOLT] Add auto parsing for Linux kernel .altinstructions (#95068)

.altinstructions section contains a list of structures where fields can
have different sizes while other fields could be present o

[BOLT] Add auto parsing for Linux kernel .altinstructions (#95068)

.altinstructions section contains a list of structures where fields can
have different sizes while other fields could be present or not
depending on the kernel version. Add automatic detection of such
variations and use it by default. The user can still overwrite the
automatic detection with `--alt-inst-has-padlen` and
`--alt-inst-feature-size` options.

show more ...

[BOLT][NFCI] Fix return type of BC::getSignedValueAtAddress (#91664)

[BOLT] Add support for Linux kernel .smp_locks section (#90798)

Parse .smp_locks section entries and create fixups that are going to be
used to update the section before the binary emission.

[BOLT] Register Linux kernel dynamic branch offsets (#90677)

To match profile data to code we need to know branch instruction offsets
within a function. For this reason, we mark branches with the "

[BOLT] Register Linux kernel dynamic branch offsets (#90677)

To match profile data to code we need to know branch instruction offsets
within a function. For this reason, we mark branches with the "Offset"
annotation while disassembling the code. However, _dynamic_ branches in
the Linux kernel could be NOPs in disassembled code, and we ignore them
while adding annotations. We need to explicitly add the "Offset"
annotation while creating dynamic branches.

Note that without this change, `getInstructionAtOffset()` would still
return a branch instruction if the offset matched the last instruction
in a basic block (and the profile data was matched correctly). However,
the function failed for cases when the searched instruction was followed
by an unconditional jump. "Offset" annotation solves this case.

show more ...

[BOLT] Add split function support for the Linux kernel (#90541)

While rewriting the Linux kernel, we try to fit optimized functions into
their original boundaries. When a function becomes larger, w

[BOLT] Add split function support for the Linux kernel (#90541)

While rewriting the Linux kernel, we try to fit optimized functions into
their original boundaries. When a function becomes larger, we skip it
during the rewrite and end up with less than optimal code layout. To
overcome that issue, add support for --split-function option so that hot
part of the function could be fit into the original space. The cold part
should go to reserved space in the binary.

show more ...

[BOLT] Fix a warning

This patch fixes:

bolt/lib/Rewrite/LinuxKernelRewriter.cpp:855:12: error: variable
'PrevIP' set but not used [-Werror,-Wunused-but-set-variable]

[BOLT] Add ORC validation for the Linux kernel (#90660)

The Linux kernel expects ORC tables to be sorted by IP address (for
binary search to work). Add a post-emit pass in LinuxKernelRewriter that

[BOLT] Add ORC validation for the Linux kernel (#90660)

The Linux kernel expects ORC tables to be sorted by IP address (for
binary search to work). Add a post-emit pass in LinuxKernelRewriter that
validates the written .orc_unwind_ip against that expectation.

show more ...

[BOLT] Add rewriting support for Linux kernel __bug_table (#86908)

Update instruction locations in the __bug_table section after new code
is emitted. If an instruction with associated bug ID was de

[BOLT] Add rewriting support for Linux kernel __bug_table (#86908)

Update instruction locations in the __bug_table section after new code
is emitted. If an instruction with associated bug ID was deleted,
overwrite its location with zero.

show more ...

[BOLT] Skip functions with unsupported Linux kernel features (#86345)

Do not overwrite functions with alternative and paravirtual instructions
until a proper update support is implemented.

[BOLT] Fix unused variable warnings

This patch fixes:

bolt/lib/Rewrite/LinuxKernelRewriter.cpp:1664:20: error: unused
variable 'TargetAddress' [-Werror,-Wunused-variable]

bolt/lib/Rewrite/Li

[BOLT] Fix unused variable warnings

This patch fixes:

bolt/lib/Rewrite/LinuxKernelRewriter.cpp:1664:20: error: unused
variable 'TargetAddress' [-Werror,-Wunused-variable]

bolt/lib/Rewrite/LinuxKernelRewriter.cpp:1666:20: error: unused
variable 'KeyAddress' [-Werror,-Wunused-variable]

show more ...

[BOLT] Add support for Linux kernel static keys jump table (#86090)

Runtime code modification used by static keys is the most ubiquitous
self-modifying feature of the Linux kernel. The idea is to t

[BOLT] Add support for Linux kernel static keys jump table (#86090)

Runtime code modification used by static keys is the most ubiquitous
self-modifying feature of the Linux kernel. The idea is to to eliminate
the condition check and associated conditional jump on a hot path if
that condition (based on a boolean value of a static key) does not
change often. Whenever they condition changes, the kernel runtime
modifies all code paths associated with that key flipping the code
between nop and (unconditional) jump.

show more ...

[BOLT] Add support for Linux kernel PCI fixup section (#84982)

.pci_fixup section contains a table with entries allowing to invoke a
fixup hook whenever a problem is encountered with a PCI device.

[BOLT] Add support for Linux kernel PCI fixup section (#84982)

.pci_fixup section contains a table with entries allowing to invoke a
fixup hook whenever a problem is encountered with a PCI device. The
hookup code typically points to the start of a function. As we are not
relocating functions in the kernel (at least not yet), verify this
assumption while reading the table and ignore any functions with a fixup
code in the middle.

show more ...

[BOLT] Properly propagate Cursor errors (#84378)

Handle out-of-bounds reading errors correctly in LinuxKernelRewriter.

[BOLT] Add reading support for Linux kernel .altinstructions section (#84283)

Read .altinstructions and annotate instructions that have alternative
sequences with "AltInst" annotation. Note that so

[BOLT] Add reading support for Linux kernel .altinstructions section (#84283)

Read .altinstructions and annotate instructions that have alternative
sequences with "AltInst" annotation. Note that some instructions may
have more than one alternatives, in which case they will have multiple
annotations in the form "AltInst", "AltInst2", "AltInst3", etc.

show more ...

[BOLT] Add reading support for Linux kernel __bug_table section (#84082)

Read __bug_table section and annotate ud2 instructions with a
corresponding bug entry ID.

[BOLT] Add reading support for Linux kernel .parainstructions section (#83965)

Read .parainstruction section and mark call instructions with ParaSite
annotations.

[BOLT] Add reading support for Linux kernel exception table (#83100)

Read Linux exception table and ignore functions with exceptions for now.
Proper support requires an introduction of new control

[BOLT] Add reading support for Linux kernel exception table (#83100)

Read Linux exception table and ignore functions with exceptions for now.
Proper support requires an introduction of new control flow since some
instructions with memory access can cause a control flow change.

Hence looking at disassembly or CFG with exceptions annotations is
valuable for code analysis, delay marking functions with exceptions as
non-simple until immediately before emitting the code.

show more ...

[BOLT] Refactor interface for instruction labels. NFCI (#83209)

To avoid accidentally setting the label twice for the same instruction,
which can lead to a "lost" label, introduce getOrSetInstLabel

[BOLT] Refactor interface for instruction labels. NFCI (#83209)

To avoid accidentally setting the label twice for the same instruction,
which can lead to a "lost" label, introduce getOrSetInstLabel()
function. Rename existing functions to getInstLabel()/setInstLabel() to
make it explicit that they operate on instruction labels. Add an
assertion in setInstLabel() that the instruction did not have a prior
label set.

show more ...

[BOLT][NFC] Switch logging in LinuxKernelRewriter (#82195)

Use journaling streams introduced in #81524 for LinuxKernelRewriter.

[BOLT] Add support for Linux kernel static calls table (#82072)

Static calls are calls that are getting patched during runtime. Hence,
for every such call the kernel runtime needs the location of t

[BOLT] Add support for Linux kernel static calls table (#82072)

Static calls are calls that are getting patched during runtime. Hence,
for every such call the kernel runtime needs the location of the call or
jmp instruction that will be patched. Instruction locations together
with a corresponding key are stored in the static call site table. As
BOLT rewrites these instructions it needs to update the table.

show more ...