[BOLT][NFC] Clean includes, add license headers (#87200)

[BOLT] Don't terminate on trap instruction for Linux kernel (#87021)

Under normal circumstances, we terminate basic blocks on a trap
instruction. However, Linux kernel may resume execution after hi

[BOLT] Don't terminate on trap instruction for Linux kernel (#87021)

Under normal circumstances, we terminate basic blocks on a trap
instruction. However, Linux kernel may resume execution after hitting a
trap (ud2 on x86). Thus, we introduce "--terminal-trap" option that will
specify if the trap instruction should terminate the control flow. The
option is on by default except for the Linux kernel mode when it's off.

show more ...

[BOLT] Add support for Linux kernel static keys jump table (#86090)

Runtime code modification used by static keys is the most ubiquitous
self-modifying feature of the Linux kernel. The idea is to t

[BOLT] Add support for Linux kernel static keys jump table (#86090)

Runtime code modification used by static keys is the most ubiquitous
self-modifying feature of the Linux kernel. The idea is to to eliminate
the condition check and associated conditional jump on a hot path if
that condition (based on a boolean value of a static key) does not
change often. Whenever they condition changes, the kernel runtime
modifies all code paths associated with that key flipping the code
between nop and (unconditional) jump.

show more ...

[BOLT] Refactor interface for instruction labels. NFCI (#83209)

To avoid accidentally setting the label twice for the same instruction,
which can lead to a "lost" label, introduce getOrSetInstLabel

[BOLT] Refactor interface for instruction labels. NFCI (#83209)

To avoid accidentally setting the label twice for the same instruction,
which can lead to a "lost" label, introduce getOrSetInstLabel()
function. Rename existing functions to getInstLabel()/setInstLabel() to
make it explicit that they operate on instruction labels. Add an
assertion in setInstLabel() that the instruction did not have a prior
label set.

show more ...

[BOLT] Make instruction size a first-class annotation (#72167)

When NOP instructions are used to reserve space in the code, e.g. for
patching, it becomes critical to preserve their original size wh

[BOLT] Make instruction size a first-class annotation (#72167)

When NOP instructions are used to reserve space in the code, e.g. for
patching, it becomes critical to preserve their original size while
emitting the code. On x86, we rely on "Size" annotation for NOP
instructions size, as the original instruction size is lost in the
disassembly/assembly process.

This change makes instruction size a first-class annotation and is
affectively NFCI. A follow-up diff will use the annotation for code
emission.

show more ...

[BOLT] Use direct storage for Label annotations. NFCI. (#70147)

Store the Label annotation directly in the operand and avoid the extra
allocation and indirection overheads associated with MCSimpleA

[BOLT] Use direct storage for Label annotations. NFCI. (#70147)

Store the Label annotation directly in the operand and avoid the extra
allocation and indirection overheads associated with MCSimpleAnnotation.

show more ...

[BOLT] Modify MCPlus annotation internals. NFCI. (#70412)

When annotating MCInst instructions, attach extra annotation operands
directly to the annotated instruction, instead of attaching them to a

[BOLT] Modify MCPlus annotation internals. NFCI. (#70412)

When annotating MCInst instructions, attach extra annotation operands
directly to the annotated instruction, instead of attaching them to an
instruction pointed to by a special kInst operand.

With this change, it's no longer necessary to allocate MCInst and most
of the first-class annotations come with free memory as currently MCInst
is declared with:

SmallVector<MCOperand, 10> Operands;

i.e. more operands than are normally being used.

We still create a kInst operand with a nullptr instruction value to
designate the beginning of annotation operands. However, this special
operand might not be needed if we can rely on MCInstrDesc::NumOperands.

show more ...

[BOLT] Reduce the number of emitted symbols. NFCI. (#70175)

We emit a symbol before an instruction for a number of reasons, e.g. for
tracking LocSyms, debug line, or if the instruction has a label

[BOLT] Reduce the number of emitted symbols. NFCI. (#70175)

We emit a symbol before an instruction for a number of reasons, e.g. for
tracking LocSyms, debug line, or if the instruction has a label
annotation. Currently, we may emit multiple symbols per instruction.

Reuse the same label instead of creating and emitting new ones when
possible. I'm planning to refactor EH labels as well in a separate diff.

Change getLabel() to return a pointer instead of std::optional<> since
an empty label should be treated identically to no label.

show more ...

[BOLT][NFC] Add allocator id to MCPlusBuilder::setLabel (#68707)

This will be needed for some RISC-V instrumentation functions and is
also consistent with other annotation setters.

[BOLT] Improve handling of relocations targeting specific instructions (#66395)

On RISC-V, there are certain relocations that target a specific
instruction instead of a more abstract location like

[BOLT] Improve handling of relocations targeting specific instructions (#66395)

On RISC-V, there are certain relocations that target a specific
instruction instead of a more abstract location like a function or basic
block. Take the following example that loads a value from symbol `foo`:

```
nop
1: auipc t0, %pcrel_hi(foo)
ld t0, %pcrel_lo(1b)(t0)
```

This results in two relocation:
- auipc: `R_RISCV_PCREL_HI20` referencing `foo`;
- ld: `R_RISCV_PCREL_LO12_I` referencing to local label `1` which points
to the auipc instruction.

It is of utmost importance that the `R_RISCV_PCREL_LO12_I` keeps
referring to the auipc instruction; if not, the program will fail to
assemble. However, BOLT currently does not guarantee this.

BOLT currently assumes that all local symbols are jump targets and
always starts a new basic block at symbol locations. The example above
results in a CFG the looks like this:

```
.BB0:
nop
.BB1:
auipc t0, %pcrel_hi(foo)
ld t0, %pcrel_lo(.BB1)(t0)
```

While this currently works (i.e., the `R_RISCV_PCREL_LO12_I` relocation
points to the correct instruction), it has two downsides:
- Too many basic blocks are created (the example above is logically only
one yet two are created);
- If instructions are inserted in `.BB1` (e.g., by instrumentation),
things will break since the label will not point to the auipc anymore.

This patch proposes to fix this issue by teaching BOLT to track labels
that should always point to a specific instruction. This is implemented
as follows:
- Add a new annotation type (`kLabel`) that allows us to annotate
instructions with an `MCSymbol *`;
- Whenever we encounter a relocation type that is used to refer to a
specific instruction (`Relocation::isInstructionReference`), we
register it without a symbol;
- During disassembly, whenever we encounter an instruction with such a
relocation, create a symbol for its target and store it in an offset
to symbol map (to ensure multiple relocations referencing the same
instruction use the same label);
- After disassembly, iterate this map to attach labels to instructions
via the new annotation type;
- During emission, emit these labels right before the instruction.

I believe the use of annotations works quite well for this use case as
it allows us to reliably track instruction labels. If we were to store
them as offsets in basic blocks, it would be error prone to keep them
updated whenever instructions are inserted or removed.

I have chosen to add labels as first-class annotations (as opposed to a
generic one) because the documentation of `MCAnnotation` suggests that
generic annotations are to be used for optional metadata that can be
discarded without affecting correctness. As this is not the case for
labels, a first-class annotation seemed more appropriate.

show more ...

[MC] Check if register is non-null before calling isSubRegisterEq (NFCI)

D151036 adds an assertions that prohibits iterating over sub- and
super-registers of a null register. This is already the cas

[MC] Check if register is non-null before calling isSubRegisterEq (NFCI)

D151036 adds an assertions that prohibits iterating over sub- and
super-registers of a null register. This is already the case when
iterating over register units of a null register, and worked by
accident for sub- and super-registers.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D151285

show more ...

[BOLT][NFC] Add MCPlusBuilder defOperands/useOperands helpers

Make intent more explicit with the use of new helper methods.

Reviewed By: #bolt, maksfb

Differential Revision: https://reviews.llvm.o

[BOLT][NFC] Add MCPlusBuilder defOperands/useOperands helpers

Make intent more explicit with the use of new helper methods.

Reviewed By: #bolt, maksfb

Differential Revision: https://reviews.llvm.org/D150810

show more ...

[BOLT][NFC] Simplify MCPlusBuilder::getRegSize

Pre-calculate the register size table in MCPlusBuilder constructor,
similar to `AliasMap`/`SmallerAliasMap` in `initAliases`.

Reviewed By: #bolt, rafa

[BOLT][NFC] Simplify MCPlusBuilder::getRegSize

Pre-calculate the register size table in MCPlusBuilder constructor,
similar to `AliasMap`/`SmallerAliasMap` in `initAliases`.

Reviewed By: #bolt, rafauler

Differential Revision: https://reviews.llvm.org/D145828

show more ...

[BOLT][NFC] Improve performance of MCPlusBuilder::initAliases

It was using a redundant iteration over super regs to build
SmallerAliasMap. Removing this results in exactly the same alias maps
and a

[BOLT][NFC] Improve performance of MCPlusBuilder::initAliases

It was using a redundant iteration over super regs to build
SmallerAliasMap. Removing this results in exactly the same alias maps
and a noticeable performance gain on targets with a large number of
registers.

Just anecdotally: on my machine, processing a small AArch64 binary went
from 2.7s down to 80ms.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D145779

show more ...

[BOLT] Fix intermittent crash with instrumentation

When createInstrumentedIndirectCall() was invoked for tail calls, we
attached annotation instruction twice to the new call instruction.
First in cr

[BOLT] Fix intermittent crash with instrumentation

When createInstrumentedIndirectCall() was invoked for tail calls, we
attached annotation instruction twice to the new call instruction.
First in createDirectCall(), and then again while copying over the
metadata operands.

As a result, the annotations were not properly stripped for such calls
before the call to freeAnnotations() in LowerAnnotations pass. That lead
to use-after-free while restoring the offsets with setOffset() call.

Reviewed By: yota9

Differential Revision: https://reviews.llvm.org/D144806

show more ...

[BOLT] Use range-based implicit def/use accessors. NFCI

[BOLT][NFC] Remove AArch64 override for evaluateBranch

Use MCInstrAnalysis implementation

Reviewed By: #bolt, rafauler, yota9

Differential Revision: https://reviews.llvm.org/D141983

[BOLT][NFC] Use std::optional in MCPlusBuilder

Reviewed By: maksfb, #bolt

Differential Revision: https://reviews.llvm.org/D139260

[BOLT] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of ma

[BOLT] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

Use None consistently (NFC)

This patch replaces NoneType() and NoneType::None with None in
preparation for migration from llvm::Optional to std::optional.

In the std::optional world, we are not gur

Use None consistently (NFC)

This patch replaces NoneType() and NoneType::None with None in
preparation for migration from llvm::Optional to std::optional.

In the std::optional world, we are not guranteed to be able to
default-construct std::nullopt_t or peek what's inside it, so neither
NoneType() nor NoneType::None has a corresponding expression in the
std::optional world.

Once we consistently use None, we should even be able to replace the
contents of llvm/include/llvm/ADT/None.h with something like:

using NoneType = std::nullopt_t;
inline constexpr std::nullopt_t None = std::nullopt;

to ease the migration from llvm::Optional to std::optional.

Differential Revision: https://reviews.llvm.org/D138376

show more ...

[BOLT] Split functions with exceptions in shared objects and PIEs

Add functionality to allow splitting code with C++ exceptions in shared
libraries and PIEs. To overcome a limitation in exception ra

[BOLT] Split functions with exceptions in shared objects and PIEs

Add functionality to allow splitting code with C++ exceptions in shared
libraries and PIEs. To overcome a limitation in exception ranges format,
for functions with fragments spanning multiple sections, add trampoline
landing pads in the same section as the corresponding throwing range.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D127936

show more ...

[BOLT][NFC] Fix MCPlusBuilder::getAliases caching behavior

Caching behavior of `getAliases` causes a failure in unit tests where two
MCPlusBuilder objects are created corresponding to AArch64 and X8

[BOLT][NFC] Fix MCPlusBuilder::getAliases caching behavior

Caching behavior of `getAliases` causes a failure in unit tests where two
MCPlusBuilder objects are created corresponding to AArch64 and X86:
the alias cache is created for AArch64 but then used for X86.

https://lab.llvm.org/staging/#/builders/211/builds/126

The issue only affects unit tests as we only construct one MCPlusBuilder
for ELF binary.

Resolve the issue by moving alias bitvectors to MCPlusBuilder object.

Reviewed By: yota9

Differential Revision: https://reviews.llvm.org/D124942

show more ...

Add missing MC includes in bolt/

Changes needed after ef736a1c39f27ef4 that removes some implicit
dependencies from MrCV headers.

[BOLT] Fix AARCH64 registers aliasing

The aarch64 platform has special registers like X0_X1_X2_X3_X4_X5_X6_X7.
Using the downwards propagation this register will become a super
register for all X0..

[BOLT] Fix AARCH64 registers aliasing

The aarch64 platform has special registers like X0_X1_X2_X3_X4_X5_X6_X7.
Using the downwards propagation this register will become a super
register for all X0..X7 and its super registers which is not right. This
patch replaces the downwards propagation with caching all the aliases using MCRegAliasIterator.

Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D117394

show more ...

[BOLT][NFC] Move Offset annotation to Group 1

Summary:
Move the annotation to avoid dynamic memory allocations.
Improves the CPU time of instrumenting a large binary by 1% (+-0.8%, p-value 0.01)

Te

[BOLT][NFC] Move Offset annotation to Group 1

Summary:
Move the annotation to avoid dynamic memory allocations.
Improves the CPU time of instrumenting a large binary by 1% (+-0.8%, p-value 0.01)

Test Plan: NFC

Reviewers: maksfb

FBD30091656

show more ...