[BOLT][AArch64] Remove nops in functions with defined control flow (#124705)

When a function has an indirect branch with unknown control flow, we
preserve nops in order to keep all instruction offs

[BOLT][AArch64] Remove nops in functions with defined control flow (#124705)

When a function has an indirect branch with unknown control flow, we
preserve nops in order to keep all instruction offsets (from the start
of the function) the same in case the indirect branch is used by a
PC-relative jump table. However, when we know the control flow of the
function, we should be able to safely remove nops.

show more ...

[BOLT] Add support for safe-icf (#116275)

Identical Code Folding (ICF) folds functions that are identical into one
function, and updates symbol addresses to the new address. This reduces
the size

[BOLT] Add support for safe-icf (#116275)

Identical Code Folding (ICF) folds functions that are identical into one
function, and updates symbol addresses to the new address. This reduces
the size of a binary, but can lead to problems. For example when
function pointers are compared. This can be done either explicitly in
the code or generated IR by optimization passes like Indirect Call
Promotion (ICP). After ICF what used to be two different addresses
become the same address. This can lead to a different code path being
taken.

This is where safe ICF comes in. Linker (LLD) does it using address
significant section generated by clang. If symbol is in it, or an object
doesn't have this section symbols are not folded.

BOLT does not have the information regarding which objects do not have
this section, so can't re-use this mechanism.

This implementation scans code section and conservatively marks
functions symbols as unsafe. It treats symbols as unsafe if they are
used in non-control flow instruction. It also scans through the data
relocation sections and does the same for relocations that reference a
function symbol. The latter handles the case when function pointer is
stored in a local or global variable, etc. If a relocation address
points within a vtable these symbols are skipped.

show more ...

[BOLT] Extract comparator for sorting functions by index into helper function (#116217)

This change extracts the comparator for sorting functions by index into
a helper function `compareBinaryFunct

[BOLT] Extract comparator for sorting functions by index into helper function (#116217)

This change extracts the comparator for sorting functions by index into
a helper function `compareBinaryFunctionByIndex()`

Not sure why the comparator used in
`BinaryContext::getSortedFunctions()` is not same as the other two
places. I think they should use the same comparator, so I also change
`BinaryContext::getSortedFunctions()` to use
`compareBinaryFunctionByIndex()` for sorting functions.

show more ...

[mc] Add CFI directive to emit val_offset() rules (#113971)

These specify that the value of the given register in the previous frame
is the CFA plus some offset. This isn't very common but can be n

[mc] Add CFI directive to emit val_offset() rules (#113971)

These specify that the value of the given register in the previous frame
is the CFA plus some offset. This isn't very common but can be necessary
if the original value is normally reconstructed from the stack/frame
pointer instead of being saved on the stack and reloaded from there.

show more ...

[BOLT] Fix warnings (#114116)

This patch fixes:

bolt/lib/Core/BinaryFunction.cpp:2537:13: error: enumeration value
'OpNegateRAStateWithPC' not handled in switch [-Werror,-Wswitch]

bolt/

[BOLT] Fix warnings (#114116)

This patch fixes:

bolt/lib/Core/BinaryFunction.cpp:2537:13: error: enumeration value
'OpNegateRAStateWithPC' not handled in switch [-Werror,-Wswitch]

bolt/lib/Core/BinaryFunction.cpp:2661:13: error: enumeration value
'OpNegateRAStateWithPC' not handled in switch [-Werror,-Wswitch]

bolt/lib/Core/BinaryFunction.cpp:2805:13: error: enumeration value
'OpNegateRAStateWithPC' not handled in switch [-Werror,-Wswitch]

show more ...

[BOLT] Avoid repeated map lookups (NFC) (#112118)

[BOLT] Allow sections in --print-only flag (#109622)

While printing functions, expand --print-only flag to accept section
names. E.g., "--print-only=\.init" will only print functions from
".init"

[BOLT] Allow sections in --print-only flag (#109622)

While printing functions, expand --print-only flag to accept section
names. E.g., "--print-only=\.init" will only print functions from
".init" section.

show more ...

[BOLT] Handle internal calls in ValidateInternalCalls (#105736)

Move handling of all internal calls into the designated pass. Preserve
NOPs and mark functions as non-simple on non-X86 platforms.

[BOLT] Reduce CFI warning verbosity (#105336)

CFI programs may have more saves than restores and this is completely
benign from BOLT's perspective. Reduce the verbosity and print the
warning only

[BOLT] Reduce CFI warning verbosity (#105336)

CFI programs may have more saves than restores and this is completely
benign from BOLT's perspective. Reduce the verbosity and print the
warning only under `-v=1` and above.

show more ...

[BOLT] Turn non-empty CFI StateStack assert into a warning (#102216)

clang-15 can produce binaries with mismatched RememberState/RestoreState
CFIs. This is benign for unwinding, so replace an asser

[BOLT] Turn non-empty CFI StateStack assert into a warning (#102216)

clang-15 can produce binaries with mismatched RememberState/RestoreState
CFIs. This is benign for unwinding, so replace an assert with a warning.

show more ...

[BOLT] Support POSSIBLE_PIC_FIXED_BRANCH

Detect and support fixed PIC indirect jumps of the following form:
```
movslq En(%rip), %r1
leaq PIC_JUMP_TABLE(%rip), %r2
addq %r2, %r1
jmpq *%r1
```

w

[BOLT] Support POSSIBLE_PIC_FIXED_BRANCH

Detect and support fixed PIC indirect jumps of the following form:
```
movslq En(%rip), %r1
leaq PIC_JUMP_TABLE(%rip), %r2
addq %r2, %r1
jmpq *%r1
```

with PIC_JUMP_TABLE that looks like following:

```
JT: ----------
E1:| L1 - JT |
|----------|
E2:| L2 - JT |
|----------|
| |
......
En:| Ln - JT |
----------
```

The code could be produced by compilers, see
https://github.com/llvm/llvm-project/issues/91648.

Test Plan: updated jump-table-fixed-ref-pic.test

Reviewers: maksfb, ayermolo, dcci, rafaelauler

Reviewed By: rafaelauler

Pull Request: https://github.com/llvm/llvm-project/pull/91667

show more ...

[MC] Support .cfi_label

GNU assembler 2.26 introduced the .cfi_label directive. It does not
expand to any CFI instructions, but defines a label in
.eh_frame/.debug_frame, which can be used by runtim

[MC] Support .cfi_label

GNU assembler 2.26 introduced the .cfi_label directive. It does not
expand to any CFI instructions, but defines a label in
.eh_frame/.debug_frame, which can be used by runtime patching code to
locate the FDE. .cfi_label is not allowed for CIE's initial
instructions, and can therefore be used to force the next instruction to
be placed in a FDE instead of a CIE.

In glibc since 2018, sysdeps/riscv/start.S utilizes .cfi_label to force
DW_CFA_undefined to be placed in a FDE. arc/csky/loongarch ports have
copied this use.
```
.cfi_startproc
// DW_CFA_undefined is allowed for CIE's initial instructions.
// Without .cfi_label, gas would place DW_CFA_undefined in a CIE.
.cfi_label .Ldummy
.cfi_undefined ra
.cfi_endproc
```

No CFI instruction is associated with .cfi_label, so the `case
MCCFIInstruction::OpLabel:` code in BOLT is unreachable and onlt to make
-Wswitch happy.

Close #97222

Pull Request: https://github.com/llvm/llvm-project/pull/97922

show more ...

[BOLT] Drop macro-fusion alignment (#97358)

9d0754ada5dbbc0c009bcc2f7824488419cc5530 dropped MC support required for
optimal macro-fusion alignment in BOLT. Remove the support in BOLT as
performan

[BOLT] Drop macro-fusion alignment (#97358)

9d0754ada5dbbc0c009bcc2f7824488419cc5530 dropped MC support required for
optimal macro-fusion alignment in BOLT. Remove the support in BOLT as
performance measurements with large binaries didn't show a significant
improvement.

Test Plan:
macro-fusion alignment was never upstreamed, so no upstream tests are
affected.

show more ...

[BOLT][NFC] Separate isReversibleBranch's 2 semantics (#95572)

`isUnsupportedBranch` was renamed (and inverted) to `isReversibleBranch`, as that was how it was being used. But one use in `BinaryFu

[BOLT][NFC] Separate isReversibleBranch's 2 semantics (#95572)

`isUnsupportedBranch` was renamed (and inverted) to `isReversibleBranch`, as that was how it was being used. But one use in `BinaryFunction::disassemble` was using the original meaning to detect unsupported branches, and the `isUnsupportedBranch` had 2 separate semantic checks.

Move the unsupported branch check from `isReversibleBranch` to a new entry point: `isUnsupportedInstruction`. Call that from `BinaryFunction::disassemble`.

Move the dynamic branch check from X86's isReversibleBranch to the base class, as it is not an architecture-specific check.

Remove unnecessary `isReversibleBranch` calls from Instrumentation and X86 MCPlusBuilder.

show more ...

[BOLT][Linux] Support ORC for alternative instructions (#96709)

Alternative instruction sequences in the Linux kernel can modify the
stack and thus they need their own ORC unwind entries. Since the

[BOLT][Linux] Support ORC for alternative instructions (#96709)

Alternative instruction sequences in the Linux kernel can modify the
stack and thus they need their own ORC unwind entries. Since there's
only one ORC table, it has to be "shared" among multiple instruction
sequences. The kernel achieves this by putting a restriction on
instruction boundaries. If ORC state changes at a given IP, only one of
the alternative sequences can have an instruction starting/ending at
this IP. Then, developers can insert NOPs to guarantee the above
requirement is met.

The most common use of ORC with alternatives is "pushf; pop %rax"
sequence used for paravirtualization. Note that newer kernel versions
no longer use .parainstructions; instead, they utilize alternatives for
the same purpose.

Before we implement a better support for alternatives, we can safely
skip ORC entries associated with them.

Fixes #87052.

show more ...

[BOLT] Omit CFI state while printing functions without CFI (#96723)

If a function has no CFI program attached to it, do not print redundant
empty CFI state for every basic block.

[bolt] Add missing <stack> include (NFC)

[BOLT] Remove mutable from BB::LayoutIndex (#93224)

Removed mutability from BB::LayoutIndex, subsequently removed const from
BB::SetLayout, and changed BF::dfs to track visited blocks with a set as

[BOLT] Remove mutable from BB::LayoutIndex (#93224)

Removed mutability from BB::LayoutIndex, subsequently removed const from
BB::SetLayout, and changed BF::dfs to track visited blocks with a set as
opposed to tracking and altering LayoutIndexes for more consistent code.

show more ...

[BOLT][NFC] Define getExprValue helper (#91663)

Move out common code extracting the address of a MCExpr. To be reused in
#91667.

Test Plan: NFC

[BOLT][NFC] Avoid computing BF hash twice in YAML reader (#75096)

We compute BF hashes in `YAMLProfileReader::readProfile` when first
matching profile functions with binary functions, and second ti

[BOLT][NFC] Avoid computing BF hash twice in YAML reader (#75096)

We compute BF hashes in `YAMLProfileReader::readProfile` when first
matching profile functions with binary functions, and second time in
`YAMLProfileReader::parseFunctionProfile` during the profile assignment
(we need to do that to account for LTO private functions with
mismatching suffix).

Avoid recomputing the hash if it's been set.

show more ...

[BOLT] Process cross references between ignored functions in BAT mode (#92484)

To align YAML and fdata profiles produced in BAT mode, lift two
restrictions applied in non-relocation mode when BAT i

[BOLT] Process cross references between ignored functions in BAT mode (#92484)

To align YAML and fdata profiles produced in BAT mode, lift two
restrictions applied in non-relocation mode when BAT is present:
1) register secondary entry points from ignored functions,
2) treat functions with secondary entry points as simple.

This allows constructing CFG for non-simple functions in non-relocation
mode and emitting YAML profile for them, which can then be used for
optimizations in relocation mode.

Test Plan: added test ignored-interprocedural-reference.s

show more ...

[BOLT][NFC] Rename isUnsupportedBranch to isReversibleBranch (#92447)

`isUnsupportedBranch` is not a very informative name, and doesn't match
its corresponding `reverseBranchCondition`, as I noted

[BOLT][NFC] Rename isUnsupportedBranch to isReversibleBranch (#92447)

`isUnsupportedBranch` is not a very informative name, and doesn't match
its corresponding `reverseBranchCondition`, as I noted in PR #92018.
Here's a renaming to a more mnemonic name.

show more ...

[BOLT][NFC] Simplify CFG validation (#91977)

Remove 'Valid' local boolean that has a single use, and return directly instead.

[BOLT] Ignore returns in DataAggregator

Returns are ignored in perf/pre-aggregated/fdata profile reader (see
DataReader::convertBranchData). They are also omitted in
YAMLProfileWriter by virtue of n

[BOLT] Ignore returns in DataAggregator

Returns are ignored in perf/pre-aggregated/fdata profile reader (see
DataReader::convertBranchData). They are also omitted in
YAMLProfileWriter by virtue of not having the profile attached to them
in the reader, and YAMLProfileWriter converting the profile attached to
BinaryFunctions. Thus, return profile is universally ignored across all
profile types except BAT YAML.

To make returns ignored for YAML produced in BAT mode, we can:
1) ignore them in YAMLProfileReader,
2) omit them from YAML profile in profile conversion/writing.

The first option is prone to profile staleness issue, where the profiled
binary doesn't match the one to be optimized, and thus returns in the
profile can no longer be reliably detected (as we don't distinguish them
from calls in the profile).

The second option is robust to staleness but requires disassembling the
branch source instruction.

Test Plan: Updated bolt-address-translation-yaml.test

Reviewers: rafaelauler, dcci, ayermolo, maksfb

Reviewed By: maksfb

Pull Request: https://github.com/llvm/llvm-project/pull/90807

show more ...

[BOLT][NFC] Clean includes, add license headers (#87200)