Reapply "[BOLT] Add --pad-funcs-before=func:n (#117924)" (#121918)

- **Reapply "[BOLT] Add --pad-funcs-before=func:n (#117924)"**
- **[BOLT] Fix --pad-funcs{,-before} state misinteraction**

When --

Reapply "[BOLT] Add --pad-funcs-before=func:n (#117924)" (#121918)

- **Reapply "[BOLT] Add --pad-funcs-before=func:n (#117924)"**
- **[BOLT] Fix --pad-funcs{,-before} state misinteraction**

When --pad-funcs-before was introduced, it introduced a bug whereby the
first one to get parsed could influence the other.

Ensure that each has its own state and test that they don't interact in
this manner by testing how the `_subsequent` symbol moves when both
arguments are supplied with different padding values.

Fixed by having a function (and static state) for each of before/after.

show more ...

Revert "[BOLT] Add --pad-funcs-before=func:n (#117924)"

14dcf8214f9c66172d17c1cfaec6aec0030748e0 introduced a subtle bug with
the static `FunctionPadding` map.

If either `opts::FunctionPadSpec` or

Revert "[BOLT] Add --pad-funcs-before=func:n (#117924)"

14dcf8214f9c66172d17c1cfaec6aec0030748e0 introduced a subtle bug with
the static `FunctionPadding` map.

If either `opts::FunctionPadSpec` or `opts::FunctionPadBeforeSpec` are set,
the map is going to be populated with the respective spec in the first
invocation of `BinaryEmitter::emitFunction`. The subsequent invocations
will pick up the padding from the map irrespective of whether
`opts::FunctionPadSpec` or `opts::FunctionPadBeforeSpec` is passed as a
parameter.

This breaks an internal test, hence reverting the patch.

show more ...

[BOLT] Add --pad-funcs-before=func:n (#117924)

This complements --pad-funcs, and by using both simultaneously, enables
moving a specific function through the address space without modifying
any co

[BOLT] Add --pad-funcs-before=func:n (#117924)

This complements --pad-funcs, and by using both simultaneously, enables
moving a specific function through the address space without modifying
any code
other than the targeted function (and references to it) by doing
(before+after=constant).

See also: proposed functionality to enable inserting random padding in

https://discourse.llvm.org/t/rfc-lld-feature-for-controlling-for-code-size-dependent-measurement-bias
and https://github.com/llvm/llvm-project/pull/117653

show more ...

[BOLT] Extract comparator for sorting functions by index into helper function (#116217)

This change extracts the comparator for sorting functions by index into
a helper function `compareBinaryFunct

[BOLT] Extract comparator for sorting functions by index into helper function (#116217)

This change extracts the comparator for sorting functions by index into
a helper function `compareBinaryFunctionByIndex()`

Not sure why the comparator used in
`BinaryContext::getSortedFunctions()` is not same as the other two
places. I think they should use the same comparator, so I also change
`BinaryContext::getSortedFunctions()` to use
`compareBinaryFunctionByIndex()` for sorting functions.

show more ...

[BOLT][NFC] Log through JournalingStreams (#81524)

Make core BOLT functionality more friendly to being used as a
library instead of in our standalone driver llvm-bolt. To
accomplish this, we augme

[BOLT][NFC] Log through JournalingStreams (#81524)

Make core BOLT functionality more friendly to being used as a
library instead of in our standalone driver llvm-bolt. To
accomplish this, we augment BinaryContext with journaling streams
that are to be used by most BOLT code whenever something needs to
be logged to the screen. Users of the library can decide if logs
should be printed to a file, no file or to the screen, as
before. To illustrate this, this patch adds a new option
`--log-file` that allows the user to redirect BOLT logging to a
file on disk or completely hide it by using
`--log-file=/dev/null`. Future BOLT code should now use
`BinaryContext::outs()` for printing important messages instead of
`llvm::outs()`. A new test log.test enforces this by verifying that
no strings are print to screen once the `--log-file` option is
used.

In previous patches we also added a new BOLTError class to report
common and fatal errors, so code shouldn't call exit(1) now. To
easily handle problems as before (by quitting with exit(1)),
callers can now use
`BinaryContext::logBOLTErrorsAndQuitOnFatal(Error)` whenever code
needs to deal with BOLT errors. To test this, we have fatal.s
that checks we are correctly quitting and printing a fatal error
to the screen.

Because this is a significant change by itself, not all code was
yet ported. Code from Profiler libs (DataAggregator and friends)
still print errors directly to screen.

Co-authored-by: Rafael Auler <rafaelauler@fb.com>

Test Plan: NFC

show more ...

[BOLT][NFC] Propagate BOLTErrors from Core, RewriteInstance, and passes (2/2) (#81523)

As part of the effort to refactor old error handling code that
would directly call exit(1), in this patch cont

[BOLT][NFC] Propagate BOLTErrors from Core, RewriteInstance, and passes (2/2) (#81523)

As part of the effort to refactor old error handling code that
would directly call exit(1), in this patch continue the migration
on libCore, libRewrite and libPasses to use the new BOLTError
class whenever a failure occurs.

Test Plan: NFC

Co-authored-by: Rafael Auler <rafaelauler@fb.com>

show more ...

[BOLT][NFC] Add BOLTError and return it from passes (1/2) (#81522)

As part of the effort to refactor old error handling code that
would directly call exit(1), in this patch we add a new class
BOLT

[BOLT][NFC] Add BOLTError and return it from passes (1/2) (#81522)

As part of the effort to refactor old error handling code that
would directly call exit(1), in this patch we add a new class
BOLTError and auxiliary functions `createFatalBOLTError()` and
`createNonFatalBOLTError()` that allow BOLT code to bubble up the
problem to the caller by using the Error class as a return
type (or Expected). Also changes passes to use these.

Co-authored-by: Rafael Auler <rafaelauler@fb.com>

Test Plan: NFC

show more ...

[BOLT][NFC] Return Error from BinaryFunctionPass::runOnFunctions (#81521)

As part of the effort to refactor old error handling code that
would directly call exit(1), in this patch we change the
in

[BOLT][NFC] Return Error from BinaryFunctionPass::runOnFunctions (#81521)

As part of the effort to refactor old error handling code that
would directly call exit(1), in this patch we change the
interface to `BinaryFunctionPass` to return an Error on
`runOnFunctions()`. This gives passes the ability to report a
serious problem to the caller (RewriteInstance class), so the
caller may decide how to best handle the exceptional situation.

Co-authored-by: Rafael Auler <rafaelauler@fb.com>

Test Plan: NFC

show more ...

[BOLT] Deprecate hfsort+ in favor of cdsort (#72408)

A new function sorting algorithm (cdsort) in LLVM is an optimized
version of BOLT's hfsort+. In order to avoid code duplication and
simplify

[BOLT] Deprecate hfsort+ in favor of cdsort (#72408)

A new function sorting algorithm (cdsort) in LLVM is an optimized
version of BOLT's hfsort+. In order to avoid code duplication and
simplify maintenance, getting rid of hfsort+.

Perf-wise this is likely a neutral change, though differences on
individual benchmarks are possible, since the generated function layout
has changed. I tested cdsort vs hfsort+ on a number of open-source and
prod binaries built in different modes and record an average neutral
perf difference, perhaps with more "green" counters.

show more ...

[BOLT] Add structure of CDSplit to SplitFunctions (#73430)

This commit establishes the general structure of the CDSplit strategy in
SplitFunctions without incorporating the exact splitting logic. W

[BOLT] Add structure of CDSplit to SplitFunctions (#73430)

This commit establishes the general structure of the CDSplit strategy in
SplitFunctions without incorporating the exact splitting logic. With
-split-functions -split-strategy=cdsplit, the SplitFunctions pass will
run twice: the first time is before function reordering and functions
are hot-cold split; the second time is after function reordering and
functions are hot-warm-cold split based on the fixed function ordering.
Currently, all functions are hot-warm split after the entry block in the
second splitting pass. Subsequent commits will introduce the precise
splitting logic. NFC.

show more ...

[BOLT] Rename cds to cdsort (#69966)

Unify naming for the layout algorithms by renaming "cds" to "cdsort".
This is
NFC unless someone is already using the new algorithm (which is
unlikely).

[CodeLayout] Refactor std::vector uses, namespace, and EdgeCountT. NFC

* Place types and functions in the llvm::codelayout namespace
* Change EdgeCountT from pair<pair<uint64_t, uint64_t>, uint64_t>

[CodeLayout] Refactor std::vector uses, namespace, and EdgeCountT. NFC

* Place types and functions in the llvm::codelayout namespace
* Change EdgeCountT from pair<pair<uint64_t, uint64_t>, uint64_t> to a struct and utilize structured bindings.
It is not conventional to use the "T" suffix for structure types.
* Remove a redundant copy in ChainT::merge.
* Change {ExtTSPImpl,CDSortImpl}::run to use return value instead of an output parameter
* Rename applyCDSLayout to computeCacheDirectedLayout: (a) avoid rare
abbreviation "CDS" (cache-directed sort) (b) "compute" is more conventional
for the specific use case
* Change the parameter types from std::vector to ArrayRef so that
SmallVector arguments can be used.
* Similarly, rename applyExtTspLayout to computeExtTspLayout.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D159526

show more ...

[BOLT] Fix sorting functions by execution count

I noticed that `-reorder-functions=exec-count` doesn't work as expected due to
a bug in the comparison function (which isn't symmetric). It is questio

[BOLT] Fix sorting functions by execution count

I noticed that `-reorder-functions=exec-count` doesn't work as expected due to
a bug in the comparison function (which isn't symmetric). It is questionable
whether anyone would want to ever use the sorting method (as sorting by say
density is much better in all cases) but it is probably better to fix the bug.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D152959

show more ...

[BOLT] Fixing macOS build

Fixing build after https://reviews.llvm.org/D153039

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D156734

[BOLT] A new code layout algorithm for function reordering [3b/3]

This is a new algorithm for function layout (reordering) based on the call graph
extracted from a profile data; see diffs down the s

[BOLT] A new code layout algorithm for function reordering [3b/3]

This is a new algorithm for function layout (reordering) based on the call graph
extracted from a profile data; see diffs down the stack for more details.

This layout is very similar to the existing hfsort+, but perhaps a little better
on some benchmarks. The goals of the change is as follows:

(i) rename and replace hfsort+ with a newer (hopefully better) implementation.
I'd prefer to keep both algs together for some time to simplify evaluation and
transition, but do want to remove hfsort+ once we're confident that there are
no regressions.

(ii) unify the implementation of code layout algorithms across LLVM. Currently
Passes/HfsortPlus.cpp and Utils/CodeLayout.cpp share many implementation-specific
details; this diff unifies the code.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D153039

show more ...

[BOLT][NFC] Format ReorderFunctions.cpp

Use *{Map,Set}::contains (NFC)

[BOLT][NFC] Use llvm::make_second_range

Reviewed By: #bolt, rafauler

Differential Revision: https://reviews.llvm.org/D143019

[BOLT] Use LTO fuzzy name matching in function-order

Allow partial name matching wrt LTO suffixes in `function-order`
user-supplied function list, the same as permitted by profile matching.

Reviewe

[BOLT] Use LTO fuzzy name matching in function-order

Allow partial name matching wrt LTO suffixes in `function-order`
user-supplied function list, the same as permitted by profile matching.

Reviewed By: #bolt, rafauler

Differential Revision: https://reviews.llvm.org/D142269

show more ...

[BOLT] Emit a warning about invalid entries in function-order list

Move individual warnings under verbosity >= 1, print out a warning with
aggregate number.

Reviewed By: #bolt, rafauler

Differenti

[BOLT] Emit a warning about invalid entries in function-order list

Move individual warnings under verbosity >= 1, print out a warning with
aggregate number.

Reviewed By: #bolt, rafauler

Differential Revision: https://reviews.llvm.org/D142397

show more ...

[BOLT][NFC] Move out ReorderFunctions::printStats

Break out stats-printing code from ReorderFunctions::reorder for brevity.

Reviewed By: #bolt, maksfb

Differential Revision: https://reviews.llvm.o

[BOLT][NFC] Move out ReorderFunctions::printStats

Break out stats-printing code from ReorderFunctions::reorder for brevity.

Reviewed By: #bolt, maksfb

Differential Revision: https://reviews.llvm.org/D142250

show more ...

[BOLT] Respect -function-order in lite mode

Process functions listed in -function-order file even in lite mode.

Reviewed By: #bolt, maksfb

Differential Revision: https://reviews.llvm.org/D140435

[BOLT] Update buildCallGraph to check for split blocks

Use isSplit() instead of isCold() when building the call graph and
update parameter names to reflect this.

Reviewed By: rafauler

Differential

[BOLT] Update buildCallGraph to check for split blocks

Use isSplit() instead of isCold() when building the call graph and
update parameter names to reflect this.

Reviewed By: rafauler

Differential Revision: https://reviews.llvm.org/D132047

show more ...

[BOLT] Use boolean literals (NFC)

Identified with modernize-use-bool-literals.

[BOLT][NFC] Use range-based STL wrappers

Replace `std::` algorithms taking begin/end iterators with `llvm::` counterparts
accepting ranges.

Reviewed By: rafauler

Differential Revision: https://rev

[BOLT][NFC] Use range-based STL wrappers

Replace `std::` algorithms taking begin/end iterators with `llvm::` counterparts
accepting ranges.

Reviewed By: rafauler

Differential Revision: https://reviews.llvm.org/D128154

show more ...