[Instrumentation] Remove unused includes (NFC) (#115117)

Identified with misc-include-cleaner.

[LLVM] Add `Intrinsic::getDeclarationIfExists` (#112428)

Add `Intrinsic::getDeclarationIfExists` to lookup an existing
declaration of an intrinsic in a `Module`.

[TypeProf][PGO]Support skipping vtable comparisons for a class and its derived ones (#110575)

Performance critical core libraries could be highly-optimized for arch
or micro-arch features. For inst

[TypeProf][PGO]Support skipping vtable comparisons for a class and its derived ones (#110575)

Performance critical core libraries could be highly-optimized for arch
or micro-arch features. For instance, the absl crc library specializes
different templated classes among different hardwares [1]. In a
practical setting, it's likely that instrumented profiles are collected
on one type of machine and used to optimize binaries that run on
multiple types of hardwares.

While this kind of specialization is rare in terms of lines of code,
compiler can do a better job to skip vtable-based ICP.
* The per-class `Extend` implementation is arch-specific as well. If an
instrumented profile is collected on one arch and applied to another
arch where `Extend` implementation is different, `Extend` might be
regarded as unlikely function in the latter case. `ABSL_ATTRIBUTE_HOT`
annotation alleviates the problem by putting all `Extend` implementation
into the hot text section [2]

This change introduces a comma-separated list to specify the mangled
vtable names, and ICP pass will skip vtable-based comparison if a vtable
variable definition is shown to be in its class hierarchy (per LLVM type
metadata).

[1]
https://github.com/abseil/abseil-cpp/blob/c6b27359c3d27438b1313dddd7598914c1274a50/absl/crc/internal/crc_x86_arm_combined.cc#L621-L650
[2]
https://github.com/abseil/abseil-cpp/blame/c6b27359c3d27438b1313dddd7598914c1274a50/absl/crc/internal/crc_x86_arm_combined.cc#L370C3-L370C21

show more ...

[Instrumentation] Move out to Utils (NFC) (#108532)

Utility functions have been moved out to Utils. Minor opportunity to
drop the header where not needed.

[ICP] Fix warnings

This patch fixes:

llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp:845:12:
error: variable 'RemainingVTableCount' set but not used
[-Werror,-Wunused-but-set-va

[ICP] Fix warnings

This patch fixes:

llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp:845:12:
error: variable 'RemainingVTableCount' set but not used
[-Werror,-Wunused-but-set-variable]

llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp:306:23:
error: private field 'PSI' is not used
[-Werror,-Wunused-private-field]

Here are a couple of domino effects:

- Once I remove PSI, I need to update the contructor and its caller.

- Once I remove RemainingVTableCount, I don't need TotalCount, so I am
updating the caller as well.

show more ...

[TypeProf][ICP]Allow vtable-comparison as long as vtable count is comparable with function count for each candidate (#106260)

The current cost-benefit analysis between vtable comparison and function

[TypeProf][ICP]Allow vtable-comparison as long as vtable count is comparable with function count for each candidate (#106260)

The current cost-benefit analysis between vtable comparison and function
comparison require the indirect fallback branch to be cold. This is too
conservative.

This change allows vtable-comparison as long as vtable count is
comparable with function count for each function candidate and removes
the cold indirect fallback requirement.

Tested:
1. Testing this on benchmarks uplifts the measurable performance wins.
Counting the (possibly-duplicated) remarks (because of linkonce_odr
functions, cross-module import of functions) show the number of vtable
remarks increases from ~30k-ish to 50k-ish.
2. https://gcc.godbolt.org/z/sbGK7Pacn shows vtable-comparison doesn't
happen today (using the same IR input)

show more ...

[IndirectCallPromotion] Migrate to a new version of getValueProfDataFromInst (#97357)

[IndirectCallPromotion] Use ArrayRef<PromotionCandidate> (NFC) (#97236)

Aside from the fact that LLVM Programmer's Manual prefers ArrayRef to
const std::vector &, ArrayRef<PromotionCandidate> here

[IndirectCallPromotion] Use ArrayRef<PromotionCandidate> (NFC) (#97236)

Aside from the fact that LLVM Programmer's Manual prefers ArrayRef to
const std::vector &, ArrayRef<PromotionCandidate> here makes it easier
to switch the underlying type to something like SmallVector. Note
that we typically do not have a lot of candidates.

show more ...

[TypeProf] Include unordered_map after #81442

Needed by VTableAddressPointOffsetValMap.

[TypeProf][InstrFDO]Implement more efficient comparison sequence for indirect-call-promotion with vtable profiles. (#81442)

Clang's `-fwhole-program-vtables` is required for this optimization to
ta

[TypeProf][InstrFDO]Implement more efficient comparison sequence for indirect-call-promotion with vtable profiles. (#81442)

Clang's `-fwhole-program-vtables` is required for this optimization to
take place. If `-fwhole-program-vtables` is not enabled, this change is
no-op.

* Function-comparison (before):

```
%vtable = load ptr, ptr %obj
%vfn = getelementptr inbounds ptr, ptr %vtable, i64 1
%func = load ptr, ptr %vfn
%cond = icmp eq ptr %func, @callee
br i1 %cond, label bb1, label bb2:

bb1:
call @callee

bb2:
call %func
```

* VTable-comparison (after):

```
%vtable = load ptr, ptr %obj
%cond = icmp eq ptr %vtable, @vtable-address-point
br i1 %cond, label bb1, label bb2:

bb1:
call @callee

bb2:
%vfn = getelementptr inbounds ptr, ptr %vtable, i64 1
%func = load ptr, ptr %vfn
call %func
```

Key changes:
1. Find out virtual calls and the vtables they come from.
- The ICP relies on type intrinsic `llvm.type.test` to find out virtual
calls and the
compatible vtables, and relies on type metadata to find the address
point for comparison.
2. ICP pass does cost-benefit analysis and compares vtable only when the
number of vtables for a function candidate is within (option specified)
threshold.
3. Sink the function addressing and vtable load instruction to indirect
fallback.
- The sink helper functions are simplified versions of
`InstCombinerImpl::tryToSinkInstruction`. Currently debug intrinsics are
not handled. Ideally `InstCombinerImpl::tryToSinkInstructionDbgValues`
and `InstCombinerImpl::tryToSinkInstructionDbgVariableRecords` could be
moved into Transforms/Utils/Local.cpp (or another util cpp file) to
handle debug intrinsics when moving instructions across basic blocks.
4. Keep value profiles updated
1) Update vtable value profiles after inline
2) For either function-based comparison or vtable-based comparison,
update both vtable and indirect call value profiles.

show more ...

[IndirectCallPromotion] Use ArrayRef consistently (NFC) (#96412)

[Analysis] Clean up getPromotionCandidatesForInstruction (NFC) (#95624)

Callers of getPromotionCandidatesForInstruction pass NumVals as an out
parameter for the number of value-count pairs of the v

[Analysis] Clean up getPromotionCandidatesForInstruction (NFC) (#95624)

Callers of getPromotionCandidatesForInstruction pass NumVals as an out
parameter for the number of value-count pairs of the value profiling
data, but nobody uses the out parameter.

This patch removes the parameter and updates the callers. Note that
the number of value-count pairs is still available via
getPromotionCandidatesForInstruction(...).size().

show more ...

Reapply "[llvm][IR] Extend BranchWeightMetadata to track provenance o… (#95281)

…f weights" #95136

Reverts #95060, and relands #86609, with the unintended code generation
changes addressed.

T

Reapply "[llvm][IR] Extend BranchWeightMetadata to track provenance o… (#95281)

…f weights" #95136

Reverts #95060, and relands #86609, with the unintended code generation
changes addressed.

This patch implements the changes to LLVM IR discussed in
https://discourse.llvm.org/t/rfc-update-branch-weights-metadata-to-allow-tracking-branch-weight-origins/75032

In this patch, we add an optional field to MD_prof meatdata nodes for
branch weights, which can be used to distinguish weights added from
llvm.expect* intrinsics from those added via other methods, e.g. from
profiles or inserted by the compiler.

One of the major motivations, is for use with MisExpect diagnostics,
which need to know if branch_weight metadata originates from an
llvm.expect intrinsic. Without that information, we end up checking
branch weights multiple times in the case if ThinLTO + SampleProfiling,
leading to some inaccuracy in how we report MisExpect related
diagnostics to users.

Since we change the format of MD_prof metadata in a fundamental way, we
need to update code handling branch weights in a number of places.

We also update the lang ref for branch weights to reflect the change.

show more ...

Revert "[llvm][IR] Extend BranchWeightMetadata to track provenance of weights" (#95060)

Reverts llvm/llvm-project#86609

This change causes compile-time regressions for stage2 builds
(https://llv

Revert "[llvm][IR] Extend BranchWeightMetadata to track provenance of weights" (#95060)

Reverts llvm/llvm-project#86609

This change causes compile-time regressions for stage2 builds
(https://llvm-compile-time-tracker.com/compare.php?from=3254f31a66263ea9647c9547f1531c3123444fcd&to=c5978f1eb5eeca8610b9dfce1fcbf1f473911cd8&stat=instructions:u).
It also introduced unintended changes to `.text` which should be
addressed before relanding.

show more ...

[llvm][IR] Extend BranchWeightMetadata to track provenance of weights (#86609)

This patch implements the changes to LLVM IR discussed in

https://discourse.llvm.org/t/rfc-update-branch-weights-met

[llvm][IR] Extend BranchWeightMetadata to track provenance of weights (#86609)

This patch implements the changes to LLVM IR discussed in

https://discourse.llvm.org/t/rfc-update-branch-weights-metadata-to-allow-tracking-branch-weight-origins/75032

In this patch, we add an optional field to MD_prof metadata nodes for
branch weights, which can be used to distinguish weights added from
`llvm.expect*` intrinsics from those added via other methods, e.g.
from profiles or inserted by the compiler.

One of the major motivations, is for use with MisExpect diagnostics,
which need to know if branch_weight metadata originates from an
llvm.expect intrinsic. Without that information, we end up checking
branch weights multiple times in the case if ThinLTO + SampleProfiling,
leading to some inaccuracy in how we report MisExpect related
diagnostics to users.

Since we change the format of MD_prof metadata in a fundamental way, we
need to update code handling branch weights in a number of places.

We also update the lang ref for branch weights to reflect the change.

show more ...

[NFC][IndirectCallProm] Refactor function-based conditional devirtualization and indirect call value profile update into one helper function (#80762)

* The motivation is to move indirect callee prof

[NFC][IndirectCallProm] Refactor function-based conditional devirtualization and indirect call value profile update into one helper function (#80762)

* The motivation is to move indirect callee profile update inside the
function-based speculative indirect-call promotion, so that there are
fewer diffs the vtable-based transformation and profile update is
implemented in a follow-up patch.
* The Parent patch is https://github.com/llvm/llvm-project/pull/79381

show more ...

Add setBranchWeigths convenience function. NFC (#72446)

Add `setBranchWeights` convenience function to ProfDataUtils.h and use
it where appropriate.

[nfc] Renamed ICallPromotionFunc to InidrectCallPromoter

For (subjectively) more clarity; also updated the comment describing it.

Differential Revision: https://reviews.llvm.org/D155888

[nfc] small maintainability IndirectCallPromotion changes

- we can remove the `Module` field, it's obtainable from `F` and used in
only one place
- a few fields can be `const`-ed, thus enforcing c

[nfc] small maintainability IndirectCallPromotion changes

- we can remove the `Module` field, it's obtainable from `F` and used in
only one place
- a few fields can be `const`-ed, thus enforcing compile-time
initialization checking (and we don't need to support `operator=`)

Differential Revision: https://reviews.llvm.org/D155212

show more ...

[nfc] rename `AM` to `MAM` in PGOInstrumentation

Consistency with how we tend to name `ModuleAnalysisManager` parameters.

[nfc] simplify promoteIndirectCalls

There's no caller to `promoteIndirectCalls` that would pass a nullptr
`ModuleAnalysisManager`, so passing it by reference does away with a
bunch of nullptr tests,

[nfc] simplify promoteIndirectCalls

There's no caller to `promoteIndirectCalls` that would pass a nullptr
`ModuleAnalysisManager`, so passing it by reference does away with a
bunch of nullptr tests, and also removes the need for a "OwnedORE".

Differential Revision: https://reviews.llvm.org/D155027

show more ...

[llvm] Qualify auto (NFC)

Identified with readability-qualified-auto.

[llvm] Remove unneeded cl::ZeroOrMore for cl::opt options. NFC

Some cl::ZeroOrMore were added to avoid the `may only occur zero or one times!`
error. More were added due to cargo cult. Since the err

[llvm] Remove unneeded cl::ZeroOrMore for cl::opt options. NFC

Some cl::ZeroOrMore were added to avoid the `may only occur zero or one times!`
error. More were added due to cargo cult. Since the error has been removed,
cl::ZeroOrMore is unneeded.

Also remove cl::init(false) while touching the lines.

show more ...

[iwyu] Handle regressions in libLLVM header include

Running iwyu-diff on LLVM codebase since fa5a4e1b95c8f37796 detected a few
regressions, fixing them.

Differential Revision: https://reviews.llvm.

[iwyu] Handle regressions in libLLVM header include

Running iwyu-diff on LLVM codebase since fa5a4e1b95c8f37796 detected a few
regressions, fixing them.

Differential Revision: https://reviews.llvm.org/D124847

show more ...

[PGO] Remove legacy PM passes

Legacy PM for optimization pipeline was deprecated in 13.0.0 and Clang dropped
legacy PM support in D123609. This change removes legacy PM passes for PGO so
that downst

[PGO] Remove legacy PM passes

Legacy PM for optimization pipeline was deprecated in 13.0.0 and Clang dropped
legacy PM support in D123609. This change removes legacy PM passes for PGO so
that downstream projects won't be able to use it. It seems appropriate to start
removing such "add-on" features like instrumentations, before we remove more
stuff after 15.x is branched.

I have checked many LLVM users and only ldc[1] uses the legacy PGO pass.

[1]: https://github.com/ldc-developers/ldc/issues/3961

Reviewed By: davidxl

Differential Revision: https://reviews.llvm.org/D123834

show more ...