[ctx_prof] Simple ICP criteria during module inliner (#109881)

This is mostly for test: under contextual profiling, we perform ICP for those indirect callsites which have targets marked as `alwaysin

[ctx_prof] Simple ICP criteria during module inliner (#109881)

This is mostly for test: under contextual profiling, we perform ICP for those indirect callsites which have targets marked as `alwaysinline`.

This helped uncover a bug with the way the profile was updated upon ICP, where we were skipping over the update if the target wasn't called in that context. That was resulting in incorrect counts for the indirect BB.

Also flyby fix to the total/direct count values, they should be 64-bit (as all counters are in the contextual profile)

show more ...

[ctx_prof] Insert the ctx prof flattener after the module inliner (#107499)

This patch enables experimenting with the contextual profile. ICP is currently disabled in this case - will reenable it su

[ctx_prof] Insert the ctx prof flattener after the module inliner (#107499)

This patch enables experimenting with the contextual profile. ICP is currently disabled in this case - will reenable it subsequently. Also subsequently the inline cost model / decision making would be updated to be context-aware. Right now, this just achieves "complete use" of the profile, in that it's ingested, maintained, and sunk to a flat profile when not needed anymore.

Issue [#89287](https://github.com/llvm/llvm-project/issues/89287)

show more ...

[ctx_prof] Add Inlining support (#106154)

Add an overload of `InlineFunction` that updates the contextual profile. If there is no contextual profile, this overload is equivalent to the non-contextua

[ctx_prof] Add Inlining support (#106154)

Add an overload of `InlineFunction` that updates the contextual profile. If there is no contextual profile, this overload is equivalent to the non-contextual profile variant.

Post-inlining, the update mainly consists of:
- making the PGO instrumentation of the callee "the caller's": the owner function (the "name" parameter of the instrumentation instructions) becomes the caller, and new index values are allocated for each of the callee's indices (this happens for both increment and callsite instrumentation instructions)
- in the contextual profile:
- each context corresponding to the caller has its counters updated to incorporate the counters inherited from the callee at the inlined callsite. Counter values are copied as-is because no scaling is required since the profile is contextual.
- the contexts of the callee (at the inlined callsite) are moved to the caller.
- the callee context at the inlined callsite is deleted.

show more ...

[ModuleInliner] Remove an inapplicable comment

The module inliner has its own logic in deciding the order in which
call sites are inlined, so the comment is inapplicable.

[AlwaysInliner] Make legacy pass like the new pass

The legacy pass is only used in AMDGPU codegen, which doesn't care about running it in call graph order (it actually has to work around that fact).

[AlwaysInliner] Make legacy pass like the new pass

The legacy pass is only used in AMDGPU codegen, which doesn't care about running it in call graph order (it actually has to work around that fact).

Make the legacy pass a module pass and share code with the new pass.

This allows us to remove the legacy inliner infrastructure.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D146446

show more ...

[InlineOrder] Plugin Inline Order

Adds the ability to load a plugin to control the inline order.
This allows developing and distributing inlining heuristics
outside of tree. And together with the in

[InlineOrder] Plugin Inline Order

Adds the ability to load a plugin to control the inline order.
This allows developing and distributing inlining heuristics
outside of tree. And together with the inline advisor plugins
allows for fine grained control of the inliner.

The PluginInlineOrderAnalysis class serves as the entry point
for dynamic advisors. Plugins must register instances of this
class to provide their own InlineOrder.

Reviewed By: kazu

Differential Revision: https://reviews.llvm.org/D140637

show more ...

Revert "[InlineOrder] Plugin Inline Order"

This commit is causing a CMake error on AIX. Will recommit with
a fix.

This reverts commit e46d8a731535afcf0c5c2a2f6cf3c5d4fb69cd5b.

[InlineOrder] Plugin Inline Order

This allows developing and distributing inlining heuristics
outside of tree. And together with the inline advisor plugins
allows for fine grained control of the inl

[InlineOrder] Plugin Inline Order

This allows developing and distributing inlining heuristics
outside of tree. And together with the inline advisor plugins
allows for fine grained control of the inliner.

The PluginInlineOrderAnalysis class serves as the entry point
for dynamic advisors. Plugins must register instances of this
class to provide their own InlineOrder.

I'm checking in this patch on behalf of ibricchi
<ibricchi@student.ethz.ch>.

Differential Revision: https://reviews.llvm.org/D140637

show more ...

[IPO] Reorder parameters of InlineFunction (NFC)

With the recent addition of new parameter MergeAttributes (D134117),
callers need to specify several default parameters before getting to
specify the

[IPO] Reorder parameters of InlineFunction (NFC)

With the recent addition of new parameter MergeAttributes (D134117),
callers need to specify several default parameters before getting to
specify the new parameter.

This patch reorders the parameters so that callers do not have to
specify as many default parameters.

Differential Revision: https://reviews.llvm.org/D134125

show more ...

[ModuleInliner] Remove InlineOrder::front (NFC)

InlineOrder::front is a remnant from the era when we had a nested
"while" loops in the module inliner, with the inner one grouping the
call sites with

[ModuleInliner] Remove InlineOrder::front (NFC)

InlineOrder::front is a remnant from the era when we had a nested
"while" loops in the module inliner, with the inner one grouping the
call sites with the same caller.

Now that we have a simple "while" loop draining the priority queue, we
can just use InlineOrder::pop.

Differential Revision: https://reviews.llvm.org/D134121

show more ...

Silence unused variable warning in release builds. NFC

[Transforms] Merge function attributes within InlineFunction (NFC)

In the past, we've had a bug resulting in a compiler crash after
forgetting to merge function attributes (D105729).

This patch tea

[Transforms] Merge function attributes within InlineFunction (NFC)

In the past, we've had a bug resulting in a compiler crash after
forgetting to merge function attributes (D105729).

This patch teaches InlineFunction to merge function attributes. This
way, we minimize the "time" when the IR is valid, but the function
attributes are not.

Differential Revision: https://reviews.llvm.org/D134117

show more ...

[ModuleInliner] Set Changed earlier (NFC)

It makes more sense to set Changed to true immediately after a
successful inlining.

[ModuleInliner] Don't include SetVector.h (NFC)

We don't use SetVector in the module inliner.

[ModuleInliner] Move UseInlinePriority to InlineOrder.cpp (NFC)

UseInlinePriority specifies the priority function. This patch
simplifies the code by moving UseInlinePriority closer to the actual
co

[ModuleInliner] Move UseInlinePriority to InlineOrder.cpp (NFC)

UseInlinePriority specifies the priority function. This patch
simplifies the code by moving UseInlinePriority closer to the actual
consumer -- the switch statement inside getInlineOrder.

Differential Revision: https://reviews.llvm.org/D134100

show more ...

[Inliner] Retire DefaultInlineOrder (NFC)

DefaultInlineOrder was largely an exercise in generalizing the
traversal order of call sites within the inliner.

Now that the module inliner is starting to

[Inliner] Retire DefaultInlineOrder (NFC)

DefaultInlineOrder was largely an exercise in generalizing the
traversal order of call sites within the inliner.

Now that the module inliner is starting to form its shape, there is no
point in sharing DefaultInlineOrder between the module inliner and the
CGSCC inliner. DefaultInlineOrder and all the other inline orders are
mutually exclusive in the following sense:

- The use of DefaultInlineOrder doesn't make sense in the module
inliner because there is no priority inherent in the order in which
call sites are added to the list of call sites -- SmallVector.

- The use of any other inline order doesn't make sense in the CGSCC
inliner because little prioritization can be done within one CGSCC.

This patch essentially reverts the addition of DefaultInlineOrder so
that the loop structure of Inliner.cpp looks like the state just
before we started working on the module inliner (circa June 2021).

At the same time, ww remove the choice of DefaultInlineOrder from
UseInlinePriority.

Differential Revision: https://reviews.llvm.org/D134080

show more ...

[ModuleInliner] clang-format ModuleInliner.cpp (NFC)

[ModuleInliner] Remove a stale comment (NFC)

These comments refer to the nested loop in the module inliner where
the inner loop grouped call sites from the same caller. We don't
group call sites an

[ModuleInliner] Remove a stale comment (NFC)

These comments refer to the nested loop in the module inliner where
the inner loop grouped call sites from the same caller. We don't
group call sites anymore, so the comment has become stale.

show more ...

[ModuleInliner] Remove a redundaunt variable (NFC)

In the CGSCC inliner, DidInline was used as an indicator to update the call graph.

In the module inliner, DidInline is always true at the end of t

[ModuleInliner] Remove a redundaunt variable (NFC)

In the CGSCC inliner, DidInline was used as an indicator to update the call graph.

In the module inliner, DidInline is always true at the end of the
"while" loop, so can just drop it.

show more ...

[ModuleInliner] Remove a write-only variable (NFC)

InlinedCallees is a remnant from the CGSCC inliner. We don't use it
in the module inliner.

[IPO] Simplify the module inliner loop (NFC)

In the bottom-up inliner, we have a two-level nested "while" loop,
with the inner one grouping call sites with the same caller. We need
to do so to keep

[IPO] Simplify the module inliner loop (NFC)

In the bottom-up inliner, we have a two-level nested "while" loop,
with the inner one grouping call sites with the same caller. We need
to do so to keep CGSCC up to date.

Now, with the module inliner, we don't have any per-caller work. We
don't update CGSCC. Plus, the caller will likely keep changing as we
pop call sites in some priority order.

This patch simply removes the inner "while" loop while indenting its
body. Further cleanup is possible, but that's left for follow-up
patches.

Differential Revision: https://reviews.llvm.org/D133969

show more ...

[llvm][ModuleInliner] Add inline cost priority for module inliner

This patch introduces the inline cost priority into the
module inliner, which uses the same computation as
InlineCost.

Reviewed By:

[llvm][ModuleInliner] Add inline cost priority for module inliner

This patch introduces the inline cost priority into the
module inliner, which uses the same computation as
InlineCost.

Reviewed By: kazu

Differential Revision: https://reviews.llvm.org/D130012

show more ...

Revert "[llvm][ModuleInliner] Add inline cost priority for module inliner"

This reverts commit bb7f62bbbd35840006a1d202228e835909f591cf.

[llvm][ModuleInliner] Add inline cost priority for module inliner

This patch introduces the inline cost priority into the
module inliner, which uses the same computation as
InlineCost.

Reviewed By:

[llvm][ModuleInliner] Add inline cost priority for module inliner

This patch introduces the inline cost priority into the
module inliner, which uses the same computation as
InlineCost.

Reviewed By: kazu

Differential Revision: https://reviews.llvm.org/D130012

show more ...

[Inline] Annotate inline pass name with link phase information for analysis.

The annotation is flag gated; flag is turned off by default.

Differential Revision: https://reviews.llvm.org/D125495