[NFC][DebugInfo] Rewrite more call-sites to insert with iterators (#124288)

As part of the "RemoveDIs" work to eliminate debug intrinsics, we're
replacing methods that use Instruction*'s as position

[NFC][DebugInfo] Rewrite more call-sites to insert with iterators (#124288)

As part of the "RemoveDIs" work to eliminate debug intrinsics, we're
replacing methods that use Instruction*'s as positions with iterators. The
call-sites updated in this patch are those where the dyn_cast_or_null cast
utility doesn't compose well with iterator insertion. It can distinguish
between nullptr and a "present" (non-null) Instruction pointer, but not
between a legal and illegal instruction iterator. This can lead to
end-iterator dereferences and thus crashes.

We can improve this in the future (as parent-pointers can now be accessed
from ilist nodes), but for the moment, add explicit tests for end()
iterators at the five call sites affected by this.

show more ...

[NFC][DebugInfo] Make some block-start-position methods return iterators (#124287)

As part of the "RemoveDIs" work to eliminate debug intrinsics, we're
replacing methods that use Instruction*'s as

[NFC][DebugInfo] Make some block-start-position methods return iterators (#124287)

As part of the "RemoveDIs" work to eliminate debug intrinsics, we're
replacing methods that use Instruction*'s as positions with iterators. A
number of these (such as getFirstNonPHIOrDbg) are sufficiently
infrequently used that we can just replace the pointer-returning version
with an iterator-returning version, hopefully without much/any
disruption.

Thus this patch has getFirstNonPHIOrDbg and
getFirstNonPHIOrDbgOrLifetime return an iterator, and updates all
call-sites. There are no concerns about the iterators returned being
converted to Instruction*'s and losing the debug-info bit: because the
methods skip debug intrinsics, the iterator head bit is always false
anyway.

show more ...

[NFC][DebugInfo] Use iterators for instruction insertion in more places (#124291)

As part of the "RemoveDIs" work to eliminate debug intrinsics, we're
replacing methods that use Instruction*'s as p

[NFC][DebugInfo] Use iterators for instruction insertion in more places (#124291)

As part of the "RemoveDIs" work to eliminate debug intrinsics, we're
replacing methods that use Instruction*'s as positions with iterators.
This patch changes some more complex call-sites, those crossing file
boundaries and where I've had to perform some minor rewrites.

show more ...

[Coro] Prebuild a module-level debug info set and share it between all coroutine clones (#118628)


Summary:
CoroCloner, by calling into CloneFunctionInto, does a lot of repeated
work priming DIFinde

[Coro] Prebuild a module-level debug info set and share it between all coroutine clones (#118628)


Summary:
CoroCloner, by calling into CloneFunctionInto, does a lot of repeated
work priming DIFinder and building a list of common module-level debug
info metadata. For programs compiled with full debug info this can get
very expensive.

This diff builds the data once and shares it between all clones.

Anecdata for a sample cpp source file compiled with full debug info:

| | Baseline | IdentityMD set | Prebuilt CommonDI (cur.) |
|-----------------|----------|----------------|--------------------------|
| CoroSplitPass | 306ms | 221ms | 68ms |
| CoroCloner | 101ms | 72ms | 0.5ms |
| CollectCommonDI | - | - | 63ms |
| Speed up | 1x | 1.4x | 4.5x |

Note that CollectCommonDebugInfo happens once *per coroutine* rather than per clone.

Test Plan:
ninja check-llvm-unit
ninja check-llvm

Compiled a sample internal source file, checked time trace output for scope timings.

show more ...

[IR] Replace of PointerType::getUnqual(Type) with opaque version (NFC) (#123909)

Follow up to https://github.com/llvm/llvm-project/issues/123569

[Coroutines][NFC] Refactor CoroCloner (#116885)

* Move CoroCloner to its own header. For now, the header is located in llvm/lib/Transforms/Coroutines
* Change private to protected to allow inherita

[Coroutines][NFC] Refactor CoroCloner (#116885)

* Move CoroCloner to its own header. For now, the header is located in llvm/lib/Transforms/Coroutines
* Change private to protected to allow inheritance
* Create CoroSwitchCloner and move some of the switch specific code into this cloner. More code will follow in later commits.

show more ...

[NFC][Coroutines] Use structured binding with llvm::enumerate in CoroSplit (#116879)

Avoid repeated calls to value() and index() using structured binding
with llvm::enumerate.

[NFC][Coroutines] Remove integer indexing in several CoroSplit loops (#115954)

Use helpers such as llvm::enumerate and llvm::zip in places to avoid
using loop counters. Also refactored AnyRetconABI

[NFC][Coroutines] Remove integer indexing in several CoroSplit loops (#115954)

Use helpers such as llvm::enumerate and llvm::zip in places to avoid
using loop counters. Also refactored AnyRetconABI::splitCoroutine to
avoid some awkward indexing that came about by putting ContinuationPhi
into the ReturnPHIs vector and mistaking i with I and e with E.

show more ...

[Coroutines] Respect noinline attributes when eliding heap allocation (#115384)

[Coro] Use poison instead of undef as placeholder [NFC]

[NFC][Coro] Add helpers for coro cloning with a TimeTraceScope (#112948)

A helper (2 overloads) that consolidates corocloner creation and the
actual cloning. The helpers create a TimeTraceScope to

[NFC][Coro] Add helpers for coro cloning with a TimeTraceScope (#112948)

A helper (2 overloads) that consolidates corocloner creation and the
actual cloning. The helpers create a TimeTraceScope to make it easier to
see how long the cloning takes.

Extracted from #109032 (commit 1)

show more ...

[Coroutines] Improve use of unique_ptr (#111870)

* Replace usage of unique_ptr<>(new ...) -> make_unique<>();

[Coroutines] Support for Custom ABIs (#111755)

This change extends the current method for creating ABI object to allow
users (plugin libraries) to create custom ABI objects for their needs.
This i

[Coroutines] Support for Custom ABIs (#111755)

This change extends the current method for creating ABI object to allow
users (plugin libraries) to create custom ABI objects for their needs.
This is accomplished by inheriting one of the common ABIs and overriding
one or more of the methods to create a custom ABI. To use a custom ABI
for a given coroutine the coro.begin.custom.abi intrinsic is used in
place of the coro.begin intrinsic. This takes an additional i32 arg that
specifies the index of an ABI generator for the custom ABI object in a
SmallVector passed to the CoroSplitPass ctor.

The detailed changes include:
* Add the llvm.coro.begin.custom intrinsic used to specify the index of
the custom ABI to use for the given coroutine.
* Add constructors to CoroSplit that take a list of generators that
create the custom ABI object.
* Extend the CreateNewABI function used by CoroSplit to return a
unique_ptr to an ABI object.
* Add has/getCustomABI methods to CoroBeginInst class.
* Add a unittest for a custom ABI.

See doc update here: https://github.com/llvm/llvm-project/pull/111781

show more ...

[Coroutines] Move util headers to include/llvm (#111599)

Plugin libraries that use coroutines can do so right now, however, to
provide their own ABI they need to be able to use various headers, som

[Coroutines] Move util headers to include/llvm (#111599)

Plugin libraries that use coroutines can do so right now, however, to
provide their own ABI they need to be able to use various headers, some
of which such are required (such as the ABI header). This change exposes
the coro utils and required headers by moving them to
include/llvm/Transforms/Coroutines. My experience with our out-of-tree
plugin ABI has been that at least these headers are needed. The headers
moved are:
* ABI.h (ABI object)
* CoroInstr.h (helpers)
* Coroshape.h (Shape object)
* MaterializationUtils.h (helpers)
* SpillingUtils.h (helpers)
* SuspendCrossingInfo.h (analysis)

This has no code changes other than those required to move the headers
and these are:
* include guard name changes
* include path changes
* minor clang-format induced changes
* removal of LLVM_LIBRARY_VISIBILITY

show more ...

[Coroutines] Move OptimizeFrame out of Shape (#111017)

* OptimizeFrame is not really a part of the Coroutine Shape info, rather
it is specifically for the addFieldForAllocas method called indirectly

[Coroutines] Move OptimizeFrame out of Shape (#111017)

* OptimizeFrame is not really a part of the Coroutine Shape info, rather
it is specifically for the addFieldForAllocas method called indirectly
by buildCoroutineFrame.
* This patch passes OptimizeFrame directly to buildCoroutineFrame so it
can be provided to addFieldForAllocas instead of keeping it in the
Shape.

Co-authored-by: tnowicki <tnowicki.nowicki@amd.com>

show more ...

[Coroutines] Fix warnings

This patch fixes:

llvm/include/llvm/Transforms/Coroutines/CoroSplit.h:26:1: note: did
you mean struct here?

llvm/lib/Transforms/Coroutines/CoroSplit.cpp:2225:16: er

[Coroutines] Fix warnings

This patch fixes:

llvm/include/llvm/Transforms/Coroutines/CoroSplit.h:26:1: note: did
you mean struct here?

llvm/lib/Transforms/Coroutines/CoroSplit.cpp:2225:16: error: moving
a local object in a return statement prevents copy elision
[-Werror,-Wpessimizing-move]

llvm/lib/Transforms/Coroutines/CoroSplit.cpp:2236:16: error: moving
a local object in a return statement prevents copy elision
[-Werror,-Wpessimizing-move]

show more ...

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109713)

This patch re-lands https://github.com/llvm/llvm-project/pull/109338 and
fixes the various

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109713)

This patch re-lands https://github.com/llvm/llvm-project/pull/109338 and
fixes the various test failures.

--- Original description ---

* Adds an ABI object class hierarchy to implement the coroutine ABIs
(Switch, Asyc, and Retcon{Once})
* The ABI object improves the separation of the code related to users,
ABIs and utilities.
* No code changes are required by any existing users.
* Each ABI overrides delegate methods for initialization, building the
coroutine frame and splitting the coroutine, other methods may be added
later.
* CoroSplit invokes a generator lambda to instantiate the ABI object and
calls the ABI object to carry out its primary operations. In a follow-up
change this will be used to instantiated customized ABIs according to a
new intrinsic llvm.coro.begin.custom.abi.
* Note, in a follow-up change additional constructors will be added to
the ABI objects (Switch, Asyc, and AnyRetcon) to allow a set of
generator lambdas to be passed in from a CoroSplit constructor that will
also be added. The init() method is separate from the constructor to
avoid duplication of its code. It is a virtual method so it can be
invoked by CreateAndInitABI or potentially CoroSplit::run(). I wasn't
sure if we should call init() from within CoroSplit::run(),
CreateAndInitABI, or perhaps hard code a call in each constructor. One
consideration is that init() can change the IR, so perhaps it should
appear in CoroSplit::run(), but this looks a bit odd. Invoking init() in
the constructor would result in the call appearing 4 times per ABI
(after adding the new constructors). Invoking it in CreateAndInitABI
seems to be a balance between these.

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

Revert "[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)"

This reverts commit 2e414799d0ad511cd7999895014a2cae2ea5e3e3.

Reason: buildbot br

Revert "[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)"

This reverts commit 2e414799d0ad511cd7999895014a2cae2ea5e3e3.

Reason: buildbot breakage
(https://lab.llvm.org/buildbot/#/builders/51/builds/4105)
(This was the only new CL.)

/home/b/sanitizer-aarch64-linux/build/llvm-project/llvm/lib/Transforms/Coroutines/ABI.h:71:31: error: 'llvm::coro::AsyncABI' has virtual functions but non-virtual destructor [-Werror,-Wnon-virtual-dtor]
71 | class LLVM_LIBRARY_VISIBILITY AsyncABI : public BaseABI {

etc.

show more ...

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)

* Adds an ABI object class hierarchy to implement the coroutine ABIs
(Switch, Asyc, and Re

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)

* Adds an ABI object class hierarchy to implement the coroutine ABIs
(Switch, Asyc, and Retcon{Once})
* The ABI object improves the separation of the code related to users,
ABIs and utilities.
* No code changes are required by any existing users.
* Each ABI overrides delegate methods for initialization, building the
coroutine frame and splitting the coroutine, other methods may be added
later.
* CoroSplit invokes a generator lambda to instantiate the ABI object and
calls the ABI object to carry out its primary operations.

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[CoroSplit][DebugInfo] Adjust heuristic for moving DIScope of funclets (#108611)

CoroSplit has a heuristic where the scope line for funclets is adjusted
to match the line of the suspend intrinsic t

[CoroSplit][DebugInfo] Adjust heuristic for moving DIScope of funclets (#108611)

CoroSplit has a heuristic where the scope line for funclets is adjusted
to match the line of the suspend intrinsic that caused the split. This
is useful as it avoids a jump on the line table from the original
function declaration to the line where the split happens.

However, very often using the line of the split is not ideal: if we can
avoid it, we should not have a line entry for the split location, as
this would cause breakpoints by line to match against two functions: the
funclet before and the funclet after the split.

This patch adjusts the heuristics to look for the first instruction with
a non-zero line number after the split. In other words, this patch makes
breakpoints on `await foo()` lines behave much more like a regular
function call.

show more ...

[Coroutines] properly update CallGraph in CoroSplit (#107935)

Fixes https://github.com/llvm/llvm-project/issues/107139.

We weren't updating the call graph properly in CoroSplit. This crash is
du

[Coroutines] properly update CallGraph in CoroSplit (#107935)

Fixes https://github.com/llvm/llvm-project/issues/107139.

We weren't updating the call graph properly in CoroSplit. This crash is
due to the await_suspend() function calling the coroutine, forming a
multi-node SCC. The issue bisected to
https://github.com/llvm/llvm-project/pull/79712 but I think this is red
herring. We haven't been properly updating the call graph.

Added an example of such code as a test case.

show more ...

[Coroutines] Split buildCoroutineFrame into normalization and frame building (#108076)

* Split buildCoroutineFrame into code related to normalization and code
related to actually building the corou

[Coroutines] Split buildCoroutineFrame into normalization and frame building (#108076)

* Split buildCoroutineFrame into code related to normalization and code
related to actually building the coroutine frame.
* This will enable future specialization of buildCoroutineFrame for
different ABIs while the normalization can be done by splitCoroutine
prior to calling buildCoroutineFrame.

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (#99283)

This patch is episode two of the coroutine HALO improvement project
published on discou

[LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (#99283)

This patch is episode two of the coroutine HALO improvement project
published on discourse:
https://discourse.llvm.org/t/language-extension-for-better-more-deterministic-halo-for-c-coroutines/80044

Previously CoroElide depends on inlining, and its analysis does not work
very well with code generated by the C++ frontend due the existence of
many customization points. There has been issue reported to upstream how
ineffective the original CoroElide was in real world applications.

For C++ users, this set of patches aim to fix this problem by providing
library authors and users deterministic HALO behaviour for some
well-behaved coroutine `Task` types. The stack begins with a library
side attribute on the `Task` class that guarantees no unstructured
concurrency when coroutines are awaited directly with `co_await`ed as a
prvalue. This attribute on Task types gives us lifetime guarantees and
makes C++ FE capable to telling the ME which coroutine calls are
elidable. We convey such information from FE through the attribute
`coro_elide_safe`.

This patch modifies CoroSplit to create a variant of the coroutine ramp
function that 1) does not use heap allocated frame, instead take an
additional parameter as the pointer to the frame. Such parameter is
attributed with `dereferenceble` and `align` to convey size and align
requirements for the frame. 2) always stores cleanup instead of destroy
address for `coro.destroy()` actions.

In a later patch, we will have a new pass that runs right after
CoroSplit to find usages of the callee coroutine attributed
`coro_elide_safe` in presplit coroutine callers, allocates the frame on
its "stack", transform those usages to call the `noalloc` ramp function
variant.

(note I put quotes on the word "stack" here, because for presplit
coroutine, any alloca will be spilled into the frame when it's being
split)

The C++ Frontend attribute implementation that works with this change
can be found at https://github.com/llvm/llvm-project/pull/99282
The pass that makes use of the new `noalloc` split can be found at
https://github.com/llvm/llvm-project/pull/99285

show more ...

[Coroutines] Salvage the debug information for coroutine frames within optimizations

This patch tries to salvage the debug information for the coroutine
frames within optimizations by creating the h

[Coroutines] Salvage the debug information for coroutine frames within optimizations

This patch tries to salvage the debug information for the coroutine
frames within optimizations by creating the help alloca varaibles with
optimizations too. We didn't do this when I implement it initially. I
roughtly remember the reason was, we feel the additional help alloca
variable may pessimize the performance, which is almost the most
important thing under optimizations. But now, it looks like the new
inserted help alloca variables can be optimized out by the following
optimizations. So it looks like the time to make it available within
optimizations.

And also, it looks like the following optimizations will convert the
generated dbg.declare instrinsic into dbg.value intrinsic within
optimizations.

In LLVM's test, there is a slightly regression
that a dbg.declare for the promise object failed to be remained after
this change. But it looks like we won't have a chance to see dbg.declare
for the promise object when we split the coroutine as that dbg.declare
will be converted into a dbg.value in early stage.

So everything looks fine.

show more ...

[llvm/llvm-project][Coroutines] Improve debugging and minor refactoring (#104642)

No Functional Changes

* Fix comments in several places
* Instead of using BB-getName() (in dump methods) use
ge

[llvm/llvm-project][Coroutines] Improve debugging and minor refactoring (#104642)

No Functional Changes

* Fix comments in several places
* Instead of using BB-getName() (in dump methods) use
getBasicBlockLabel. This fixes the poor output of the dumped info that
resulted in missing BB labels.
* Use RPO when dumping SuspendCrossingInfo. Without this the dump order
is determined by the ptr addresses and so is not consistent from run to
run making IR diffs difficult to read.
* Inference -> Interference
* Pull the logic that determines insertion location out of insertSpills
and into getSpillInsertionPt, to differentiate between these two
operations.
* Use Shape getters for CoroId instead of getting it manually.

---------

Co-authored-by: tnowicki <tnowicki.nowicki@amd.com>

show more ...