[NFC][Coroutines] Remove invalid coroutine intrinsic name (#114543)

Removes `llvm.coro.async.store_resume` from the list of coroutine
intrinsics. This is not a valid intrinsic name, and was likely

[NFC][Coroutines] Remove invalid coroutine intrinsic name (#114543)

Removes `llvm.coro.async.store_resume` from the list of coroutine
intrinsics. This is not a valid intrinsic name, and was likely added by
mistake with [this](https://reviews.llvm.org/D90612) change.

show more ...

[NFC] Eliminate use of `lookupLLVMIntrinsicByName` in Coroutines (#114851)

Eliminate use of `lookupLLVMIntrinsicByName` from Coroutines in
preparation of changing it to support a different form of

[NFC] Eliminate use of `lookupLLVMIntrinsicByName` in Coroutines (#114851)

Eliminate use of `lookupLLVMIntrinsicByName` from Coroutines in
preparation of changing it to support a different form of intrinsic name
table generated by intrinsic emitter.

Also eliminate call to `isCoroutineIntrinsicName` from
`declaresAnyIntrinsic` as the list of names traversed is the same list
which `isCoroutineIntrinsicName` checks.

show more ...

[Coro] Use poison instead of undef as placeholder [NFC]

[NFC] Rename `Intrinsic::getDeclaration` to `getOrInsertDeclaration` (#111752)

Rename the function to reflect its correct behavior and to be consistent
with `Module::getOrInsertFunction`. This is a

[NFC] Rename `Intrinsic::getDeclaration` to `getOrInsertDeclaration` (#111752)

Rename the function to reflect its correct behavior and to be consistent
with `Module::getOrInsertFunction`. This is also in preparation of
adding a new `Intrinsic::getDeclaration` that will have behavior similar
to `Module::getFunction` (i.e, just lookup, no creation).

show more ...

[Coroutines] Support for Custom ABIs (#111755)

This change extends the current method for creating ABI object to allow
users (plugin libraries) to create custom ABI objects for their needs.
This i

[Coroutines] Support for Custom ABIs (#111755)

This change extends the current method for creating ABI object to allow
users (plugin libraries) to create custom ABI objects for their needs.
This is accomplished by inheriting one of the common ABIs and overriding
one or more of the methods to create a custom ABI. To use a custom ABI
for a given coroutine the coro.begin.custom.abi intrinsic is used in
place of the coro.begin intrinsic. This takes an additional i32 arg that
specifies the index of an ABI generator for the custom ABI object in a
SmallVector passed to the CoroSplitPass ctor.

The detailed changes include:
* Add the llvm.coro.begin.custom intrinsic used to specify the index of
the custom ABI to use for the given coroutine.
* Add constructors to CoroSplit that take a list of generators that
create the custom ABI object.
* Extend the CreateNewABI function used by CoroSplit to return a
unique_ptr to an ABI object.
* Add has/getCustomABI methods to CoroBeginInst class.
* Add a unittest for a custom ABI.

See doc update here: https://github.com/llvm/llvm-project/pull/111781

show more ...

[Coroutines] Move util headers to include/llvm (#111599)

Plugin libraries that use coroutines can do so right now, however, to
provide their own ABI they need to be able to use various headers, som

[Coroutines] Move util headers to include/llvm (#111599)

Plugin libraries that use coroutines can do so right now, however, to
provide their own ABI they need to be able to use various headers, some
of which such are required (such as the ABI header). This change exposes
the coro utils and required headers by moving them to
include/llvm/Transforms/Coroutines. My experience with our out-of-tree
plugin ABI has been that at least these headers are needed. The headers
moved are:
* ABI.h (ABI object)
* CoroInstr.h (helpers)
* Coroshape.h (Shape object)
* MaterializationUtils.h (helpers)
* SpillingUtils.h (helpers)
* SuspendCrossingInfo.h (analysis)

This has no code changes other than those required to move the headers
and these are:
* include guard name changes
* include path changes
* minor clang-format induced changes
* removal of LLVM_LIBRARY_VISIBILITY

show more ...

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109713)

This patch re-lands https://github.com/llvm/llvm-project/pull/109338 and
fixes the various

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109713)

This patch re-lands https://github.com/llvm/llvm-project/pull/109338 and
fixes the various test failures.

--- Original description ---

* Adds an ABI object class hierarchy to implement the coroutine ABIs
(Switch, Asyc, and Retcon{Once})
* The ABI object improves the separation of the code related to users,
ABIs and utilities.
* No code changes are required by any existing users.
* Each ABI overrides delegate methods for initialization, building the
coroutine frame and splitting the coroutine, other methods may be added
later.
* CoroSplit invokes a generator lambda to instantiate the ABI object and
calls the ABI object to carry out its primary operations. In a follow-up
change this will be used to instantiated customized ABIs according to a
new intrinsic llvm.coro.begin.custom.abi.
* Note, in a follow-up change additional constructors will be added to
the ABI objects (Switch, Asyc, and AnyRetcon) to allow a set of
generator lambdas to be passed in from a CoroSplit constructor that will
also be added. The init() method is separate from the constructor to
avoid duplication of its code. It is a virtual method so it can be
invoked by CreateAndInitABI or potentially CoroSplit::run(). I wasn't
sure if we should call init() from within CoroSplit::run(),
CreateAndInitABI, or perhaps hard code a call in each constructor. One
consideration is that init() can change the IR, so perhaps it should
appear in CoroSplit::run(), but this looks a bit odd. Invoking init() in
the constructor would result in the call appearing 4 times per ABI
(after adding the new constructors). Invoking it in CreateAndInitABI
seems to be a balance between these.

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[Core] Skip over target name in intrinsic name lookup (#109971)

When searching for an intrinsic name in a target specific slice of the
intrinsic name table, skip over the target prefix. For such ca

[Core] Skip over target name in intrinsic name lookup (#109971)

When searching for an intrinsic name in a target specific slice of the
intrinsic name table, skip over the target prefix. For such cases,
currently the first loop iteration in `lookupLLVMIntrinsicByName` does
nothing (i.e., `Low` and `High` stay unchanged and it does not shrink
down the search window), so we can skip this useless first iteration by
skipping over the target prefix.

show more ...

Revert "[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)"

This reverts commit 2e414799d0ad511cd7999895014a2cae2ea5e3e3.

Reason: buildbot br

Revert "[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)"

This reverts commit 2e414799d0ad511cd7999895014a2cae2ea5e3e3.

Reason: buildbot breakage
(https://lab.llvm.org/buildbot/#/builders/51/builds/4105)
(This was the only new CL.)

/home/b/sanitizer-aarch64-linux/build/llvm-project/llvm/lib/Transforms/Coroutines/ABI.h:71:31: error: 'llvm::coro::AsyncABI' has virtual functions but non-virtual destructor [-Werror,-Wnon-virtual-dtor]
71 | class LLVM_LIBRARY_VISIBILITY AsyncABI : public BaseABI {

etc.

show more ...

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)

* Adds an ABI object class hierarchy to implement the coroutine ABIs
(Switch, Asyc, and Re

[Coroutines] ABI Objects to improve code separation between different ABIs, users and utilities. (#109338)

* Adds an ABI object class hierarchy to implement the coroutine ABIs
(Switch, Asyc, and Retcon{Once})
* The ABI object improves the separation of the code related to users,
ABIs and utilities.
* No code changes are required by any existing users.
* Each ABI overrides delegate methods for initialization, building the
coroutine frame and splitting the coroutine, other methods may be added
later.
* CoroSplit invokes a generator lambda to instantiate the ABI object and
calls the ABI object to carry out its primary operations.

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[Coroutines] Fix a warning

This patch fixes:

llvm/lib/Transforms/Coroutines/Coroutines.cpp:475:3: error: default
label in switch which covers all enumeration values
[-Werror,-Wcovered-switch-

[Coroutines] Fix a warning

This patch fixes:

llvm/lib/Transforms/Coroutines/Coroutines.cpp:475:3: error: default
label in switch which covers all enumeration values
[-Werror,-Wcovered-switch-default]

show more ...

[Coroutines] Refactor CoroShape::buildFrom for future use by ABI objects (#108623)

* Refactor buildFrom to separate the analysis, abi related operations,
cleaning and invalidating.
* In a follow-u

[Coroutines] Refactor CoroShape::buildFrom for future use by ABI objects (#108623)

* Refactor buildFrom to separate the analysis, abi related operations,
cleaning and invalidating.
* In a follow-up PR the code in initABI will be moved to an ABI object
init method.
* In a follow-up PR the OptimizeFrame flag will also be removed from the
Shape and instead will be passed directly to buildCoroutineFrame
(although it would be nice to find another way to trigger this
optimization). This is the only thing that Shape cannot determine from
the Function/Coroutine, but it is only needed within
buildCoroutineFrame.

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[Coroutines] Move Shape to its own header (#108242)

* To create custom ABIs plugin libraries need access to CoroShape.
* As a step in enabling plugin libraries, move Shape into its own header
* Th

[Coroutines] Move Shape to its own header (#108242)

* To create custom ABIs plugin libraries need access to CoroShape.
* As a step in enabling plugin libraries, move Shape into its own header
* The header will eventually be moved into include/llvm/Transforms/Coroutines

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[Coroutines] Move materialization code into its own utils (#108240)

* Move materialization out of CoroFrame to MaterializationUtils.h
* Move spill related utilities that were used by materializatio

[Coroutines] Move materialization code into its own utils (#108240)

* Move materialization out of CoroFrame to MaterializationUtils.h
* Move spill related utilities that were used by materialization to
SpillUtils
* Move isSuspendBlock (needed by materialization) to CoroInternal

See RFC for more info:
https://discourse.llvm.org/t/rfc-abi-objects-for-coroutines/81057

show more ...

[LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (#99283)

This patch is episode two of the coroutine HALO improvement project
published on discou

[LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (#99283)

This patch is episode two of the coroutine HALO improvement project
published on discourse:
https://discourse.llvm.org/t/language-extension-for-better-more-deterministic-halo-for-c-coroutines/80044

Previously CoroElide depends on inlining, and its analysis does not work
very well with code generated by the C++ frontend due the existence of
many customization points. There has been issue reported to upstream how
ineffective the original CoroElide was in real world applications.

For C++ users, this set of patches aim to fix this problem by providing
library authors and users deterministic HALO behaviour for some
well-behaved coroutine `Task` types. The stack begins with a library
side attribute on the `Task` class that guarantees no unstructured
concurrency when coroutines are awaited directly with `co_await`ed as a
prvalue. This attribute on Task types gives us lifetime guarantees and
makes C++ FE capable to telling the ME which coroutine calls are
elidable. We convey such information from FE through the attribute
`coro_elide_safe`.

This patch modifies CoroSplit to create a variant of the coroutine ramp
function that 1) does not use heap allocated frame, instead take an
additional parameter as the pointer to the frame. Such parameter is
attributed with `dereferenceble` and `align` to convey size and align
requirements for the frame. 2) always stores cleanup instead of destroy
address for `coro.destroy()` actions.

In a later patch, we will have a new pass that runs right after
CoroSplit to find usages of the callee coroutine attributed
`coro_elide_safe` in presplit coroutine callers, allocates the frame on
its "stack", transform those usages to call the `noalloc` ramp function
variant.

(note I put quotes on the word "stack" here, because for presplit
coroutine, any alloca will be spilled into the frame when it's being
split)

The C++ Frontend attribute implementation that works with this change
can be found at https://github.com/llvm/llvm-project/pull/99282
The pass that makes use of the new `noalloc` split can be found at
https://github.com/llvm/llvm-project/pull/99285

show more ...

[coro] Lower `llvm.coro.await.suspend.handle` to resume with tail call (#89751)

The C++ standard requires that symmetric transfer from one coroutine to
another is performed via a tail call. Failure

[coro] Lower `llvm.coro.await.suspend.handle` to resume with tail call (#89751)

The C++ standard requires that symmetric transfer from one coroutine to
another is performed via a tail call. Failure to do so is a miscompile
and often breaks programs by quickly overflowing the stack.

Until now, the coro split pass tried to ensure this in the
`addMustTailToCoroResumes()` function by searching for
`llvm.coro.resume` calls to lower as tail calls if the conditions were
right: the right function arguments, attributes, calling convention
etc., and if a `ret void` was sure to be reached after traversal with
some ad-hoc constant folding following the call.

This was brittle, as the kind of implicit variants required for a tail
call to happen could easily be broken by other passes (e.g. if some
instruction got in between the `resume` and `ret`), see for example
9d1cb18d19862fc0627e4a56e1e491a498e84c71 and
284da049f5feb62b40f5abc41dda7895e3d81d72.

Also the logic seemed backwards: instead of searching for possible tail
call candidates and doing them if the circumstances are right, it seems
better to start with the intention of making the tail calls we need, and
forcing the circumstances to be right.

Now that we have the `llvm.coro.await.suspend.handle` intrinsic (since
f78688134026686288a8d310b493d9327753a022) which corresponds exactly to
symmetric transfer, change the lowering of that to also include the
`resume` part, always lowered as a tail call.

show more ...

[coroutine] Implement llvm.coro.await.suspend intrinsic (#79712)

Implement `llvm.coro.await.suspend` intrinsics, to deal with performance
regression after prohibiting `.await_suspend` inlining, as

[coroutine] Implement llvm.coro.await.suspend intrinsic (#79712)

Implement `llvm.coro.await.suspend` intrinsics, to deal with performance
regression after prohibiting `.await_suspend` inlining, as suggested in
#64945.
Actually, there are three new intrinsics, which directly correspond to
each of three forms of `await_suspend`:
```
void llvm.coro.await.suspend.void(ptr %awaiter, ptr %frame, ptr @wrapperFunction)
i1 llvm.coro.await.suspend.bool(ptr %awaiter, ptr %frame, ptr @wrapperFunction)
ptr llvm.coro.await.suspend.handle(ptr %awaiter, ptr %frame, ptr @wrapperFunction)
```
There are three different versions instead of one, because in `bool`
case it's result is used for resuming via a branch, and in
`coroutine_handle` case exceptions from `await_suspend` are handled in
the coroutine, and exceptions from the subsequent `.resume()` are
propagated to the caller.

Await-suspend block is simplified down to intrinsic calls only, for
example for symmetric transfer:
```
%id = call token @llvm.coro.save(ptr null)
%handle = call ptr @llvm.coro.await.suspend.handle(ptr %awaiter, ptr %frame, ptr @wrapperFunction)
call void @llvm.coro.resume(%handle)
%result = call i8 @llvm.coro.suspend(token %id, i1 false)
switch i8 %result, ...
```
All await-suspend logic is moved out into a wrapper function, generated
for each suspension point.
The signature of the function is `<type> wrapperFunction(ptr %awaiter,
ptr %frame)` where `<type>` is one of `void` `i1` or `ptr`, depending on
the return type of `await_suspend`.
Intrinsic calls are lowered during `CoroSplit` pass, right after the
split.

Because I'm new to LLVM, I'm not sure if the helper function generation,
calls to them and lowering are implemented in the right way, especially
with regard to various metadata and attributes, i. e. for TBAA. All
things that seemed questionable are marked with `FIXME` comments.

There is another detail: in case of symmetric transfer raw pointer to
the frame of coroutine, that should be resumed, is returned from the
helper function and a direct call to `@llvm.coro.resume` is generated.
C++ standard demands, that `.resume()` method is evaluated. Not sure how
important is this, because code has been generated in the same way
before, sans helper function.

show more ...

[NFC][RemoveDIs] Insert instruction using iterators in Transforms/

As part of the RemoveDIs project we need LLVM to insert instructions using
iterators wherever possible, so that the iterators can c

[NFC][RemoveDIs] Insert instruction using iterators in Transforms/

As part of the RemoveDIs project we need LLVM to insert instructions using
iterators wherever possible, so that the iterators can carry a bit of
debug-info. This commit implements some of that by updating the contents of
llvm/lib/Transforms/Utils to always use iterator-versions of instruction
constructors.

There are two general flavours of update:
* Almost all call-sites just call getIterator on an instruction
* Several make use of an existing iterator (scenarios where the code is
actually significant for debug-info)
The underlying logic is that any call to getFirstInsertionPt or similar
APIs that identify the start of a block need to have that iterator passed
directly to the insertion function, without being converted to a bare
Instruction pointer along the way.

Noteworthy changes:
* FindInsertedValue now takes an optional iterator rather than an
instruction pointer, as we need to always insert with iterators,
* I've added a few iterator-taking versions of some value-tracking and
DomTree methods -- they just unwrap the iterator. These are purely
convenience methods to avoid extra syntax in some passes.
* A few calls to getNextNode become std::next instead (to keep in the
theme of using iterators for positions),
* SeparateConstOffsetFromGEP has it's insertion-position field changed.
Noteworthy because it's not a purely localised spelling change.

All this should be NFC.

show more ...

[llvm][Coroutines] Remove no-op ptr-to-ptr bitcasts (NFC) (#73427)

Opaque ptr cleanup effort

[llvm] Drop some bitcasts and references related to typed pointers

Differential Revision: https://reviews.llvm.org/D157551

[Coroutines] Remove unused variable (NFC)

[llvm] Remove uses of isOpaqueOrPointeeTypeEquals() (NFC)

[llvm][clang] Remove uses of isOpaquePointerTy() (NFC)

This now always returns true (for pointer types).

[Transforms] Qualify auto in range-based for loops (NFC)

Identified with readability-qualified-auto.

[Coroutines] Store the index for final suspend point if there is unwind coro end

Closing https://github.com/llvm/llvm-project/issues/57339

The root cause for this issue is an pre-mature optimizatio

[Coroutines] Store the index for final suspend point if there is unwind coro end

Closing https://github.com/llvm/llvm-project/issues/57339

The root cause for this issue is an pre-mature optimization to eliminate
the index for the final suspend point since we feel like we can judge
if a coroutine is suspended at the final suspend by if resume_fn_addr is
null. However this is not true if the coroutine exists via an exception
in promise.unhandled_exception(). According to
[dcl.fct.def.coroutine]p14:

> If the evaluation of the expression promise.unhandled_exception()
> exits via an exception, the coroutine is considered suspended at the
> final suspend point.

But from the perspective of the implementation, we can't set the coro
index to the final suspend point directly since it breaks the states.

To fix the issue, we block the optimization if we find there is any
unwind coro end, which indicates that it is possible that the coroutine
exists via an exception from promise.unhandled_exception().

Test Plan: folly

show more ...