[RS4GC] Cleanup meetBDVState. NFC.

meetBDVState looks pretty difficult to read and follow.
This is purely NFC but doing several things:

1) Combine meet and meetBDVState
2) Move the function to be a

[RS4GC] Cleanup meetBDVState. NFC.

meetBDVState looks pretty difficult to read and follow.
This is purely NFC but doing several things:

1) Combine meet and meetBDVState
2) Move the function to be a member of BDVState
3) Make BDVState be a mutable object
4) Convert switch to sequence of ifs
5) Adds comments.

Reviewers: reames, dantrushin
Reviewed By: reames
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D99064

show more ...

[rs4gc] Use loops instead of straightline code for attribute stripping [nfc]

Mostly because I'm about to add more attributes and the straightline copies get much uglier. What's currently there isn'

[rs4gc] Use loops instead of straightline code for attribute stripping [nfc]

Mostly because I'm about to add more attributes and the straightline copies get much uglier. What's currently there isn't too bad.

show more ...

[rs4gc] Strip nofree and nosync attributes when lowering from abstract model

The safepoints being inserted exists to free memory, or coordinate with another thread to do so. Thus, we must strip any

[rs4gc] Strip nofree and nosync attributes when lowering from abstract model

The safepoints being inserted exists to free memory, or coordinate with another thread to do so. Thus, we must strip any inferred attributes and reinfer them after the lowering.

I'm not aware of any active miscompiles caused by this, but since I'm working on strengthening inference of both and leveraging them in the optimization decisions, I figured a bit of future proofing was warranted.

show more ...

[RS4GC] Fix hang on infinite loop

meetBDVState utility may sets the base pointer for the conflict state.
At this moment the base for conflict state does not have any meaning but
is used in compariso

[RS4GC] Fix hang on infinite loop

meetBDVState utility may sets the base pointer for the conflict state.
At this moment the base for conflict state does not have any meaning but
is used in comparison of BDV states. This comparison is used as an indicator
of progress done on iteration and RS4GC pass uses infinite loop to reach
fixed point.
As a result for added test on each iteration state for some phi nodes is updated
with other base value for conflict state and it indicates as a progress while
for conflict state there is no any progress more possible.
In reality the base value is transferred from one state to another and pass
detects the progress on these states.

The test is very fragile. The traversal order of states and operands of phi nodes
plays important role.

Reviewers: reames, dantrushin
Reviewed By: reames
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D99058

show more ...

[rs4gc] Simplify code by cloning existing instructions when inserting base chain [NFC]

Previously we created a new node, then filled in the pieces. Now, we clone the existing node, then change the r

[rs4gc] Simplify code by cloning existing instructions when inserting base chain [NFC]

Previously we created a new node, then filled in the pieces. Now, we clone the existing node, then change the respective fields. The only change in handling is with phis since we have to handle multiple incoming edges from the same block a bit differently.

Differential Revision: https://reviews.llvm.org/D98316

show more ...

[rs4gc] don't force a conflict for a canonical broadcast

A broadcast is a shufflevector where only one input is used. Because of the way we handle constants (undef is a constant), the canonical shuf

[rs4gc] don't force a conflict for a canonical broadcast

A broadcast is a shufflevector where only one input is used. Because of the way we handle constants (undef is a constant), the canonical shuffle sees a meet of (some value) and (nullptr). Given this, every broadcast gets treated as a conflict and a new base pointer computation is added.

The other way to tackle this would be to change constant handling specifically for undefs, but this seems easier.

Differential Revision: https://reviews.llvm.org/D98315

show more ...

[rs4gc] don't duplicate existing values which are provably base pointers

RS4GC needs to rewrite the IR to ensure that every relocated pointer has an associated base pointer. The existing code isn't

[rs4gc] don't duplicate existing values which are provably base pointers

RS4GC needs to rewrite the IR to ensure that every relocated pointer has an associated base pointer. The existing code isn't particularly smart about avoiding duplication of existing IR when it turns out the original pointer we were asked to materialize a base pointer for is itself a base pointer.

This patch adds a stage to the algorithm which prunes nodes proven (with a simple forward dataflow fixed point) to be base pointers from the list of nodes considered for duplication. This does require changing some of the later invariants slightly, that's probably the riskiest part of the change.

Differential Revision: D98122

show more ...

[rs4gc] common bdv operand visitation [nfc]

[rs4gc] Fix -Wunused-variable in -DLLVM_ENABLE_ASSERTIONS=off builds

[rs4gc] track the original value in the state use for base pointer rewriting

I'd originally intended to build on this for another purpose and have decided not to, but at a minimum, the stronger asse

[rs4gc] track the original value in the state use for base pointer rewriting

I'd originally intended to build on this for another purpose and have decided not to, but at a minimum, the stronger asserts are useful.

show more ...

[rs4gc] minor code style improvement

[rs4gc] avoid insert base computation instructions for deopt uses

If we have a value live over a call which is used for deopt at the call, we know that the value must be a base pointer. We can avoid

[rs4gc] avoid insert base computation instructions for deopt uses

If we have a value live over a call which is used for deopt at the call, we know that the value must be a base pointer. We can avoid potentially inserting IR to materialize a base for this value.

In it's current form, this is mostly a compile time optimization. Building the base pointer graph (and then optimizing it away again) is a relatively expensive operation. We also sometimes end up with better codegen in practice - due to failures in optimizing away the inserted base pointer propogation - but those are optimization bugs we're fixing concurrently.

The alternative to this would be to extend the base pointer inference with the ability to generally reuse multiple-base input instructions (phis and selects). That's somewhat invasive and complicated, so we're defering it a bit longer.

Differential Revision: https://reviews.llvm.org/D97885

show more ...

[llvm] Construct SmallVector with iterator ranges (NFC)

[NFC][InstructionCost] Use InstructionCost in Transforms/Scalar/RewriteStatepointsForGC.cpp

In places where we calculate costs using TTI.getXXXCost() interfaces
I have changed the code to use Instru

[NFC][InstructionCost] Use InstructionCost in Transforms/Scalar/RewriteStatepointsForGC.cpp

In places where we calculate costs using TTI.getXXXCost() interfaces
I have changed the code to use InstructionCost instead of unsigned.
The change is non functional since InstructionCost behaves in the
same way as an integer for valid costs. Currently the getXXXCost()
functions used in this file do not return invalid costs.

See this patch for the introduction of the type: https://reviews.llvm.org/D91174
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2020-November/146408.html

Differential revision: https://reviews.llvm.org/D94484

show more ...

Revert "[llvm] Use BasicBlock::phis() (NFC)"

Reverting because this causes crashes on the 2-stage buildbots, for
example http://lab.llvm.org:8011/#/builders/7/builds/1140.

This reverts commit 9b228

Revert "[llvm] Use BasicBlock::phis() (NFC)"

Reverting because this causes crashes on the 2-stage buildbots, for
example http://lab.llvm.org:8011/#/builders/7/builds/1140.

This reverts commit 9b228f107d43341ef73af92865f73a9a076c5a76.

show more ...

[llvm] Use BasicBlock::phis() (NFC)

[RS4GC] Lazily set changed flag when folding single entry phis

The function FoldSingleEntryPHINodes() is changed to return if
it has changed IR or not. This return value is used by RS4GC to
set the

[RS4GC] Lazily set changed flag when folding single entry phis

The function FoldSingleEntryPHINodes() is changed to return if
it has changed IR or not. This return value is used by RS4GC to
set the MadeChange flag respectively.

Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D93810

show more ...

[Transforms] Use llvm::append_range (NFC)

[Transforms] Use llvm::erase_if (NFC)

GC-parseable element atomic memcpy/memmove

This change introduces a GC parseable lowering for element atomic
memcpy/memmove intrinsics. This way runtime can provide an
implementation which can take

GC-parseable element atomic memcpy/memmove

This change introduces a GC parseable lowering for element atomic
memcpy/memmove intrinsics. This way runtime can provide an
implementation which can take a safepoint during copy operation.

See "GC-parseable element atomic memcpy/memmove" thread on llvm-dev
for the background and details:
https://groups.google.com/g/llvm-dev/c/NnENHzmX-b8/m/3PyN8Y2pCAAJ

Differential Revision: https://reviews.llvm.org/D88861

show more ...

[RS4GC] NFC. Preparatory refactoring to make GC parseable memcpy

For GC parseable element atomic memcpy/memmove we'll need to
shuffle statepoint arguments. Make it possible by storing the
arguments

[RS4GC] NFC. Preparatory refactoring to make GC parseable memcpy

For GC parseable element atomic memcpy/memmove we'll need to
shuffle statepoint arguments. Make it possible by storing the
arguments as Value *, not Use *.

show more ...

[Analysis] TTI: Add CastContextHint for getCastInstrCost

Currently, getCastInstrCost has limited information about the cast it's
rating, often just the opcode and types. Sometimes there is a contex

[Analysis] TTI: Add CastContextHint for getCastInstrCost

Currently, getCastInstrCost has limited information about the cast it's
rating, often just the opcode and types. Sometimes there is a context
instruction as well, but it isn't trustworthy: for instance, when the
vectorizer is rating a plan, it calls getCastInstrCost with the old
instructions when, in fact, it's trying to evaluate the cost of the
instruction post-vectorization. Thus, the current system can get the
cost of certain casts incorrect as the correct cost can vary greatly
based on the context in which it's used.

For example, if the vectorizer queries getCastInstrCost to evaluate the
cost of a sext(load) with tail predication enabled, getCastInstrCost
will think it's free most of the time, but it's not always free. On ARM
MVE, a VLD2 group cannot be extended like a normal VLDR can. Similar
situations can come up with how masked loads can be extended when being
split.

To fix that, this path adds a new parameter to getCastInstrCost to give
it a hint about the context of the cast. It adds a CastContextHint enum
which contains the type of the load/store being created by the
vectorizer - one for each of the types it can produce.

Original patch by Pierre van Houtryve

Differential Revision: https://reviews.llvm.org/D79162

show more ...

[SVE] Remove calls to VectorType::getNumElements from Scalar

Reviewers: efriedma, fhahn, reames, kmclaughlin, sdesmalen

Reviewed By: sdesmalen

Subscribers: tschuett, hiraditya, rkruppe, psnobl, da

[SVE] Remove calls to VectorType::getNumElements from Scalar

Reviewers: efriedma, fhahn, reames, kmclaughlin, sdesmalen

Reviewed By: sdesmalen

Subscribers: tschuett, hiraditya, rkruppe, psnobl, dantrushin, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D82243

show more ...

[SVE] Eliminate calls to default-false VectorType::get() from Scalar

Reviewers: efriedma, kmclaughlin, sdesmalen, fhahn, bkramer, anna, gchatelet, c-rhodes, david-arm, fpetrogalli

Reviewed By: davi

[SVE] Eliminate calls to default-false VectorType::get() from Scalar

Reviewers: efriedma, kmclaughlin, sdesmalen, fhahn, bkramer, anna, gchatelet, c-rhodes, david-arm, fpetrogalli

Reviewed By: david-arm

Subscribers: tschuett, hiraditya, rkruppe, psnobl, dantrushin, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D80336

show more ...

[Statepoint] Switch RS4GC to using gc-live bundle form

Now that we have an operand based form for the GC arguments to a statepoint intrinsic, update RS4GC to use it and update tests to reflect. This

[Statepoint] Switch RS4GC to using gc-live bundle form

Now that we have an operand based form for the GC arguments to a statepoint intrinsic, update RS4GC to use it and update tests to reflect. This is pretty straight forward. I nearly landed without review, but figured a second set of eyes didn't hurt.

Differential Revision: https://reviews.llvm.org/D81121

show more ...