[VPlan] Use recursive traversal iterator in VPSlotTracker.

This patch simplifies VPSlotTracker by using the recursive traversal
iterator to traverse all blocks in a VPlan in reverse post-order when

[VPlan] Use recursive traversal iterator in VPSlotTracker.

This patch simplifies VPSlotTracker by using the recursive traversal
iterator to traverse all blocks in a VPlan in reverse post-order when
numbering VPValues in a plan.

This depends on a fix to RPOT (D100169). It also extends the traversal
unit tests to check RPOT.

Reviewed By: a.elovikov

Differential Revision: https://reviews.llvm.org/D100176

show more ...

[NFC][VPlan] Guard print routines with "#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)"

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D98897

[VPlan] Manage scalarized values using VPValues.

This patch updates codegen to use VPValues to manage the generated
scalarized instructions.

Reviewed By: gilr

Differential Revision: https://review

[VPlan] Manage scalarized values using VPValues.

This patch updates codegen to use VPValues to manage the generated
scalarized instructions.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D92285

show more ...

[VPlan] Use VPUser to manage CondBit

VP blocks keep track of a condition, which is a VPValue. This patch
updates VPBlockBase to manage the value using VPUser, so
replaceAllUsesWith properly updates

[VPlan] Use VPUser to manage CondBit

VP blocks keep track of a condition, which is a VPValue. This patch
updates VPBlockBase to manage the value using VPUser, so
replaceAllUsesWith properly updates the condition bit as well.

This is required to enable VP2VP transformations and it helps with
simplifying some of the code required to manage condition bits.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D95382

show more ...

[VPlan] Unify value/recipe printing after VPDef transition.

This patch unifies the way recipes and VPValues are printed after the
transition to VPDef.

VPSlotTracker has been updated to iterate over

[VPlan] Unify value/recipe printing after VPDef transition.

This patch unifies the way recipes and VPValues are printed after the
transition to VPDef.

VPSlotTracker has been updated to iterate over all recipes and all
their defined values to number those. There is no need to number
values in Value2VPValue.

It also updates a few places that only used slot numbers for
VPInstruction. All recipes now can produce numbered VPValues.

show more ...

[VPlan] Add getLiveInIRValue accessor to VPValue.

This patch adds a new getLiveInIRValue accessor to VPValue, which
returns the underlying value, if the VPValue is defined outside of
VPlan. This is

[VPlan] Add getLiveInIRValue accessor to VPValue.

This patch adds a new getLiveInIRValue accessor to VPValue, which
returns the underlying value, if the VPValue is defined outside of
VPlan. This is required to handle scalars in VPTransformState, which
requires dealing with scalars defined outside of VPlan.

We can simply check VPValue::Def to determine if the value is defined
inside a VPlan.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D92281

show more ...

[VPlan] Use public VPValue constructor in VPPRedInstPHIRecipe (NFC).

VPPredInstPHIRecipe does not need access to VPValue via friendship. It
can just use the public constructor,

Discussed as part of

[VPlan] Use public VPValue constructor in VPPRedInstPHIRecipe (NFC).

VPPredInstPHIRecipe does not need access to VPValue via friendship. It
can just use the public constructor,

Discussed as part of D92281.

show more ...

[VPlan] Make VPInstruction a VPDef

This patch turns updates VPInstruction to manage the value it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration instead of the

[VPlan] Make VPInstruction a VPDef

This patch turns updates VPInstruction to manage the value it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration instead of the plain IR reference where possible.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90565

show more ...

[VPlan] Make VPRecipeBase inherit from VPDef.

This patch makes VPRecipeBase a direct subclass of VPDef, moving the
SubclassID to VPDef.

Reviewed By: gilr

Differential Revision: https://reviews.llv

[VPlan] Make VPRecipeBase inherit from VPDef.

This patch makes VPRecipeBase a direct subclass of VPDef, moving the
SubclassID to VPDef.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90564

show more ...

[VPlan] Make VPWidenMemoryInstructionRecipe a VPDef.

This patch updates VPWidenMemoryInstructionRecipe to use VPDef
to manage the value it produces instead of inheriting from VPValue.

Reviewed By:

[VPlan] Make VPWidenMemoryInstructionRecipe a VPDef.

This patch updates VPWidenMemoryInstructionRecipe to use VPDef
to manage the value it produces instead of inheriting from VPValue.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90563

show more ...

[Transforms] Use is_contained (NFC)

[VPlan] Use VPUser to manage VPPredInstPHIRecipe operand (NFC).

VPPredInstPHIRecipe is one of the recipes that was missed during the
initial conversion. This patch adjusts the recipe to also manage

[VPlan] Use VPUser to manage VPPredInstPHIRecipe operand (NFC).

VPPredInstPHIRecipe is one of the recipes that was missed during the
initial conversion. This patch adjusts the recipe to also manage its
operand using VPUser.

show more ...

[VPlan] Turn VPReplicateRecipe into a VPValue.

Update VPReplicateRecipe to inherit from VPValue. This still does not
update scalarizeInstruction to set the result for the VPValue of
VPReplicateRecip

[VPlan] Turn VPReplicateRecipe into a VPValue.

Update VPReplicateRecipe to inherit from VPValue. This still does not
update scalarizeInstruction to set the result for the VPValue of
VPReplicateRecipe, because this first requires tracking scalar values in
VPTransformState.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D91500

show more ...

[VPlan] Add VPReductionSC to VPUser::classof, unify VPValue IDs.

This is a follow-up to 00a66011366c7b037d6680e6015524a41b761c34 to make
isa<VPReductionRecipe> work and unifies the VPValue ID names,

[VPlan] Add VPReductionSC to VPUser::classof, unify VPValue IDs.

This is a follow-up to 00a66011366c7b037d6680e6015524a41b761c34 to make
isa<VPReductionRecipe> work and unifies the VPValue ID names, by making
sure they all consistently start with VPV*.

show more ...

[VPlan] Switch VPWidenRecipe to be a VPValue

Similar to other patches, this makes VPWidenRecipe a VPValue. Because of
the way it interacts with the reduction code it also slightly alters the
way tha

[VPlan] Switch VPWidenRecipe to be a VPValue

Similar to other patches, this makes VPWidenRecipe a VPValue. Because of
the way it interacts with the reduction code it also slightly alters the
way that VPValues are registered, removing the up front NeedDef and
using getOrAddVPValue to create them on-demand if needed instead.

Differential Revision: https://reviews.llvm.org/D88447

show more ...

[VPlan] Turn VPReductionRecipe into a VPValue

This converts the VPReductionRecipe into a VPValue, like other
VPRecipe's in preparation for traversing def-use chains. It also makes
it a VPUser, now s

[VPlan] Turn VPReductionRecipe into a VPValue

This converts the VPReductionRecipe into a VPValue, like other
VPRecipe's in preparation for traversing def-use chains. It also makes
it a VPUser, now storing the used VPValues as operands.

It doesn't yet change how the VPReductionRecipes are created. It will
need to call replaceAllUsesWith from the original recipe they replace,
but that is not done yet as VPWidenRecipe need to be created first.

Differential Revision: https://reviews.llvm.org/D88382

show more ...

[VPlan] Add VPDef class.

This patch introduces a new VPDef class, which can be used to
manage VPValues defined by recipes/VPInstructions.

The idea here is to mirror VPUser for values defined by a r

[VPlan] Add VPDef class.

This patch introduces a new VPDef class, which can be used to
manage VPValues defined by recipes/VPInstructions.

The idea here is to mirror VPUser for values defined by a recipe. A
VPDef can produce either zero (e.g. a store recipe), one (most recipes)
or multiple (VPInterleaveRecipe) result VPValues.

To traverse the def-use chain from a VPDef to its users, one has to
traverse the users of all values defined by a VPDef.

VPValues now contain a pointer to their corresponding VPDef, if one
exists. To traverse the def-use chain upwards from a VPValue, we first
need to check if the VPValue is defined by a VPDef. If it does not have
a VPDef, this means we have a VPValue that is not directly defined
iniside the plan and we are done.

If we have a VPDef, it is defined inside the region by a recipe, which
is a VPUser, and the upwards def-use chain traversal continues by
traversing all its operands.

Note that we need to add an additional field to to VPVAlue to link them
to their defs. The space increase is going to be offset by being able to
remove the SubclassID field in future patches.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D90558

show more ...

[VPlan] Use VPValue def for VPWidenGEPRecipe.

This patch turns VPWidenGEPRecipe into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possib

[VPlan] Use VPValue def for VPWidenGEPRecipe.

This patch turns VPWidenGEPRecipe into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possible.

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D84683

show more ...

Recommit "[VPlan] Use VPValue def for VPWidenSelectRecipe."

This reverts the revert commit c8d73d939fa4fda9c87b3979225d02d63062bd68.

It includes a fix for cases where we missed inserting VPValues
f

Recommit "[VPlan] Use VPValue def for VPWidenSelectRecipe."

This reverts the revert commit c8d73d939fa4fda9c87b3979225d02d63062bd68.

It includes a fix for cases where we missed inserting VPValues
for some selects, which should fix PR48142.

show more ...

Revert "[VPlan] Use VPValue def for VPWidenSelectRecipe."

This reverts commit a8e50f1c6e7b404aab8fedb972f003a4d6a6434e.

This reportedly breaks building the Linux kernel.
https://bugs.llvm.org/sho

Revert "[VPlan] Use VPValue def for VPWidenSelectRecipe."

This reverts commit a8e50f1c6e7b404aab8fedb972f003a4d6a6434e.

This reportedly breaks building the Linux kernel.
https://bugs.llvm.org/show_bug.cgi?id=48142

show more ...

[VPlan] Use VPValue def for VPWidenSelectRecipe.

This patch turns VPWidenSelectRecipe into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where

[VPlan] Use VPValue def for VPWidenSelectRecipe.

This patch turns VPWidenSelectRecipe into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possible.

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D84682

show more ...

[VPlan] Use VPValue def for VPWidenCall.

This patch turns VPWidenCall into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possible.

Revie

[VPlan] Use VPValue def for VPWidenCall.

This patch turns VPWidenCall into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possible.

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D84681

show more ...

[VPlan] Add printOperands helper to VPUser (NFC).

Factor out the code for printing operands of a VPUser so it can be
re-used when printing other recipes.

[VPlan] Assert no users remaining when deleting a VPValue.

When deleting a VPValue, all users must already by deleted. Add an
assertion to make sure and catch violations.

Recommit "[VPlan] Use VPValue def for VPMemoryInstructionRecipe."

This reverts the revert commit 710aceb645e7dba4de7053eef2c616311b9163d4
and includes a fix for a memsan failure.

Original message:

Recommit "[VPlan] Use VPValue def for VPMemoryInstructionRecipe."

This reverts the revert commit 710aceb645e7dba4de7053eef2c616311b9163d4
and includes a fix for a memsan failure.

Original message:

This patch turns VPMemoryInstructionRecipe into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possible.

show more ...