[VPlan] Add plain text (not DOT's digraph) dumps

I foresee two uses for this:
1) It's easier to use those in debugger.
2) Once we start implementing more VPlan-to-VPlan transformations (especially

[VPlan] Add plain text (not DOT's digraph) dumps

I foresee two uses for this:
1) It's easier to use those in debugger.
2) Once we start implementing more VPlan-to-VPlan transformations (especially
inner loop massaging stuff), using the vectorized LLVM IR as CHECK targets in
LIT test would become too obscure. I can imagine that we'd want to CHECK
against VPlan dumps after multiple transformations instead. That would be
easier with plain text dumps than with DOT format.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D96628

show more ...

[SVE][LoopVectorize] Add support for extracting the last lane of a scalable vector

There are certain loops like this below:

for (int i = 0; i < n; i++) {
a[i] = b[i] + 1;
*inv = a[i];
}

[SVE][LoopVectorize] Add support for extracting the last lane of a scalable vector

There are certain loops like this below:

for (int i = 0; i < n; i++) {
a[i] = b[i] + 1;
*inv = a[i];
}

that can only be vectorised if we are able to extract the last lane of the
vectorised form of 'a[i]'. For fixed width vectors this already works since
we know at compile time what the final lane is, however for scalable vectors
this is a different story. This patch adds support for extracting the last
lane from a scalable vector using a runtime determined lane value. I have
added support to VPIteration for runtime-determined lanes that still permit
the caching of values. I did this by introducing a new class called VPLane,
which describes the lane we're dealing with and provides interfaces to get
both the compile-time known lane and the runtime determined value. Whilst
doing this work I couldn't find any explicit tests for extracting the last
lane values of fixed width vectors so I added tests for both scalable and
fixed width vectors.

Differential Revision: https://reviews.llvm.org/D95139

show more ...

[NFCI][VPlan] Modify Recipes' print methods to honor Indent parameter

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D97787

[NFC][VPlan] Use VPUser to store block's predicate

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D96529

[VPlan] Skip VPWidenPHIRecipe in VPInterleavedACcessInfo.

Update unit tests that did not expect VPWidenPHIRecipes after
15a74b64dfa9.

[LV] Fold single-use variable into assert. NFC.

[LV] Remove VPCallback.

Now that all state for generated instructions is managed directly in
VPTransformState, VPCallBack is no longer needed. This patch updates the
last use of `getOrCreateScalarVa

[LV] Remove VPCallback.

Now that all state for generated instructions is managed directly in
VPTransformState, VPCallBack is no longer needed. This patch updates the
last use of `getOrCreateScalarValue` to instead manage the value
directly in VPTransformState and removes VPCallback.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D95383

show more ...

[VPlan] Manage scalarized values using VPValues.

This patch updates codegen to use VPValues to manage the generated
scalarized instructions.

Reviewed By: gilr

Differential Revision: https://review

[VPlan] Manage scalarized values using VPValues.

This patch updates codegen to use VPValues to manage the generated
scalarized instructions.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D92285

show more ...

[VPlan] Make VPRecipeBase inherit from VPUser directly (NFC).

The individual recipes have been updated to manage their operands using
VPUser a while back. Now that the transition is done, we can ins

[VPlan] Make VPRecipeBase inherit from VPUser directly (NFC).

The individual recipes have been updated to manage their operands using
VPUser a while back. Now that the transition is done, we can instead
make VPRecipeBase a VPUser and get rid of the toVPUser helper.

show more ...

[VPlan] Use VPUser to manage CondBit

VP blocks keep track of a condition, which is a VPValue. This patch
updates VPBlockBase to manage the value using VPUser, so
replaceAllUsesWith properly updates

[VPlan] Use VPUser to manage CondBit

VP blocks keep track of a condition, which is a VPValue. This patch
updates VPBlockBase to manage the value using VPUser, so
replaceAllUsesWith properly updates the condition bit as well.

This is required to enable VP2VP transformations and it helps with
simplifying some of the code required to manage condition bits.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D95382

show more ...

[VPlan] Manage induction value creation using VPValues.

This patch updates the induction value creation to use VPValues of
recipes to map the created values. This should bring is one step closer
to

[VPlan] Manage induction value creation using VPValues.

This patch updates the induction value creation to use VPValues of
recipes to map the created values. This should bring is one step closer
to being able to optimize induction recipes directly in VPlan.

Currently widenIntOrFpInduction also generates vector values for a cast
of the induction, if it exists. Make this explicit by adding the cast
instruction to the values defined by the recipe.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D92284

show more ...

[VPlan][NFC] Introduce constructors for VPIteration

This patch adds constructors to VPIteration as a cleaner way of
initialising the struct and replaces existing constructions of
the form:

{Part,

[VPlan][NFC] Introduce constructors for VPIteration

This patch adds constructors to VPIteration as a cleaner way of
initialising the struct and replaces existing constructions of
the form:

{Part, Lane}

with

VPIteration(Part, Lane)

I have also added a default constructor, which is used by VPlan.cpp
when deciding whether to replicate a block or not.

This refactoring will be required in a later patch that adds more
members and functions to VPIteration.

Differential Revision: https://reviews.llvm.org/D95676

show more ...

[VPlan] Handle scalarized values in VPTransformState.

This patch adds plumbing to handle scalarized values directly in
VPTransformState.

Reviewed By: gilr

Differential Revision: https://reviews.ll

[VPlan] Handle scalarized values in VPTransformState.

This patch adds plumbing to handle scalarized values directly in
VPTransformState.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D92282

show more ...

[llvm] Construct SmallVector with iterator ranges (NFC)

[VPlan] Unify value/recipe printing after VPDef transition.

This patch unifies the way recipes and VPValues are printed after the
transition to VPDef.

VPSlotTracker has been updated to iterate over

[VPlan] Unify value/recipe printing after VPDef transition.

This patch unifies the way recipes and VPValues are printed after the
transition to VPDef.

VPSlotTracker has been updated to iterate over all recipes and all
their defined values to number those. There is no need to number
values in Value2VPValue.

It also updates a few places that only used slot numbers for
VPInstruction. All recipes now can produce numbered VPValues.

show more ...

[VPlan] Move initial quote emission from ::print to ::dumpBasicBlock.

This means there will be no stray " when printing individual recipes
using print()/dump() in a debugger, for example.

[LV] Don't sink into replication regions

The new test case here contains a first order recurrences and an
instruction that is replicated. The first order recurrence forces an
instruction to be sunk

[LV] Don't sink into replication regions

The new test case here contains a first order recurrences and an
instruction that is replicated. The first order recurrence forces an
instruction to be sunk _into_, as opposed to after the replication
region. That causes several things to go wrong including registering
vector instructions multiple times and failing to create dominance
relations correctly.

Instead we should be sinking to after the replication region, which is
what this patch makes sure happens.

Differential Revision: https://reviews.llvm.org/D93629

show more ...

[LoopUtils] remove redundant opcode parameter; NFC

While here, rename the inaccurate getRecurrenceBinOp()
because that was also used to get CmpInst opcodes.

The recurrence/reduction kind should alw

[LoopUtils] remove redundant opcode parameter; NFC

While here, rename the inaccurate getRecurrenceBinOp()
because that was also used to get CmpInst opcodes.

The recurrence/reduction kind should always refer to the
expected opcode for a reduction. SLP appears to be the
only direct caller of createSimpleTargetReduction(), and
that calling code ideally should not be carrying around
both an opcode and a reduction kind.

This should allow us to generalize reduction matching to
use intrinsics instead of only binops.

show more ...

[VPlan] Make VPInstruction a VPDef

This patch turns updates VPInstruction to manage the value it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration instead of the

[VPlan] Make VPInstruction a VPDef

This patch turns updates VPInstruction to manage the value it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration instead of the plain IR reference where possible.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90565

show more ...

[VPlan] Make VPRecipeBase inherit from VPDef.

This patch makes VPRecipeBase a direct subclass of VPDef, moving the
SubclassID to VPDef.

Reviewed By: gilr

Differential Revision: https://reviews.llv

[VPlan] Make VPRecipeBase inherit from VPDef.

This patch makes VPRecipeBase a direct subclass of VPDef, moving the
SubclassID to VPDef.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90564

show more ...

[VPlan] Use VPDef for VPInterleaveRecipe.

This patch turns updates VPInterleaveRecipe to manage the values it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration i

[VPlan] Use VPDef for VPInterleaveRecipe.

This patch turns updates VPInterleaveRecipe to manage the values it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration instead of the plain IR reference where possible.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90562

show more ...

[VPlan] Make VPWidenMemoryInstructionRecipe a VPDef.

This patch updates VPWidenMemoryInstructionRecipe to use VPDef
to manage the value it produces instead of inheriting from VPValue.

Reviewed By:

[VPlan] Make VPWidenMemoryInstructionRecipe a VPDef.

This patch updates VPWidenMemoryInstructionRecipe to use VPDef
to manage the value it produces instead of inheriting from VPValue.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90563

show more ...

[VPlan] Use interleaveComma in printOperands() (NFC).

[VPlan] Use VPUser to manage VPPredInstPHIRecipe operand (NFC).

VPPredInstPHIRecipe is one of the recipes that was missed during the
initial conversion. This patch adjusts the recipe to also manage

[VPlan] Use VPUser to manage VPPredInstPHIRecipe operand (NFC).

VPPredInstPHIRecipe is one of the recipes that was missed during the
initial conversion. This patch adjusts the recipe to also manage its
operand using VPUser.

show more ...

[VPlan] Use VPValue and VPUser ops to print VPReplicateRecipe.