[VPlan] Use cast<VPRecipeBase> in verifier (NFC).

All users of VPValue must be a VPRecipeBase, use cast.

[VPlan] Verify scalar types in VPlanVerifier. NFCI (#122679)

VTypeAnalysis contains some assertions which can be useful for reasoning
that the types of various operands match.

This patch teaches VP

[VPlan] Verify scalar types in VPlanVerifier. NFCI (#122679)

VTypeAnalysis contains some assertions which can be useful for reasoning
that the types of various operands match.

This patch teaches VPlanVerifier to invoke VTypeAnalysis to check them,
and catches some issues with VPInstruction types that are also fixed
here:

* Handles the missing cases for CalculateTripCountMinusVF,
CanonicalIVIncrementForPart and AnyOf
* Fixes ICmp and ActiveLaneMask to return i1 (to align with `icmp` and
`@llvm.get.active.lane.mask` in the LangRef)

The VPlanVerifier unit tests also need to be fleshed out a bit more to
satisfy the stricter assertions

show more ...

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy ske

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy skeleton creation in
VPIRBasicBlocks and hook them into the VPlan. This means the skeleton
CFG is represented in VPlan, just before execute. This allows moving
parts of skeleton creation into recipes in the VPBBs gradually.

Note that this allows retiring some manual DT updates, as this will be
handled automatically during VPlan execution.

PR: https://github.com/llvm/llvm-project/pull/114292

show more ...

[VPlan] Simplify and unify code in verifyEVLRecipe using all_of. (NFCI)

Use all_of instead of explicit loop to reduce indentation, also properly
check VPScalarCastRecipe operand.

[VPlan] Support VPReverseVectorPointer in DataWithEVL vectorization (#113667)

VPReverseVectorPointer relies on the runtime VF, but in DataWithEVL
tail-folding, EVL (which can be less than VF at run

[VPlan] Support VPReverseVectorPointer in DataWithEVL vectorization (#113667)

VPReverseVectorPointer relies on the runtime VF, but in DataWithEVL
tail-folding, EVL (which can be less than VF at runtime) should be used
instead.

This patch updates the logic to check the users of VF and replaces the
second operand if the user is VPReverseVectorPointer.

show more ...

[Vectorize] Remove unused includes (NFC) (#114643)

Identified with misc-include-cleaner.

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handl

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handled by adding
operands to the wrapped phis in the scalar loop header.

Note that the current version only includes the scalar loop header, no
other loop blocks and also does not wrap it in a region block.

PR: https://github.com/llvm/llvm-project/pull/109975

show more ...

[VPlan] Use VPWidenIntrinsicRecipe to vp.select. (#110489)

Use VPWidenIntrinsicRecipe
(https://github.com/llvm/llvm-project/pull/110486)
to create vp.select intrinsics. This potentially offers an

[VPlan] Use VPWidenIntrinsicRecipe to vp.select. (#110489)

Use VPWidenIntrinsicRecipe
(https://github.com/llvm/llvm-project/pull/110486)
to create vp.select intrinsics. This potentially offers an alternative
to duplicating EVL recipes for all existing recipes.

There are some recipes that will need duplicates (at least at the
moment), due to extra code-gen needs (e.g. widening loads and stores).
But in cases the intrinsic can directly be used, creating the widened
intrinsic directly would reduce the need to duplicate some recipes.


PR: https://github.com/llvm/llvm-project/pull/110489

show more ...

[LV] Added verification of EVL recipes (#107630)

[VPlan] Add VPIRInstruction, use for exit block live-outs. (#100735)

Add a new VPIRInstruction recipe to wrap existing IR instructions not to
be modified during execution, execept for PHIs. For PHI

[VPlan] Add VPIRInstruction, use for exit block live-outs. (#100735)

Add a new VPIRInstruction recipe to wrap existing IR instructions not to
be modified during execution, execept for PHIs. For PHIs, a single
VPValue
operand is allowed, and it is used to add a new incoming value for the
single predecessor VPBB. Expect PHIs, VPIRInstructions cannot have any
operands.

Depends on https://github.com/llvm/llvm-project/pull/100658.

PR: https://github.com/llvm/llvm-project/pull/100735

show more ...

[Vectorize] Avoid repeated hash lookups (NFC) (#107729)

[VPlan] Model branch cond to enter scalar epilogue in VPlan. (#92651)

This patch moves branch condition creation to enter the scalar epilogue
loop to VPlan. Modeling the branch in the middle block

[VPlan] Model branch cond to enter scalar epilogue in VPlan. (#92651)

This patch moves branch condition creation to enter the scalar epilogue
loop to VPlan. Modeling the branch in the middle block also requires
modeling the successor blocks. This is done using the recently
introduced VPIRBasicBlock.

Note that the middle.block is still created as part of the skeleton and
then patched in during VPlan execution. Unfortunately the skeleton needs
to create the middle.block early on, as it is also used for induction
resume value creation and is also needed to properly update the
dominator tree during skeleton creation.

After this patch lands, I plan to move induction resume value and phi
node creation in the scalar preheader to VPlan. Once that is done, we
should be able to create the middle.block in VPlan directly.

This is a re-worked version based on the earlier
https://reviews.llvm.org/D150398 and the main change is the use of
VPIRBasicBlock.

Depends on https://github.com/llvm/llvm-project/pull/92525

PR: https://github.com/llvm/llvm-project/pull/92651

show more ...

[VPlan] Add VPIRBasicBlock, use to model pre-preheader. (#93398)

This patch adds a new special type of VPBasicBlock that wraps an
existing IR basic block. Recipes of the block get added before the

[VPlan] Add VPIRBasicBlock, use to model pre-preheader. (#93398)

This patch adds a new special type of VPBasicBlock that wraps an
existing IR basic block. Recipes of the block get added before the
terminator of the wrapped IR basic block. Making it a subclass of
VPBasicBlock avoids duplicating various APIs to manage recipes in a
block, as well as makes sure the traversals filtering VPBasicBlocks
automatically apply as well.

Initially VPIRBasicBlock are only used for the pre-preheader (wrapping
the original preheader of the scalar loop).

As follow-up, this will be used to move more parts of the skeleton
inside VPlan, starting with the branch and condition in the middle
block.

Separated out of https://github.com/llvm/llvm-project/pull/92651

PR: https://github.com/llvm/llvm-project/pull/93398

show more ...

[VPlan] Move verifier to class to reduce need to pass via args. (NFC)

Move VPlan verification functions to avoid the need to pass VPDT across
multiple calls. This also allows easier extensions in th

[VPlan] Move verifier to class to reduce need to pass via args. (NFC)

Move VPlan verification functions to avoid the need to pass VPDT across
multiple calls. This also allows easier extensions in the future.

show more ...

[VPlan] Remove custom checks for EVL placement in verifier (NFCI).

After e2a72fa583d9, def-use chains of EVL are modeled explicitly.
So there's no need for a custom check of its placement, as regula

[VPlan] Remove custom checks for EVL placement in verifier (NFCI).

After e2a72fa583d9, def-use chains of EVL are modeled explicitly.
So there's no need for a custom check of its placement, as regular
def-use verification will catch mis-placements.

show more ...

[VPlan] Split VPWidenMemoryInstructionRecipe (NFCI). (#87411)

This patch introduces a new VPWidenMemoryRecipe base class and distinct
sub-classes to model loads and stores.

This is a first step

[VPlan] Split VPWidenMemoryInstructionRecipe (NFCI). (#87411)

This patch introduces a new VPWidenMemoryRecipe base class and distinct
sub-classes to model loads and stores.

This is a first step in an effort to simplify and modularize code
generation for widened loads and stores and enable adding further more
specialized memory recipes.

PR: https://github.com/llvm/llvm-project/pull/87411

show more ...

[LV, VP]VP intrinsics support for the Loop Vectorizer + adding new tail-folding mode using EVL. (#76172)

This patch introduces generating VP intrinsics in the Loop Vectorizer.

Currently the Loop

[LV, VP]VP intrinsics support for the Loop Vectorizer + adding new tail-folding mode using EVL. (#76172)

This patch introduces generating VP intrinsics in the Loop Vectorizer.

Currently the Loop Vectorizer supports vector predication in a very
limited capacity via tail-folding and masked load/store/gather/scatter
intrinsics. However, this does not let architectures with active vector
length predication support take advantage of their capabilities.
Architectures with general masked predication support also can only take
advantage of predication on memory operations. By having a way for the
Loop Vectorizer to generate Vector Predication intrinsics, which (will)
provide a target-independent way to model predicated vector
instructions. These architectures can make better use of their
predication capabilities.

Our first approach (implemented in this patch) builds on top of the
existing tail-folding mechanism in the LV (just adds a new tail-folding
mode using EVL), but instead of generating masked intrinsics for memory
operations it generates VP intrinsics for loads/stores instructions. The
patch adds a new VPlanTransforms to replace the wide header predicate
compare with EVL and updates codegen for load/stores to use VP
store/load with EVL.

Other important part of this approach is how the Explicit Vector Length
is computed. (VP intrinsics define this vector length parameter as
Explicit Vector Length (EVL)). We use an experimental intrinsic
`get_vector_length`, that can be lowered to architecture specific
instruction(s) to compute EVL.

Also, added a new recipe to emit instructions for computing EVL. Using
VPlan in this way will eventually help build and compare VPlans
corresponding to different strategies and alternatives.

Differential Revision: https://reviews.llvm.org/D99750

show more ...

[VPlan] Verify CFG invariants first (NFCI).

Verifying CFG invariants of a block before verifying its contents allows
contents verification to rely on the CFG invariants (e.g. that there's a
vector l

[VPlan] Verify CFG invariants first (NFCI).

Verifying CFG invariants of a block before verifying its contents allows
contents verification to rely on the CFG invariants (e.g. that there's a
vector loop region that can be retrieved).

This avoids extra checks in
https://github.com/llvm/llvm-project/pull/76172.

show more ...

[VPlan] Merge main VPlan verifer with HCFG verifier.

Unify VPlan verifiers in verifyVPlanIsValid. This adds verification for
various properties on blocks to the verifier used for VPlans generated
by

[VPlan] Merge main VPlan verifer with HCFG verifier.

Unify VPlan verifiers in verifyVPlanIsValid. This adds verification for
various properties on blocks to the verifier used for VPlans generated
by the inner loop vectorizer. It also adds def-use checks for the
verifier used in the VPlan native path.

This drops the separate flag to enable HCFG verification. Instead, all
VPlans are verified once they have been created, if assertions are
enabled.

This also removes VPWidenPHIRecipe from VPHeaderPHIRecipe; it is used to
model any phi node in the native path.

show more ...

[VPlan] Use VPDominatorTree in VPlanVerifier .

Use VPDominatorTree to generalize def-use verification.

Depends on D140513.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D140514

[VPlan] Add vp_depth_first_deep (NFC)

Similar to vp_depth_first_shallow (D140512) add vp_depth_first_deep to
make existing code clearer and more compact.

Reviewed By: Ayal

Differential Revision: h

[VPlan] Add vp_depth_first_deep (NFC)

Similar to vp_depth_first_shallow (D140512) add vp_depth_first_deep to
make existing code clearer and more compact.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D142055

show more ...

[VPlan] Add vp_depth_first_shallow + graph traits for wrapper(NFC)

This patch adds a new VPBlockShallowTraversalWrapper struct to
provide graph traits specialization that do not traverse through
VPR

[VPlan] Add vp_depth_first_shallow + graph traits for wrapper(NFC)

This patch adds a new VPBlockShallowTraversalWrapper struct to
provide graph traits specialization that do not traverse through
VPRegionBlocks. This matches the behavior of the existing traits for
plain VPBlockBase and is a step before moving the graph traits for
VPBlockBase to traverse through VPRegionBlocks to enable cross region
support in VPDominatorTree.

Depends on D140511.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D140512

show more ...

[VPlan] Move GraphTraits definitions to separate header (NFC).

This reduces the size of VPlan.h and avoids future growth of the file
when the graph traits are extended in future patches.

Reviewed B

[VPlan] Move GraphTraits definitions to separate header (NFC).

This reduces the size of VPlan.h and avoids future growth of the file
when the graph traits are extended in future patches.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D140500

show more ...

[VPlan] Verify that header only contains header phi recipes.

Add verification that VPHeaderPHIRecipes are only in header VPBBs. Also
adds missing checks for VPPointerInductionRecipe to
VPHeaderPHIRe

[VPlan] Verify that header only contains header phi recipes.

Add verification that VPHeaderPHIRecipes are only in header VPBBs. Also
adds missing checks for VPPointerInductionRecipe to
VPHeaderPHIRecipe::classof.

Split off from D119661.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D131989

show more ...

[Transforms] Qualify auto in range-based for loops (NFC)

Identified with readability-qualified-auto.