[VPlan] Introduce VPScalarPHIRecipe, use for can & EVL IV codegen (NFC). (#114305)

Introduce a general recipe to generate a scalar phi. Lower
VPCanonicalIVPHIRecipe and VPEVLBasedIVRecipe to VPScal

[VPlan] Introduce VPScalarPHIRecipe, use for can & EVL IV codegen (NFC). (#114305)

Introduce a general recipe to generate a scalar phi. Lower
VPCanonicalIVPHIRecipe and VPEVLBasedIVRecipe to VPScalarIVPHIrecipe
before plan execution, avoiding the need for duplicated ::execute
implementations. There are other cases that could benefit, including
in-loop reduction phis and pointer induction phis.

Builds on a similar idea as
https://github.com/llvm/llvm-project/pull/82270.

PR: https://github.com/llvm/llvm-project/pull/114305

show more ...

[VPlan] Simplify code to re-use existing basic blocks (NFCI).

Restructure and slightly simplify code to re-use existing basic blocks.

[NFC][VectorUtils][TargetTransformInfo] Add `isVectorIntrinsicWithOverloadTypeAtArg` api (#114849)

This changes allows target intrinsics to specify and overwrite overloaded types.

- Updates `Repl

[NFC][VectorUtils][TargetTransformInfo] Add `isVectorIntrinsicWithOverloadTypeAtArg` api (#114849)

This changes allows target intrinsics to specify and overwrite overloaded types.

- Updates `ReplaceWithVecLib` to not provide TTI as there most probably won't be a use-case
- Updates `SLPVectorizer` to use available TTI
- Updates `VPTransformState` to pass down TTI
- Updates `VPlanRecipe` to use passed-down TTI

This change will let us add scalarization for `asdouble`: #114847

show more ...

[VPlan] Consistently use DEBUG_TYPE loop-vectorize.

This ensures debug messages in VPlan.cpp are included in the commonly
used -debug-only=loop-vectorize.

[VPlan] Remove unneeded code connecting blocks in VPBB:splitAt (NFC).

insertBlockAfter already takes care of transferring successors. Remove
unneeded code to transfer them manually.

[VPlan] Share logic to connect predecessors in VPBB/VPIRBB execute (NFC)

This moves the common logic to connect IRBBs created for a VPBB to their
predecessors in the VPlan CFG, making it easier to k

[VPlan] Share logic to connect predecessors in VPBB/VPIRBB execute (NFC)

This moves the common logic to connect IRBBs created for a VPBB to their
predecessors in the VPlan CFG, making it easier to keep in sync in the
future.

show more ...

[Vectorize] Remove unused includes (NFC) (#114643)

Identified with misc-include-cleaner.

[VPlan][NFC] Add new getMiddleBlock interface to VPlan (#113558)

This work is in preparation for PRs #112138 and #88385 where
the middle block is not guaranteed to be the immediate successor
to th

[VPlan][NFC] Add new getMiddleBlock interface to VPlan (#113558)

This work is in preparation for PRs #112138 and #88385 where
the middle block is not guaranteed to be the immediate successor
to the region block. I've simply add new getMiddleBlock()
interfaces to VPlan that for now just return

cast<VPBasicBlock>(VectorRegion->getSingleSuccessor())

Once PR #112138 lands we'll need to do more work to discover
the middle block.

show more ...

[VPlan] Fix long comment added in b021464d35ca (NFC).

Fix formatting of comment added in b021464d35ca.

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handl

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handled by adding
operands to the wrapped phis in the scalar loop header.

Note that the current version only includes the scalar loop header, no
other loop blocks and also does not wrap it in a region block.

PR: https://github.com/llvm/llvm-project/pull/109975

show more ...

[VPlan] Replace getIRBasicBlock with IRBB in VPIRBB::execute (NFC).

Suggested in https://github.com/llvm/llvm-project/pull/109975. This
makes the function consistent throughout.

[VPlan] Introduce and use getVectorPreheader (NFC).

Introduce a dedicated function to retrieve the vector preheader. This
ensures the correct block is used, even if the skeleton is exetended.

[VPlan] Implement VPWidenSelectRecipe::computeCost.

Implement VPlan-based cost computation for VPWidenSelectRecipe.

[VPlan] Use VPInstruction::Name when assigning names (NFCI).

This slightly improves the printing of VPInstructions. NFC except debug
output.

[NFC][LoopVectorize] Make replaceVPBBWithIRVPBB more efficient (#111514)

In replaceVPBBWithIRVPBB we spend time erasing and appending
predecessors and successors from a list, when all we really hav

[NFC][LoopVectorize] Make replaceVPBBWithIRVPBB more efficient (#111514)

In replaceVPBBWithIRVPBB we spend time erasing and appending
predecessors and successors from a list, when all we really have to do
is replace the old with the new. Not only is this more efficient, but it
also preserves the ordering of successors and predecessors. This is
something which may become important for vectorising early exit loops
(see PR #88385), since a VPIRInstruction is the wrapper for a live-out
phi with extra operands that map to the incoming block according to the
block's predecessor.

show more ...

[LoopVectorize] In LoopVectorize.cpp start using getSymbolicMaxBackedgeTakenCount (#108833)

LoopVectorizationLegality currently only treats a loop as legal to vectorise
if PredicatedScalarEvolution

[LoopVectorize] In LoopVectorize.cpp start using getSymbolicMaxBackedgeTakenCount (#108833)

LoopVectorizationLegality currently only treats a loop as legal to vectorise
if PredicatedScalarEvolution::getBackedgeTakenCount returns a valid
SCEV, or more precisely that the loop must have an exact backedge taken
count. Therefore, in LoopVectorize.cpp we can safely replace all calls to
getBackedgeTakenCount with calls to getSymbolicMaxBackedgeTakenCount,
since the result is the same.

This also helps prepare the loop vectoriser for PR #88385.

show more ...

[VPlan] Move createVPIRBasicBlock helper to VPIRBasicBlock (NFC).

Move the helper to VPIRBasicBlock to allow easier re-use outside
VPlan.cpp

[VPlan] Remove VPIteration, update to use directly VPLane instead (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
only the lane part of VPIteration is used.

Simplify th

[VPlan] Remove VPIteration, update to use directly VPLane instead (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
only the lane part of VPIteration is used.

Simplify the code by replacing remaining uses of VPIteration with VPLane directly.

show more ...

[LV] Remove more references of unrolled parts after 57f5d8f2fe.

Continue to clean up some now stale references of unroll parts and
related terminology as pointed out post-commit for 06c3a7d.

[VPlan] Only store single scalar array per VPValue in VPTransState (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
VPTransformState only stores a single scalar vector pe

[VPlan] Only store single scalar array per VPValue in VPTransState (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
VPTransformState only stores a single scalar vector per VPValue.

Simplify the code by replacing the nested SmallVector in PerPartScalars with
a single SmallVector and rename to VPV2Scalars for clarity.

show more ...

[VPlan] Only store single vector per VPValue in VPTransformState. (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
VPTransformState only stores a single vector value per

[VPlan] Only store single vector per VPValue in VPTransformState. (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
VPTransformState only stores a single vector value per VPValue.

Simplify the code by replacing the SmallVector in PerPartOutput with a
single Value * and rename to VPV2Vector for clarity.

Also remove the redundant Part argument from various accessors.

show more ...

[VPlan] Remove unneeded State.UF after 8ec406757cb92 (NFC).

State.UF is not needed any longer after 8ec406757cb92
(https://github.com/llvm/llvm-project/pull/95842). Clean it up,
simplifying ::execut

[VPlan] Remove unneeded State.UF after 8ec406757cb92 (NFC).

State.UF is not needed any longer after 8ec406757cb92
(https://github.com/llvm/llvm-project/pull/95842). Clean it up,
simplifying ::execute of existing recipes.

show more ...

[VPlan] Implement unrolling as VPlan-to-VPlan transform. (#95842)

This patch implements explicit unrolling by UF as VPlan transform. In
follow up patches this will allow simplifying VPTransform st

[VPlan] Implement unrolling as VPlan-to-VPlan transform. (#95842)

This patch implements explicit unrolling by UF as VPlan transform. In
follow up patches this will allow simplifying VPTransform state (no need
to store unrolled parts) as well as recipe execution (no need to
generate code for multiple parts in an each recipe). It also allows for
more general optimziations (e.g. avoid generating code for recipes that
are uniform-across parts).

It also unifies the logic dealing with unrolled parts in a single place,
rather than spreading it out across multiple places (e.g. VPlan post
processing for header-phi recipes previously.)

In the initial implementation, a number of recipes still take the
unrolled part as additional, optional argument, if their execution
depends on the unrolled part.

The computation for start/step values for scalable inductions changed
slightly. Previously the step would be computed as scalar and then
splatted, now vscale gets splatted and multiplied by the step in a
vector mul.

This has been split off https://github.com/llvm/llvm-project/pull/94339
which also includes changes to simplify VPTransfomState and recipes'
::execute.

The current version mostly leaves existing ::execute untouched and
instead sets VPTransfomState::UF to 1.

A follow-up patch will clean up all references to VPTransformState::UF.

Another follow-up patch will simplify VPTransformState to only store a
single vector value per VPValue.

PR: https://github.com/llvm/llvm-project/pull/95842

show more ...

[VPlan] Generalize VPValue::isDefinedOutsideLoopRegions.

Update isDefinedOutsideLoopRegions to check if a recipe is defined
outside any region. Split off already approved
https://github.com/llvm/llv

[VPlan] Generalize VPValue::isDefinedOutsideLoopRegions.

Update isDefinedOutsideLoopRegions to check if a recipe is defined
outside any region. Split off already approved
https://github.com/llvm/llvm-project/pull/95842 now that this can be
tested separately after landing VPlan-based LICM
https://github.com/llvm/llvm-project/issues/107501

show more ...

[VPlan] Rename isDefinedOutside[Vector]Regions -> [Loop] (NFC)

Clarify name of helper, split off from
https://github.com/llvm/llvm-project/pull/95842/files#r1765556732.