[VPlan] Add helper to run VPlan passes, verify after run (NFC). (#123640)

Add new runPass helpers to run a VPlan transformation. This makes it
easier to add additional checks/functionality for each

[VPlan] Add helper to run VPlan passes, verify after run (NFC). (#123640)

Add new runPass helpers to run a VPlan transformation. This makes it
easier to add additional checks/functionality for each transform run. In
this patch, an option is added to run the verifier after each VPlan
transform.

Follow-ups will use the same helper to also support printing VPlans
after each transform.

Note that the verifier at the moment requires there to be a canonical IV
and vector loop region, so the final lowering transforms aren't run via
runPass yet.

PR: https://github.com/llvm/llvm-project/pull/123640

show more ...

[VPlan] Don't collect live-ins in collectUsersInExitBlocks. (NFC) (#123819)

Live-ins don't need to be handled, other than adding to the exit phi
recipe. Do that early and assert that otherwise the

[VPlan] Don't collect live-ins in collectUsersInExitBlocks. (NFC) (#123819)

Live-ins don't need to be handled, other than adding to the exit phi
recipe. Do that early and assert that otherwise the exit value is
defined in the vector loop region.

This should enable simply skipping other exit values that do not need
further fixing, e.g. if handling the exit value from the early exit
directly in handleUncountableEarlyExit.

PR: https://github.com/llvm/llvm-project/pull/123819

show more ...

Reapply "[VPlan] Update final IV exit value via VPlan. (#112147)"

This reverts the revert commit 58326f1d5b5b379590af92dd129b2f3b3e96af46.

The build failure in sanitizer stage2 builds has been fixe

Reapply "[VPlan] Update final IV exit value via VPlan. (#112147)"

This reverts the revert commit 58326f1d5b5b379590af92dd129b2f3b3e96af46.

The build failure in sanitizer stage2 builds has been fixed with
0d39fe6f5bb3edf0bddec09a8c6417377390aeac.

Original commit message:
Model updating IV users directly in VPlan, replace fixupIVUsers.

Now simple extracts are created for all phis in the exit block during
initial VPlan construction. A later VPlan transform
(optimizeInductionExitUsers) replaces extracts of inductions with
their pre-computed values if possible.

This completes the transition towards modeling all live-outs directly in
VPlan.

There are a few follow-ups:
* emit extracts initially also for resume phis, and optimize them
tougher with IV exit users
* support for VPlans with multiple exits in optimizeInductionExitUsers.

Depends on https://github.com/llvm/llvm-project/pull/110004,
https://github.com/llvm/llvm-project/pull/109975 and
https://github.com/llvm/llvm-project/pull/112145.

show more ...

Revert "[VPlan] Update final IV exit value via VPlan. (#112147)"

This reverts commit c2d15ac4d4432788557e77c15ce572ac655a8fec.

Causes build failures on PPC stage2 & fuchsia bots
https://lab.llv

Revert "[VPlan] Update final IV exit value via VPlan. (#112147)"

This reverts commit c2d15ac4d4432788557e77c15ce572ac655a8fec.

Causes build failures on PPC stage2 & fuchsia bots
https://lab.llvm.org/buildbot/#/builders/168/builds/7650
https://lab.llvm.org/buildbot/#/builders/11/builds/11248

show more ...

[VPlan] Update final IV exit value via VPlan. (#112147)

Model updating IV users directly in VPlan, replace fixupIVUsers.

Now simple extracts are created for all phis in the exit block during
ini

[VPlan] Update final IV exit value via VPlan. (#112147)

Model updating IV users directly in VPlan, replace fixupIVUsers.

Now simple extracts are created for all phis in the exit block during
initial VPlan construction. A later VPlan transform
(optimizeInductionExitUsers) replaces extracts of inductions with
their pre-computed values if possible.

This completes the transition towards modeling all live-outs directly in
VPlan.

There are a few follow-ups:
* emit extracts initially also for resume phis, and optimize them
tougher with IV exit users
* support for VPlans with multiple exits in optimizeInductionExitUsers.


Depends on https://github.com/llvm/llvm-project/pull/110004,
https://github.com/llvm/llvm-project/pull/109975 and
https://github.com/llvm/llvm-project/pull/112145.

show more ...

[VPlan] Fix mutating whilst iterating over users in EVL transform (#122885)

This fixes a miscompilation extracted from 525.x264_r, where we were
failing to update the runtime VF of a VPReverseVector

[VPlan] Fix mutating whilst iterating over users in EVL transform (#122885)

This fixes a miscompilation extracted from 525.x264_r, where we were
failing to update the runtime VF of a VPReverseVectorPointerRecipe.

We were removing a use of VF whilst iterating over the users() iterator,
which messed up the iterator in-flight and caused us to miss some
recipes. This fixes it by copying the users into a SmallVector first.

Fixes #122681
Fixes #122682

show more ...

[LV] Don't consider IV increments uniform if exit value is used outside.

In some cases, there might be a chain of uniform instructions producing
the exit value. To generate correct code in all cases

[LV] Don't consider IV increments uniform if exit value is used outside.

In some cases, there might be a chain of uniform instructions producing
the exit value. To generate correct code in all cases, consider the IV
increment not uniform, if there are users outside the loop.

Instead, let VPlan narrow the IV, if possible using the logic from
3ff1d01985752.

Test case from #122602 verified with Alive2:
https://alive2.llvm.org/ce/z/bA4EGj

Fixes https://github.com/llvm/llvm-project/issues/122496.
Fixes https://github.com/llvm/llvm-project/issues/122602.

show more ...

Recommit "[VPlan] Try to narrow wide and replicating recipes to uniform recipes."

This reverts commit 0ebb3ac7c92c4c1c44e7f3d17832d75ec5a42a67.

Re-applies commit with typos fixed.

Revert "[VPlan] Try to narrow wide and replicating recipes to uniform recipes."

This reverts commit 1afba19913253dda865a8e57b37b9f4dabead1ac.

Typo breaking the build

[VPlan] Try to narrow wide and replicating recipes to uniform recipes.

Use the existing VPlan-based analysis to identify recipes that only have
their first lane demanded and transform them to unifor

[VPlan] Try to narrow wide and replicating recipes to uniform recipes.

Use the existing VPlan-based analysis to identify recipes that only have
their first lane demanded and transform them to uniform recpliate
recipes. This simplifies the generated code in some places and prepares
for fixing https://github.com/llvm/llvm-project/issues/122496.

show more ...

[VPlan] Skip non-induction phi recipes in legalizeAndOptimizeInductions.

The body of the loop only applies to wide induction recipes, skip any other
header phi recipes up-frond

[VPlan] Remove dead ToRemove (NFC).

[VPlan] Make sure last IV increment value is available if needed.

Legalize extract-from-ends using uniform VPReplicateRecipe of wide
inductions to use regular VPReplicateRecipe, so the correct end v

[VPlan] Make sure last IV increment value is available if needed.

Legalize extract-from-ends using uniform VPReplicateRecipe of wide
inductions to use regular VPReplicateRecipe, so the correct end value
is available.

Fixes https://github.com/llvm/llvm-project/issues/121745.

show more ...

[VPlan] Add and use debug location for VPScalarCastRecipe.

Update the recipe it always take a debug location and set it.

[VPlan] Remove loop region in optimizeForVFAndUF. (#108378)

Update optimizeForVFAndUF to completely remove the vector loop region
when possible. At the moment, we cannot remove the region if it con

[VPlan] Remove loop region in optimizeForVFAndUF. (#108378)

Update optimizeForVFAndUF to completely remove the vector loop region
when possible. At the moment, we cannot remove the region if it contains

* widened IVs: the recipe is needed to generate the step vector
* reductions: ComputeReductionResults requires the reduction phi recipe
for codegen.

Both cases can be addressed by more explicit modeling.

The patch also includes a number of updates to allow executing VPlans
without a vector loop region.

Depends on https://github.com/llvm/llvm-project/pull/110004

show more ...

[VPlan] Fix crash with EVL tail folding intrinsic with no corresponding VP (#121542)

This fixes a crash when building SPEC CPU 2017 with EVL tail folding
when widening @llvm.log10 intrinsics.

@llvm

[VPlan] Fix crash with EVL tail folding intrinsic with no corresponding VP (#121542)

This fixes a crash when building SPEC CPU 2017 with EVL tail folding
when widening @llvm.log10 intrinsics.

@llvm.log10 and some other intrinsics don't have a corresponding VP
intrinsic, so this fixes the crash by removing the assert and bailing
instead.

show more ...

[VPlan] Use removeDeadRecipes in optimizeForVFAndUF (NFCI)

Split off from https://github.com/llvm/llvm-project/pull/108378.

[VPlan] Move simplifyRecipe(s) definitions up to allow re-use (NFC)

Move definitions to allow easy reuse in
https://github.com/llvm/llvm-project/pull/108378.

[VPlan] Manage created blocks directly in VPlan. (NFC) (#120918)

This patch changes the way blocks are managed by VPlan. Previously all
blocks reachable from entry would be cleaned up when a VPlan

[VPlan] Manage created blocks directly in VPlan. (NFC) (#120918)

This patch changes the way blocks are managed by VPlan. Previously all
blocks reachable from entry would be cleaned up when a VPlan is
destroyed. With this patch, each VPlan keeps track of blocks created for
it in a list and this list is then used to delete all blocks in the list
when the VPlan is destroyed. To do so, block creation is funneled
through helpers in directly in VPlan.

The main advantage of doing so is it simplifies CFG transformations, as
those do not have to take care of deleting any blocks, just adjusting
the CFG. This helps to simplify
https://github.com/llvm/llvm-project/pull/108378 and
https://github.com/llvm/llvm-project/pull/106748.

This also simplifies handling of 'immutable' blocks a VPlan holds
references to, which at the moment only include the scalar header block.

PR: https://github.com/llvm/llvm-project/pull/120918

show more ...

[VPlan] Replace else-if dyn_cast with cast (NFC).

The recipes handled here are either VPWidenIntrinsic or VPWidenCast, so
replace the else-if dyn_cast with a single else + cast.

[LV][VPlan] Initialize the variable 'VPID' of the createEVLRecipe (#120926)

Resolve the compilation error caused by the merge issue: #119510

[LV][VPlan] Extract the implementation of transform Recipe to EVLRecipe into a small function. NFC (#119510)

[VPlan] Simplify redundant VPDerivedIVRecipe (NFC).

Split DerivedIV simplification off from
https://github.com/llvm/llvm-project/pull/112145 and use to remove the
need for extra checks in createScal

[VPlan] Simplify redundant VPDerivedIVRecipe (NFC).

Split DerivedIV simplification off from
https://github.com/llvm/llvm-project/pull/112145 and use to remove the
need for extra checks in createScalarIVSteps. Required an extra
simplification run after IV transforms.

show more ...

[LV][VPlan] Use opcode to retrieve the VPID of the CallRecipe, rather than underlying instruction (#120816)

This patch may cause the flags in the CallRecipe to be lost after EVL
transformation, and

[LV][VPlan] Use opcode to retrieve the VPID of the CallRecipe, rather than underlying instruction (#120816)

This patch may cause the flags in the CallRecipe to be lost after EVL
transformation, and it has been addressed in the patch: #119847

show more ...

[VPlan] Set and use debug location for VPPredInstPHIRecipe.

Update the recipe it always set its debug location and use it during IR
generation.