[VPlan] Implement cloning of VPlans. (#73158)

This patch implements cloning for VPlans and recipes. Cloning is used in
the epilogue vectorization path, to clone the VPlan for the main vector
loop.

[VPlan] Implement cloning of VPlans. (#73158)

This patch implements cloning for VPlans and recipes. Cloning is used in
the epilogue vectorization path, to clone the VPlan for the main vector
loop. This means we won't re-use a VPlan when executing the VPlan for
the epilogue vector loop, which in turn will enable us to perform
optimizations based on UF & VF.

show more ...

[VPlan] Relax IV user assertion after 0ab539f for epilogue vec.

After 0ab539fd6748adf2f638e10514dd9419597d8863, the canonical IV in the
epilogue vector loop may be used by a trunc. Relax the corresp

[VPlan] Relax IV user assertion after 0ab539f for epilogue vec.

After 0ab539fd6748adf2f638e10514dd9419597d8863, the canonical IV in the
epilogue vector loop may be used by a trunc. Relax the corresponding
assert.

This should fix some build-bot failures, including
https://lab.llvm.org/buildbot/#/builders/187/builds/14113
https://lab.llvm.org/buildbot/#/builders/98/builds/32350
https://lab.llvm.org/buildbot/#/builders/239/builds/5473

show more ...

[VPlan] Use replaceUsesWithIf in replaceAllUseswith and add comment (NFCI).

Follow-up to post-commit commens for b1bfe221e6.

[VPlan] Introduce ComputeReductionResult VPInstruction opcode. (#70253)

This patch introduces a new ComputeReductionResult opcode to compute the
final reduction result in the middle block. The code

[VPlan] Introduce ComputeReductionResult VPInstruction opcode. (#70253)

This patch introduces a new ComputeReductionResult opcode to compute the
final reduction result in the middle block. The code from fixReduction
has been moved to ComputeReductionResult, after some earlier cleanup
changes to model parts of fixReduction explicitly elsewhere as needed.

The recipe may be broken down further in the future.

Note that the phi nodes to merge the reduction result from the trip
count check and the middle block, to be used as resume value for the
scalar remainder loop are also generated based on
ComputeReductionResult.

Once we have a VPValue for the reduction result, this can also be
modeled explicitly and moved out of the recipe.

show more ...

[VPlan] Remove unneeded getNumUsers calls in replaceAllUsesWith (NFC).

As suggested post-commit for a00227197, replace unnecessary getNumUsers
calls by boolean variable to indicate if users changed.

[VPlan] Remove unneeded getNumUsers calls in replaceAllUsesWith (NFC).

As suggested post-commit for a00227197, replace unnecessary getNumUsers
calls by boolean variable to indicate if users changed. Note that this
also requires an early exit to detect the case where a value is replaced
by itself.

show more ...

[VPlan] Initial modeling of VF * UF as VPValue. (#74761)

This patch starts initial modeling of VF * UF in VPlan.
Initially, introduce a dedicated VFxUF VPValue, which is then
populated during VPla

[VPlan] Initial modeling of VF * UF as VPValue. (#74761)

This patch starts initial modeling of VF * UF in VPlan.
Initially, introduce a dedicated VFxUF VPValue, which is then
populated during VPlan::prepareToExecute. Initially, the VF * UF
applies only to the main vector loop region. Once we extend the
scope of VPlan in the future, we may want to associate different VFxUFs
with different vector loop regions (e.g. the epilogue vector loop)

This allows explicitly parameterizing recipes that rely on the
VF * UF, like the canonical induction increment. At the moment, this
mainly helps to avoid generating some duplicated calls to vscale with
scalable vectors. It should also allow using EVL as induction increments
explicitly in D99750. Referring to VF * UF is also needed in other
places that we plan to migrate to VPlan, like the minimum trip count
check during skeleton creation.

The first version creates the value for VF * UF directly in
prepareToExecute to limit the scope of the patch. A follow-on patch will
model VF * UF computation explicitly in VPlan using recipes.

Moved from Phabricator (https://reviews.llvm.org/D157322)

show more ...

[VPlan] Add missing output of live-ins to VPlan dot printing.

Split off live-in printing to VPlan::printLiveIns and use it to print
Live-ins when printing in the DOT format.

[VPlan] Remove dead IsEpilogueVec argument from prepareToExecute (NFC).

[VPlan] Move initial skeleton construction to createInitialVPlan. (NFC)

This patch moves creating the middle VPBBs and an initial empty
vector loop region for the top-level loop to createInitialVPl

[VPlan] Move initial skeleton construction to createInitialVPlan. (NFC)

This patch moves creating the middle VPBBs and an initial empty
vector loop region for the top-level loop to createInitialVPlan.

This consolidates code to create the initial VPlan skeleton and enables
adding other bits outside the main region during initial VPlan
construction. In particular, D150398 will add the exit check & branch to
the middle block.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D158333

show more ...

[VPlan] Add VPValue::replaceUsesWithIf (NFCI).

Add replaceUsesWithIf helper and use it in a few places.

[llvm] Stop including llvm/ADT/DepthFirstIterator.h (NFC)

Identified with misc-include-cleaner.

[VPlan] Add active-lane-mask as VPlan-to-VPlan transformation.

This patch updates the mask creation code to always create compares of
the form (ICMP_ULE, wide canonical IV, backedge-taken-count) up

[VPlan] Add active-lane-mask as VPlan-to-VPlan transformation.

This patch updates the mask creation code to always create compares of
the form (ICMP_ULE, wide canonical IV, backedge-taken-count) up front
when tail folding and introduce active-lane-mask as later
transformation.

This effectively makes (ICMP_ULE, wide canonical IV, backedge-taken-count)
the canonical form for tail-folding early on. Introducing more specific
active-lane-mask recipes is treated as a VPlan-to-VPlan optimization.

This has the advantage of keeping the logic (and complexity) of
introducing active-lane-mask recipes in a single place, instead of
spreading the logic out across multiple functions. It also simplifies
initial VPlan construction and enables treating introducing EVL as
similar optimization.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D158779

show more ...

[VPlan] Remove reference to Instr when setting debug loc. (NFCI)

This allows untangling references to underlying IR for various recipes.

[VPlan] Assert that inst isnt' a debug or pseudo inst (NFCI).

Debug and pseudo instructions aren't modeled in VPlan. Turn a check into
an assertion. This will help removing the direct use of Inst he

[VPlan] Assert that inst isnt' a debug or pseudo inst (NFCI).

Debug and pseudo instructions aren't modeled in VPlan. Turn a check into
an assertion. This will help removing the direct use of Inst here in the
future.

show more ...

[VPlan] Fold if into return in prepareToExecute assertion (NFC).

Independent simplification suggested in D157194.

[VPlan] Model wrap flags directly, remove *NUW opcodes (NFC)

Model wrap flags directly using VPRecipeWithIRFlags and clean up the
duplicated *NUW opcodes.

D157144 will build on this and also model

[VPlan] Model wrap flags directly, remove *NUW opcodes (NFC)

Model wrap flags directly using VPRecipeWithIRFlags and clean up the
duplicated *NUW opcodes.

D157144 will build on this and also model FMFs for VPInstruction.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D157194

show more ...

[VPlan] Move VPTransformState::get() to VPlan.cpp (NFC).

The last dependency of code defined in LoopVectorize.cpp has been
removed a while ago. Move VPTransformState::get() to VPlan.cpp where
other

[VPlan] Move VPTransformState::get() to VPlan.cpp (NFC).

The last dependency of code defined in LoopVectorize.cpp has been
removed a while ago. Move VPTransformState::get() to VPlan.cpp where
other members are also defined.

show more ...

[LV] Move packScalarIntoVectorValue to VPTransformState (NFC).

This moves packScalarIntoVectorValue from ILV to the more approriate
VPTransformState.

[llvm] Add missing StringExtras.h includes

In preparation for removing the `#include "llvm/ADT/StringExtras.h"`
from the header to source file of `llvm/Support/Error.h`, first add in
all the missing

[llvm] Add missing StringExtras.h includes

In preparation for removing the `#include "llvm/ADT/StringExtras.h"`
from the header to source file of `llvm/Support/Error.h`, first add in
all the missing includes that were previously included transitively
through this header.

show more ...

[VPlan] Move live-out printing to VPLiveOut::print (NFC).

Preparation for D150398. This brings live-out printing in line with how
printing for recipes is handled.

[PseudoProbe] Clean up dwarf discriminator and avoid duplicating factor.

A pseudo probe is created with dwarf line information shared with its nearest instruction. If the instruction comes with a dw

[PseudoProbe] Clean up dwarf discriminator and avoid duplicating factor.

A pseudo probe is created with dwarf line information shared with its nearest instruction. If the instruction comes with a dwarf discriminator, it will be shared with the probe as well. This can confuse the later FS-AFDO discriminator assignment pass. To fix this, I'm cleaning up the discriminator fields for probes when they are inserted.

I also notice another possibility to change the discriminator field of pseudo probes in the pipeline before the FS discriminator assignment pass. That is the loop unroller, which assigns duplication factor to instruction being vectorized. I'm disabling that for pseudo probe intrinsics specifically, also for callsites with probes.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D148569

show more ...

[VPlan] Add VPWidenCastRecipe, split off from VPWidenRecipe (NFCI).

To generate cast instructions, the result type is needed. To allow
creating widened casts without underlying instruction, introduc

[VPlan] Add VPWidenCastRecipe, split off from VPWidenRecipe (NFCI).

To generate cast instructions, the result type is needed. To allow
creating widened casts without underlying instruction, introduce a new
VPWidenCastRecipe that also holds the result type.

This functionality will be used in a follow-up patch to
implement truncateToMinimalBitwidths as VPlan-to-VPlan transform.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D149081

show more ...

[VPlan] Clean up preheader block after b85a402dd899fc.

Fix a leak introduced in b85a402dd899fc and flagged by LSan
https://lab.llvm.org/buildbot#builders/5/builds/33452

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is disconnected
at the moment, as the bypass blocks of the skeleton are not yet modeled
in VPlan.

The preheader block is executed before skeleton creation, so the SCEV
expansion results can be used during skeleton creation. At the moment,
the trip count expression and induction steps are expanded in the new
preheader. The remainder of SCEV expansions will be moved gradually in
the future.

D147965 will update skeleton creation to use the steps expanded in the
pre-header to fix #58811.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147964

show more ...

[LV] Use VPValue for SCEV expansion in fixupIVUsers.

The step is already expanded in the VPlan. Use this expansion instead.
This is a step towards modeling fixing up IV users in VPlan.

It also fix

[LV] Use VPValue for SCEV expansion in fixupIVUsers.

The step is already expanded in the VPlan. Use this expansion instead.
This is a step towards modeling fixing up IV users in VPlan.

It also fixes a crash casued by SCEV-expanding the Step expression in
fixupIVUsers, where the IR is in an incomplete state

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147963

show more ...