[VPlan] Add and use debug location for VPScalarCastRecipe.

Update the recipe it always take a debug location and set it.

[VPlan] Compute induction end values in VPlan. (#112145)

Use createDerivedIV to compute IV end values directly in VPlan, instead
of creating them up-front.

This allows updating IV users outside

[VPlan] Compute induction end values in VPlan. (#112145)

Use createDerivedIV to compute IV end values directly in VPlan, instead
of creating them up-front.

This allows updating IV users outside the loop as follow-up.

Depends on https://github.com/llvm/llvm-project/pull/110004 and
https://github.com/llvm/llvm-project/pull/109975.

PR: https://github.com/llvm/llvm-project/pull/112145

show more ...

[VPlan] Propagate all GEP flags (#119899)

Store GEPNoWrapFlags instead of only InBounds and propagate them.

[VPlan] Allow setting IR name for VPDerivedIVRecipe (NFCI).

Allow setting the name to use for the generated IR value of the derived
IV in preparations for https://github.com/llvm/llvm-project/pull/1

[VPlan] Allow setting IR name for VPDerivedIVRecipe (NFCI).

Allow setting the name to use for the generated IR value of the derived
IV in preparations for https://github.com/llvm/llvm-project/pull/112145.

This is analogous to VPInstruction::Name.

show more ...

[LV] Vectorize Epilogues for loops with small VF but high IC (#108190)

- Consider MainLoopVF * IC when determining whether Epilogue
Vectorization is profitable
- Allow the same VF for the Epilogue a

[LV] Vectorize Epilogues for loops with small VF but high IC (#108190)

- Consider MainLoopVF * IC when determining whether Epilogue
Vectorization is profitable
- Allow the same VF for the Epilogue as for the main loop
- Use an upper bound for the trip count of the Epilogue when choosing
the Epilogue VF

PR: https://github.com/llvm/llvm-project/pull/108190
---------

Co-authored-by: Florian Hahn <flo@fhahn.com>

show more ...

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handl

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handled by adding
operands to the wrapped phis in the scalar loop header.

Note that the current version only includes the scalar loop header, no
other loop blocks and also does not wrap it in a region block.

PR: https://github.com/llvm/llvm-project/pull/109975

show more ...

[LV] Pass flag indicating epilogue is vectorized to executePlan (NFC)

This clarifies the flag, which is now only passed if the epilogue loop
is being vectorized.

[VPlan] Use pointer to member 0 as VPInterleaveRecipe's pointer arg. (#106431)

Update VPInterleaveRecipe to always use the pointer to member 0 as
pointer argument. This in many cases helps to remov

[VPlan] Use pointer to member 0 as VPInterleaveRecipe's pointer arg. (#106431)

Update VPInterleaveRecipe to always use the pointer to member 0 as
pointer argument. This in many cases helps to remove unneeded index
adjustments and simplifies VPInterleaveRecipe::execute.

In some rare cases, the address of member 0 does not dominate the insert
position of the interleave group. In those cases a PtrAdd VPInstruction
is emitted to compute the address of member 0 based on the address of
the insert position. Alternatively we could hoist the recipe computing
the address of member 0.

show more ...

[LV] Retrieve reduction resume values directly for epilogue vec. (NFC)

Use the reduction resume values from the phis in the scalar header,
instead of collecting them in a map. This removes some comp

[LV] Retrieve reduction resume values directly for epilogue vec. (NFC)

Use the reduction resume values from the phis in the scalar header,
instead of collecting them in a map. This removes some complexity from
the general executePlan code paths and pushes it to only the epilogue
vectorization part.

show more ...

[VPlan] Add createPtrAdd helper (NFC).

Preparation for https://github.com/llvm/llvm-project/pull/106431.

[VPlan] Implement unrolling as VPlan-to-VPlan transform. (#95842)

This patch implements explicit unrolling by UF as VPlan transform. In
follow up patches this will allow simplifying VPTransform st

[VPlan] Implement unrolling as VPlan-to-VPlan transform. (#95842)

This patch implements explicit unrolling by UF as VPlan transform. In
follow up patches this will allow simplifying VPTransform state (no need
to store unrolled parts) as well as recipe execution (no need to
generate code for multiple parts in an each recipe). It also allows for
more general optimziations (e.g. avoid generating code for recipes that
are uniform-across parts).

It also unifies the logic dealing with unrolled parts in a single place,
rather than spreading it out across multiple places (e.g. VPlan post
processing for header-phi recipes previously.)

In the initial implementation, a number of recipes still take the
unrolled part as additional, optional argument, if their execution
depends on the unrolled part.

The computation for start/step values for scalable inductions changed
slightly. Previously the step would be computed as scalar and then
splatted, now vscale gets splatted and multiplied by the step in a
vector mul.

This has been split off https://github.com/llvm/llvm-project/pull/94339
which also includes changes to simplify VPTransfomState and recipes'
::execute.

The current version mostly leaves existing ::execute untouched and
instead sets VPTransfomState::UF to 1.

A follow-up patch will clean up all references to VPTransformState::UF.

Another follow-up patch will simplify VPTransformState to only store a
single vector value per VPValue.

PR: https://github.com/llvm/llvm-project/pull/95842

show more ...

[VPlan] Use VPBuilder to create scalar IV steps and derived IV (NFCI).

Extend VPBuilder to allow creating VPDerivedIVRecipe, VPScalarCastRecipe
and VPScalarIVStepsRecipe.

Use them to simplify the c

[VPlan] Use VPBuilder to create scalar IV steps and derived IV (NFCI).

Extend VPBuilder to allow creating VPDerivedIVRecipe, VPScalarCastRecipe
and VPScalarIVStepsRecipe.

Use them to simplify the code to create scalar IV steps slightly.

show more ...

VPlan/Builder: inline VPBuilder::createICmp (NFC) (#105650)

Inline VPBuilder::createICmp in the header, in line with the other
VPBuilder functions.

[VPlan] Simplify VPBuilder insert point when adding users in exit block.

Simplifies setting the insert point, addressing a TODO.

[VPlan] Factor out precomputing costs from LVP::cost (NFC).

Move the logic for pre-computing costs of certain instructions to a
separate helper function, allowing re-use in a follow-up patch.

[VPlan] Only use selectVectorizationFactor for cross-check (NFCI). (#103033)

Use getBestVF to select VF up-front and only use
selectVectorizationFactor to get the VF legacy VF to check the
vectori

[VPlan] Only use selectVectorizationFactor for cross-check (NFCI). (#103033)

Use getBestVF to select VF up-front and only use
selectVectorizationFactor to get the VF legacy VF to check the
vectorization decision matches the VPlan-based cost model.

PR: https://github.com/llvm/llvm-project/pull/103033

show more ...

[VPlan] Rename getBestPlanFor -> getPlanFor (NFC).

As suggested in https://github.com/llvm/llvm-project/pull/103033, more
accurately rename to getPlanFor , as it simplify returns the VPlan for
VF, r

[VPlan] Rename getBestPlanFor -> getPlanFor (NFC).

As suggested in https://github.com/llvm/llvm-project/pull/103033, more
accurately rename to getPlanFor , as it simplify returns the VPlan for
VF, relying on the fact that there is a single VPlan for each VF at the
moment.

show more ...

[VPlan] Rename getBestVF -> computeBestVF (NFC).

As suggested in https://github.com/llvm/llvm-project/pull/103033, more
accurately rename to computeBestVF, as it now does not simply return the
best

[VPlan] Rename getBestVF -> computeBestVF (NFC).

As suggested in https://github.com/llvm/llvm-project/pull/103033, more
accurately rename to computeBestVF, as it now does not simply return the
best VF, but directly computes it.

show more ...

[LV] Collect profitable VFs in ::getBestVF. (NFCI)

Move collectig profitable VFs to ::getBestVF, in preparation for
retiring selectVectorizationFactor.

[VPlan] Port invalid cost remarks to VPlan. (#99322)

This patch moves the logic to create remarks for instructions with
invalid costs to work on recipes and decoupling it from
selectVectorizationF

[VPlan] Port invalid cost remarks to VPlan. (#99322)

This patch moves the logic to create remarks for instructions with
invalid costs to work on recipes and decoupling it from
selectVectorizationFactor. This is needed to replace the remaining uses
of selectVectorizationFactor with getBestPlan using the VPlan-based cost
model.

The current implementation iterates over all VPlans and their recipes
again, to find recipes with invalid costs, which is more work but will
only be done when remarks for LV are enabled. Once the remaining uses of
selectVectorizationFactor are retired, we can collect VPlans with
invalid costs as part of getBestPlan if we want to optimize the remarks
case a bit, at the cost of adding additional complexity.

PR: https://github.com/llvm/llvm-project/pull/99322

show more ...

[VPlan] Replace getBestPlan by getBestVF use also for epilogue vec. (#98821)

Replace getBestPlan by getBestVF which simply finds the best
VF out of the VFs for the available VPlans.

Then use get

[VPlan] Replace getBestPlan by getBestVF use also for epilogue vec. (#98821)

Replace getBestPlan by getBestVF which simply finds the best
VF out of the VFs for the available VPlans.

Then use getBestPlan to retrieve the corresponding VPlan.

This allows using getBestVF & getBestPlan for epilogue vectorization
as well. As the same plan may be used to vectorize both the main
and epilogue loop, restricting the VF of the best plan would cause
issues.

PR: https://github.com/llvm/llvm-project/pull/98821

show more ...

Recommit "[VPlan] First step towards VPlan cost modeling. (#92555)"

This reverts commit 6f538f6a2d3224efda985e9eb09012fa4275ea92.

A number of crashes have been fixed by separate fixes, including
tt

Recommit "[VPlan] First step towards VPlan cost modeling. (#92555)"

This reverts commit 6f538f6a2d3224efda985e9eb09012fa4275ea92.

A number of crashes have been fixed by separate fixes, including
ttps://github.com/llvm/llvm-project/pull/96622. This version of the
PR also pre-computes the costs for branches (except the latch) instead
of computing their costs as part of costing of replicate regions, as
there may not be a direct correspondence between original branches and
number of replicate regions.

Original message:
This adds a new interface to compute the cost of recipes, VPBasicBlocks,
VPRegionBlocks and VPlan, initially falling back to the legacy cost model
for all recipes. Follow-up patches will gradually migrate recipes to
compute their own costs step-by-step.

It also adds getBestPlan function to LVP which computes the cost of all
VPlans and picks the most profitable one together with the most
profitable VF.

The VPlan selected by the VPlan cost model is executed and there is an
assert to catch cases where the VPlan cost model and the legacy cost
model disagree. Even though I checked a number of different build
configurations on AArch64 and X86, there may be some differences
that have been missed.

Additional discussions and context can be found in @arcbbb's
https://github.com/llvm/llvm-project/pull/67647 and
https://github.com/llvm/llvm-project/pull/67934 which is an earlier
version of the current PR.

PR: https://github.com/llvm/llvm-project/pull/92555

show more ...

[VPlan] Retrieve LatchVPBB from region in adjustRecipesForRed (NFC)

The HeaderVPBB is retrieved in a similar fashion already.

This is in preparation for splitting up the very large
tryToBuildVPlanW

[VPlan] Retrieve LatchVPBB from region in adjustRecipesForRed (NFC)

The HeaderVPBB is retrieved in a similar fashion already.

This is in preparation for splitting up the very large
tryToBuildVPlanWithVPRecipes into several distinct functions, as
suggested multiple times, including in
https://github.com/llvm/llvm-project/pull/94760

show more ...

[LV] Move check if any vector insts will be generated to VPlan. (#96622)

This patch moves the check if any vector instructions will be generated
from getInstructionCost to be based on VPlan. This s

[LV] Move check if any vector insts will be generated to VPlan. (#96622)

This patch moves the check if any vector instructions will be generated
from getInstructionCost to be based on VPlan. This simplifies
getInstructionCost, is more accurate as we check the final result and
also allows us to exit early once we visit a recipe that generates
vector instructions.

The helper can then be re-used by the VPlan-based cost model to match
the legacy selectVectorizationFactor behavior, this fixing a crash and
paving the way to recommit
https://github.com/llvm/llvm-project/pull/92555.

PR: https://github.com/llvm/llvm-project/pull/96622

show more ...

[VPlan] Iterate over VPlans to get VFs to compute cost for (NFCI).

Instead for iterating over all VFs when computing costs, simply iterate
over the VFs available in the created VPlans.

Split off fr

[VPlan] Iterate over VPlans to get VFs to compute cost for (NFCI).

Instead for iterating over all VFs when computing costs, simply iterate
over the VFs available in the created VPlans.

Split off from https://github.com/llvm/llvm-project/pull/92555.

This also prepares for moving the check if any vector instructions will
be generated to be based on VPlan, to unblock recommitting
https://github.com/llvm/llvm-project/pull/92555.

show more ...