[VPlan] Remove redundant getVPSingleValue for VPSingleDefRecipes (NFC).

[VPlan] Don't created GEP x, 0 for interleave group pointers.

The GEP with offet 0 is redundant, remove it. This addresses a TODO
from 7f74651837b ((#106431).

[VPlan] Use pointer to member 0 as VPInterleaveRecipe's pointer arg. (#106431)

Update VPInterleaveRecipe to always use the pointer to member 0 as
pointer argument. This in many cases helps to remov

[VPlan] Use pointer to member 0 as VPInterleaveRecipe's pointer arg. (#106431)

Update VPInterleaveRecipe to always use the pointer to member 0 as
pointer argument. This in many cases helps to remove unneeded index
adjustments and simplifies VPInterleaveRecipe::execute.

In some rare cases, the address of member 0 does not dominate the insert
position of the interleave group. In those cases a PtrAdd VPInstruction
is emitted to compute the address of member 0 based on the address of
the insert position. Alternatively we could hoist the recipe computing
the address of member 0.

show more ...

[VPlan] Only generate first lane for VPPredInstPHI if no others used.

IF only the first lane of the result is used, only generate the first
lane.

Fixes https://github.com/llvm/llvm-project/issues/1

[VPlan] Only generate first lane for VPPredInstPHI if no others used.

IF only the first lane of the result is used, only generate the first
lane.

Fixes https://github.com/llvm/llvm-project/issues/111042.

show more ...

[VPlan] Manage FMFs for VPWidenCall via VPRecipeWithIRFlags. (NFC)

Update VPWidenCallRecipe to manage fast-math flags directly via
VPRecipeWithIRFlags. This addresses a TODO and allows adjusting the

[VPlan] Manage FMFs for VPWidenCall via VPRecipeWithIRFlags. (NFC)

Update VPWidenCallRecipe to manage fast-math flags directly via
VPRecipeWithIRFlags. This addresses a TODO and allows adjusting the FMFs
directly on the recipe. Also fixes printing for flags for
VPWidenCallRecipe.

show more ...

[LV] Reuse VPReplicateRecipe to handle scalar stores in exit block. (#106342)

This patch separates the computation of the final reduction result and
the intermediate stores of reduction.

-------

[LV] Reuse VPReplicateRecipe to handle scalar stores in exit block. (#106342)

This patch separates the computation of the final reduction result and
the intermediate stores of reduction.

---------

Co-authored-by: Florian Hahn <flo@fhahn.com>

show more ...

[LV] Vectorize histogram operations (#99851)

This patch implements autovectorization support for the 'all-in-one'
histogram intrinsic, which seems to have more support than the
'standalone' intrin

[LV] Vectorize histogram operations (#99851)

This patch implements autovectorization support for the 'all-in-one'
histogram intrinsic, which seems to have more support than the
'standalone' intrinsic. See
https://discourse.llvm.org/t/rfc-vectorization-support-for-histogram-count-operations/74788/
for an overview of the work and my notes on the tradeoffs between the
two approaches.

show more ...

[llvm] Replace uses of Type::getPointerTo() (NFC) (#110163)

Replace uses of `Type::getPointerTo()` which is to be removed.

---------

Co-authored-by: Nikita Popov <github@npopov.com>

[VPlan] Unify mayWriteToMemory and mayHaveSideEffects logic for VPInst.

Unify logic for mayWriteToMemory and mayHaveSideEffects for
VPInstruction, with the later relying on the former. Also extend t

[VPlan] Unify mayWriteToMemory and mayHaveSideEffects logic for VPInst.

Unify logic for mayWriteToMemory and mayHaveSideEffects for
VPInstruction, with the later relying on the former. Also extend to
handle binary operators.

Split off from https://github.com/llvm/llvm-project/pull/106441

show more ...

[VPlan] Implement VPWidenLoad/StoreEVLRecipe::computeCost(). (#109644)

Currently the EVL recipes transfer the tail masking to the EVL.
But in the legacy cost model, the mask exist and will calcula

[VPlan] Implement VPWidenLoad/StoreEVLRecipe::computeCost(). (#109644)

Currently the EVL recipes transfer the tail masking to the EVL.
But in the legacy cost model, the mask exist and will calculate the
instruction cost of the mask.
To fix the difference between the VPlan-based cost model and the legacy
cost model, we always calculate the instruction cost for the mask in the
EVL recipes.

Note that we should remove the mask cost in the EVL recipes when we
don't need to compare to the legacy cost model.

This patch also fixes #109468.

show more ...

[VPlan] Remove VPIteration, update to use directly VPLane instead (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
only the lane part of VPIteration is used.

Simplify th

[VPlan] Remove VPIteration, update to use directly VPLane instead (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
only the lane part of VPIteration is used.

Simplify the code by replacing remaining uses of VPIteration with VPLane directly.

show more ...

[TTI][RISCV] Model cost of loading constants arms of selects and compares (#109824)

This follows in the spirit of 7d82c99403f615f6236334e698720bf979959704,
and extends the costing API for compares

[TTI][RISCV] Model cost of loading constants arms of selects and compares (#109824)

This follows in the spirit of 7d82c99403f615f6236334e698720bf979959704,
and extends the costing API for compares and selects to provide
information about the operands passed in an analogous manner. This
allows us to model the cost of materializing the vector constant, as
some select-of-constants are significantly more expensive than others
when you account for the cost of materializing the constants involved.

This is a stepping stone towards fixing
https://github.com/llvm/llvm-project/issues/109466. A separate SLP patch
will be required to utilize the new API.

show more ...

[LV][EVL]Explicitly model AVL as sub, original TC, EVL_PHI.

Patch explicitly models AVL as sub original TC, EVL_PHI instead of
having it in EXPLICIT-VECTOR-LENGTH VPInstruction. Required for correct

[LV][EVL]Explicitly model AVL as sub, original TC, EVL_PHI.

Patch explicitly models AVL as sub original TC, EVL_PHI instead of
having it in EXPLICIT-VECTOR-LENGTH VPInstruction. Required for correct
safe dependence distance suport.

Reviewers: fhahn, ayalz

Reviewed By: ayalz

Pull Request: https://github.com/llvm/llvm-project/pull/108869

show more ...

[LV] Remove more references of unrolled parts after 57f5d8f2fe.

Continue to clean up some now stale references of unroll parts and
related terminology as pointed out post-commit for 06c3a7d.

[VPlan] Fix incorrect argument for CreateBinOp after 06c3a7d2d764.

06c3a7d2d764 incorrectly updated CreateBinOp to pass the debug location,
which gets interpreted as FPMath node. Remove the argument.

[VPlan] Only store single vector per VPValue in VPTransformState. (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
VPTransformState only stores a single vector value per

[VPlan] Only store single vector per VPValue in VPTransformState. (NFC)

After 8ec406757cb92 (https://github.com/llvm/llvm-project/pull/95842),
VPTransformState only stores a single vector value per VPValue.

Simplify the code by replacing the SmallVector in PerPartOutput with a
single Value * and rename to VPV2Vector for clarity.

Also remove the redundant Part argument from various accessors.

show more ...

[VPlan] Remove unneeded State.UF after 8ec406757cb92 (NFC).

State.UF is not needed any longer after 8ec406757cb92
(https://github.com/llvm/llvm-project/pull/95842). Clean it up,
simplifying ::execut

[VPlan] Remove unneeded State.UF after 8ec406757cb92 (NFC).

State.UF is not needed any longer after 8ec406757cb92
(https://github.com/llvm/llvm-project/pull/95842). Clean it up,
simplifying ::execute of existing recipes.

show more ...

[VPlan] Implement unrolling as VPlan-to-VPlan transform. (#95842)

This patch implements explicit unrolling by UF as VPlan transform. In
follow up patches this will allow simplifying VPTransform st

[VPlan] Implement unrolling as VPlan-to-VPlan transform. (#95842)

This patch implements explicit unrolling by UF as VPlan transform. In
follow up patches this will allow simplifying VPTransform state (no need
to store unrolled parts) as well as recipe execution (no need to
generate code for multiple parts in an each recipe). It also allows for
more general optimziations (e.g. avoid generating code for recipes that
are uniform-across parts).

It also unifies the logic dealing with unrolled parts in a single place,
rather than spreading it out across multiple places (e.g. VPlan post
processing for header-phi recipes previously.)

In the initial implementation, a number of recipes still take the
unrolled part as additional, optional argument, if their execution
depends on the unrolled part.

The computation for start/step values for scalable inductions changed
slightly. Previously the step would be computed as scalar and then
splatted, now vscale gets splatted and multiplied by the step in a
vector mul.

This has been split off https://github.com/llvm/llvm-project/pull/94339
which also includes changes to simplify VPTransfomState and recipes'
::execute.

The current version mostly leaves existing ::execute untouched and
instead sets VPTransfomState::UF to 1.

A follow-up patch will clean up all references to VPTransformState::UF.

Another follow-up patch will simplify VPTransformState to only store a
single vector value per VPValue.

PR: https://github.com/llvm/llvm-project/pull/95842

show more ...

[LLVM] Use {} instead of std::nullopt to initialize empty ArrayRef (#109133)

It is almost always simpler to use {} instead of std::nullopt to
initialize an empty ArrayRef. This patch changes all oc

[LLVM] Use {} instead of std::nullopt to initialize empty ArrayRef (#109133)

It is almost always simpler to use {} instead of std::nullopt to
initialize an empty ArrayRef. This patch changes all occurrences I could
find in LLVM itself. In future the ArrayRef(std::nullopt_t) constructor
could be deprecated or removed.

show more ...

[VPlan] Rename isDefinedOutside[Vector]Regions -> [Loop] (NFC)

Clarify name of helper, split off from
https://github.com/llvm/llvm-project/pull/95842/files#r1765556732.

[VPlan][NFC] Fix the value name of VECTOR_GEP (#107544)

This patch passes the string `"vector.gep"` to CreateGEP instead of
CreateMul.

[LV][NFC] Unify printing for WidenEVLReicpe with other EVL recipes (#108177)

[VPlan] Add VPIRInstruction, use for exit block live-outs. (#100735)

Add a new VPIRInstruction recipe to wrap existing IR instructions not to
be modified during execution, execept for PHIs. For PHI

[VPlan] Add VPIRInstruction, use for exit block live-outs. (#100735)

Add a new VPIRInstruction recipe to wrap existing IR instructions not to
be modified during execution, execept for PHIs. For PHIs, a single
VPValue
operand is allowed, and it is used to add a new incoming value for the
single predecessor VPBB. Expect PHIs, VPIRInstructions cannot have any
operands.

Depends on https://github.com/llvm/llvm-project/pull/100658.

PR: https://github.com/llvm/llvm-project/pull/100735

show more ...

[VPlan] Add VPValue for VF, use it for VPWidenIntOrFpInductionRecipe. (#95305)

Similar to VFxUF, also add a VF VPValue to VPlan and use it to get the
runtime VF in VPWidenIntOrFpInductionRecipe. Co

[VPlan] Add VPValue for VF, use it for VPWidenIntOrFpInductionRecipe. (#95305)

Similar to VFxUF, also add a VF VPValue to VPlan and use it to get the
runtime VF in VPWidenIntOrFpInductionRecipe. Code for VF is only
generated if there are users of VF, to avoid unnecessary test changes.

PR: https://github.com/llvm/llvm-project/pull/95305

show more ...

[Vectorize] Fix a warning

This patch fixes:

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp:1278:12: error:
unused variable 'Op0' [-Werror,-Wunused-variable]