[VPlan] Explicitly handle scalar pointer inductions. (#83068)

Add a new PtrAdd opcode to VPInstruction that corresponds to
IRBuilder::CreatePtrAdd, which creates a GEP with source element type
i8.

[VPlan] Explicitly handle scalar pointer inductions. (#83068)

Add a new PtrAdd opcode to VPInstruction that corresponds to
IRBuilder::CreatePtrAdd, which creates a GEP with source element type
i8.

This is then used to model scalarizing VPWidenPointerInductionRecipe by
introducing scalar-steps to model the index increment followed by a
PtrAdd.

Note that PtrAdd needs to be able to generate code for only the first
lane or for all lanes. This may warrant introducing a separate recipe
for scalarizing that can be created without relying on the underlying
IR.

Depends on https://github.com/llvm/llvm-project/pull/80271

PR: https://github.com/llvm/llvm-project/pull/83068

show more ...

[LV] Use IRBuilder to create and optimize middle-block compare.

Split off from D150398 to avoid builder-related diff changes there.
Using IRBuilder to create ICmps simplifies the result if both oper

[LV] Use IRBuilder to create and optimize middle-block compare.

Split off from D150398 to avoid builder-related diff changes there.
Using IRBuilder to create ICmps simplifies the result if both operands
are constants.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D158332

show more ...

[LV] Use VPValue to get expanded value for SCEV step expressions.

Update skeleton creation logic to use SCEV expansion results from
expanding the pre-header. This avoids another set of SCEV expansio

[LV] Use VPValue to get expanded value for SCEV step expressions.

Update skeleton creation logic to use SCEV expansion results from
expanding the pre-header. This avoids another set of SCEV expansions
that may happen after the CFG has been modified.

Fixes #58811.

Depends on D147964.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147965

show more ...

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is disconnected
at the moment, as the bypass blocks of the skeleton are not yet modeled
in VPlan.

The preheader block is executed before skeleton creation, so the SCEV
expansion results can be used during skeleton creation. At the moment,
the trip count expression and induction steps are expanded in the new
preheader. The remainder of SCEV expansions will be moved gradually in
the future.

D147965 will update skeleton creation to use the steps expanded in the
pre-header to fix #58811.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147964

show more ...

[IVDescriptors] Add pointer InductionDescriptors with non-constant strides (try 2)

(JFYI - This has been heavily reframed since original attempt at landing.)

This change updates the InductionDescri

[IVDescriptors] Add pointer InductionDescriptors with non-constant strides (try 2)

(JFYI - This has been heavily reframed since original attempt at landing.)

This change updates the InductionDescriptor logic to allow matching a pointer IV with a non-constant stride, but also updates the LoopVectorizer to bailout on such descriptors by default. This preserves the default vectorizer behavior.

In review, it was pointed out that there's multiple unfortunate performance implications which need to be addressed before this can be enabled. Having a flag allows us to exercise the behavior, and write test cases for logic which is otherwise unreachable (or hard to reach).

This will also enable non-constant stride pointer recurrences for other consumers. I've audited said code, and don't see any obvious issues.

Differential Revision: https://reviews.llvm.org/D147336

show more ...

[LV] Add tests for non-constant stride pointer inductions

Reduced from the case which triggered the revert of 498aa534f472, and then generalized to cover both expansion paths.