[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

[VPlan][NFC] Fix the value name of VECTOR_GEP (#107544)

This patch passes the string `"vector.gep"` to CreateGEP instead of
CreateMul.

[VPlan] Explicitly handle scalar pointer inductions. (#83068)

Add a new PtrAdd opcode to VPInstruction that corresponds to
IRBuilder::CreatePtrAdd, which creates a GEP with source element type
i8.

[VPlan] Explicitly handle scalar pointer inductions. (#83068)

Add a new PtrAdd opcode to VPInstruction that corresponds to
IRBuilder::CreatePtrAdd, which creates a GEP with source element type
i8.

This is then used to model scalarizing VPWidenPointerInductionRecipe by
introducing scalar-steps to model the index increment followed by a
PtrAdd.

Note that PtrAdd needs to be able to generate code for only the first
lane or for all lanes. This may warrant introducing a separate recipe
for scalarizing that can be created without relying on the underlying
IR.

Depends on https://github.com/llvm/llvm-project/pull/80271

PR: https://github.com/llvm/llvm-project/pull/83068

show more ...

[LV] Use IRBuilder to create and optimize middle-block compare.

Split off from D150398 to avoid builder-related diff changes there.
Using IRBuilder to create ICmps simplifies the result if both oper

[LV] Use IRBuilder to create and optimize middle-block compare.

Split off from D150398 to avoid builder-related diff changes there.
Using IRBuilder to create ICmps simplifies the result if both operands
are constants.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D158332

show more ...

[LV] Add tests for reasoning about SCEV predicates.

Add extra tests with cases where SCEV predicates can be proven to always
be false. The test in pointer-induction.ll has been adjusted to avoid
the

[LV] Add tests for reasoning about SCEV predicates.

Add extra tests with cases where SCEV predicates can be proven to always
be false. The test in pointer-induction.ll has been adjusted to avoid
the induction always to wrap.

show more ...

[LV] Use VPValue to get expanded value for SCEV step expressions.

Update skeleton creation logic to use SCEV expansion results from
expanding the pre-header. This avoids another set of SCEV expansio

[LV] Use VPValue to get expanded value for SCEV step expressions.

Update skeleton creation logic to use SCEV expansion results from
expanding the pre-header. This avoids another set of SCEV expansions
that may happen after the CFG has been modified.

Fixes #58811.

Depends on D147964.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147965

show more ...

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is disconnected
at the moment, as the bypass blocks of the skeleton are not yet modeled
in VPlan.

The preheader block is executed before skeleton creation, so the SCEV
expansion results can be used during skeleton creation. At the moment,
the trip count expression and induction steps are expanded in the new
preheader. The remainder of SCEV expansions will be moved gradually in
the future.

D147965 will update skeleton creation to use the steps expanded in the
pre-header to fix #58811.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147964

show more ...

[IVDescriptors] Add pointer InductionDescriptors with non-constant strides (try 2)

(JFYI - This has been heavily reframed since original attempt at landing.)

This change updates the InductionDescri

[IVDescriptors] Add pointer InductionDescriptors with non-constant strides (try 2)

(JFYI - This has been heavily reframed since original attempt at landing.)

This change updates the InductionDescriptor logic to allow matching a pointer IV with a non-constant stride, but also updates the LoopVectorizer to bailout on such descriptors by default. This preserves the default vectorizer behavior.

In review, it was pointed out that there's multiple unfortunate performance implications which need to be addressed before this can be enabled. Having a flag allows us to exercise the behavior, and write test cases for logic which is otherwise unreachable (or hard to reach).

This will also enable non-constant stride pointer recurrences for other consumers. I've audited said code, and don't see any obvious issues.

Differential Revision: https://reviews.llvm.org/D147336

show more ...

[LV] Add tests for non-constant stride pointer inductions

Reduced from the case which triggered the revert of 498aa534f472, and then generalized to cover both expansion paths.

[LoopVectorize] Convert some tests to opaque pointers (NFC)

Check lines for some of these tests were regenerated. The difference
is that with opaque pointers SCEVExpander always emits i8 GEPs,
makin

[LoopVectorize] Convert some tests to opaque pointers (NFC)

Check lines for some of these tests were regenerated. The difference
is that with opaque pointers SCEVExpander always emits i8 GEPs,
making the address calculation explicit. This is a known problem
that will be solved long term by making all address calculations
explicit.

show more ...

[NFC] Port all LoopVectorize tests to `-passes=` syntax

[VPlan] Remove dead recipes before sinking.

optimizeInductions may leave dead recipes which can prevent sinking.
Sinking on the other hand should not introduce new dead recipes, so
clean up dead rec

[VPlan] Remove dead recipes before sinking.

optimizeInductions may leave dead recipes which can prevent sinking.
Sinking on the other hand should not introduce new dead recipes, so
clean up dead recipes before sinking.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D133762

show more ...

[LV] Refresh autogen tests to reflect naming changes [nfc]

Purely so that these can be easily autogened without spurious diffs

[LV] Remove collectTriviallyDeadInstructions, already handled by VP DCE.

Now that removeDeadRecipes can remove most dead recipes across a whole
VPlan, there is no need to first collect some dead ins

[LV] Remove collectTriviallyDeadInstructions, already handled by VP DCE.

Now that removeDeadRecipes can remove most dead recipes across a whole
VPlan, there is no need to first collect some dead instructions.
Instead removeDeadRecipes can simply clean them up.

Depends D127580.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D128408

show more ...

[test][NewPM] Use -passes=loop-vectorize instead of -loop-vectorize

Update a bunch of loop-vectorize regression tests to use the new PM
syntax (opt -passes=loop-vectorize) instead of the deprecated

[test][NewPM] Use -passes=loop-vectorize instead of -loop-vectorize

Update a bunch of loop-vectorize regression tests to use the new PM
syntax (opt -passes=loop-vectorize) instead of the deprecated legacy
PM syntax (opt -loop-vectorize).

show more ...

Revert "[NFCI] Regenerate SROA/LoopVectorize test checks"

This reverts commit 14e3450fb57305aa9ff3e9e60687b458e43835c9.

[NFCI] Regenerate LoopVectorize test checks

Recommit "[VPlan] Introduce recipe to build scalar steps."

This reverts the revert commit ff93260bf6bddfbad1fa65c4d5184988885b900f.

The underlying issue causing the PPC bot failures has been fixed

Recommit "[VPlan] Introduce recipe to build scalar steps."

This reverts the revert commit ff93260bf6bddfbad1fa65c4d5184988885b900f.

The underlying issue causing the PPC bot failures has been fixed in
cbaac1473403 and a corresponding test case has been added in
ad2cad1c521c.

Original message:

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
induction variable needed.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D115953

show more ...

Revert "[VPlan] Introduce recipe to build scalar steps."

This reverts commit 49b23f451cf713036c99573a35daed308d2ac894.

This appears to break some PPC build bots. Revert while I investigate.

[VPlan] Introduce recipe to build scalar steps.

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
in

[VPlan] Introduce recipe to build scalar steps.

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
induction variable needed.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D115953

show more ...

[VPlan] Remove dead header-phi recipes.

This patch adds a new transform to remove dead recipes. For now, it only
removes dead recipes in the header, to keep the number tests that require
updating ma

[VPlan] Remove dead header-phi recipes.

This patch adds a new transform to remove dead recipes. For now, it only
removes dead recipes in the header, to keep the number tests that require
updating manageable. Future patches will extend this to remove dead
recipes across the whole plan.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D118051

show more ...

[VPlan] Add VPCanonicalIVPHIRecipe, partly retire createInductionVariable.

At the moment, the primary induction variable for the vector loop is
created as part of the skeleton creation. This is tied

[VPlan] Add VPCanonicalIVPHIRecipe, partly retire createInductionVariable.

At the moment, the primary induction variable for the vector loop is
created as part of the skeleton creation. This is tied to creating the
vector loop latch outside of VPlan. This prevents from modeling the
*whole* vector loop in VPlan, which in turn is required to model
preheader and exit blocks in VPlan as well.

This patch introduces a new recipe VPCanonicalIVPHIRecipe to represent the
primary IV in VPlan and CanonicalIVIncrement{NUW} opcodes for
VPInstruction to model the increment.

This allows us to partly retire createInductionVariable. At the moment,
a bit of patching up is done after executing all blocks in the plan.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D113223

show more ...

[LV] Fix incorrectly marking a pointer indvar as 'scalar'.

collectLoopScalars should only add non-uniform nodes to the list if they
are used by a load/store instruction that is marked as CM_Scalariz

[LV] Fix incorrectly marking a pointer indvar as 'scalar'.

collectLoopScalars should only add non-uniform nodes to the list if they
are used by a load/store instruction that is marked as CM_Scalarize.

Before this patch, the LV incorrectly marked pointer induction variables
as 'scalar' when they required to be widened by something else,
such as a compare instruction, and weren't used by a node marked as
'CM_Scalarize'. This case is covered by sve-widen-phi.ll.

This change also allows removing some code where the LV tried to
widen the PHI nodes with a stepvector, even though it was marked as
'scalarAfterVectorization'. Now that this code is more careful about
marking instructions that need widening as 'scalar', this code has
become redundant.

Differential Revision: https://reviews.llvm.org/D114373

show more ...

Revert "[IR] `IRBuilderBase::CreateAdd()`: short-circuit `x + 0` --> `x`"

Clang OpenMP codegen tests are failing.

This reverts commit 288f1f8abe5835180a0021f142043ee261ab3846.
This reverts commit c

Revert "[IR] `IRBuilderBase::CreateAdd()`: short-circuit `x + 0` --> `x`"

Clang OpenMP codegen tests are failing.

This reverts commit 288f1f8abe5835180a0021f142043ee261ab3846.
This reverts commit cb90e5356ac1594e95fed8e208d6e0e9b6a87db1.

show more ...

[IR] `IRBuilderBase::CreateAdd()`: short-circuit `x + 0` --> `x`

There's precedent for that in `CreateOr()`/`CreateAnd()`.

The motivation here is to avoid bloating the run-time check's IR
in `SCEVE

[IR] `IRBuilderBase::CreateAdd()`: short-circuit `x + 0` --> `x`

There's precedent for that in `CreateOr()`/`CreateAnd()`.

The motivation here is to avoid bloating the run-time check's IR
in `SCEVExpander::generateOverflowCheck()`.

Refs. https://reviews.llvm.org/D109368#3089809

show more ...