[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

This was originally done to reduce the diff for the change. Remove it
and update the remaining tests. NFC modulo reordering of

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

This was originally done to reduce the diff for the change. Remove it
and update the remaining tests. NFC modulo reordering of incoming
values.

Clean up after https://github.com/llvm/llvm-project/pull/114292.

show more ...

Reapply "[VPlan] Update scalar induction resume values in VPlan. (#110577)"

This reverts commit f09b16e2671cbcdf7cb7dc7ed705db092a9deda1.

The crash when building llvm-test-suite with stage2 should

Reapply "[VPlan] Update scalar induction resume values in VPlan. (#110577)"

This reverts commit f09b16e2671cbcdf7cb7dc7ed705db092a9deda1.

The crash when building llvm-test-suite with stage2 should have been
fixed by 1091fad31a83d5ab87eb6fa11fe3bdb3f0d152ea.

show more ...

Revert "[VPlan] Update scalar induction resume values in VPlan. (#110577)"

This reverts commit 0678e2058364ec10b94560d27ec7138dfa003287.
This reverts commit 1091fad31a83d5ab87eb6fa11fe3bdb3f0d152ea.

Revert "[VPlan] Update scalar induction resume values in VPlan. (#110577)"

This reverts commit 0678e2058364ec10b94560d27ec7138dfa003287.
This reverts commit 1091fad31a83d5ab87eb6fa11fe3bdb3f0d152ea.

Causes crashes in llvm-test-suite when using stage 2 clang.

show more ...

[VPlan] Update scalar induction resume values in VPlan. (#110577)

Updated ILV.createInductionResumeValues (now createInductionResumeVPValue)
to directly update the VPIRInstructions wrapping the ori

[VPlan] Update scalar induction resume values in VPlan. (#110577)

Updated ILV.createInductionResumeValues (now createInductionResumeVPValue)
to directly update the VPIRInstructions wrapping the original phis with the
created resume values.

This is the first step towards modeling them completely in VPlan.
Subsequent patches will move creation of the resume values completely
into VPlan.

Depends on https://github.com/llvm/llvm-project/pull/109975.

PR: https://github.com/llvm/llvm-project/pull/110577

show more ...

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

[VPlan] Run DCE after unrolling.

This cleans up a number of dead recipes after unrolling if only their
first or last parts are used. This simplifies a number of tests.

Fixes https://github.com/llvm

[VPlan] Run DCE after unrolling.

This cleans up a number of dead recipes after unrolling if only their
first or last parts are used. This simplifies a number of tests.

Fixes https://github.com/llvm/llvm-project/issues/109581.

show more ...

[VPlan] Update stale test after 9536a6286, fix formatting.

[LV] Change loops' interleave count computation (#73766)

[LV] Change loops' interleave count computation

A set of microbenchmarks in llvm-test-suite (https://github.com/llvm/llvm-test-suite/pull/

[LV] Change loops' interleave count computation (#73766)

[LV] Change loops' interleave count computation

A set of microbenchmarks in llvm-test-suite (https://github.com/llvm/llvm-test-suite/pull/56), when tested on a AArch64 platform, demonstrates that loop interleaving is beneficial when the vector loop runs at least twice or when the epilogue loop trip count (TC) is minimal. Therefore, we choose interleaving count (IC) between TC/VF & TC/2*VF (VF = vectorization factor), such that remainder TC for the epilogue loop is minimum while the IC is maximum in case the remainder TC is same for both.

The initial tests for this change were submitted in PRs:
https://github.com/llvm/llvm-project/pull/70272 and https://github.com/llvm/llvm-project/pull/74689.

show more ...

[VPlan] Model address separately. (#72164)

Move vector pointer generation to a separate VPVectorPointerRecipe.
This untangles address computation from the memory recipes future
and is also needed

[VPlan] Model address separately. (#72164)

Move vector pointer generation to a separate VPVectorPointerRecipe.
This untangles address computation from the memory recipes future
and is also needed to enable explicit unrolling in VPlan.

https://github.com/llvm/llvm-project/pull/72164

show more ...

[LoopVectorize] Convert most tests to opaque pointers (NFC)

The unsized-pointee-crash.ll and zero-sized-pointee-crash.ll tests
have been removed, because these issues are not relevant for opaque
poi

[LoopVectorize] Convert most tests to opaque pointers (NFC)

The unsized-pointee-crash.ll and zero-sized-pointee-crash.ll tests
have been removed, because these issues are not relevant for opaque
pointers.

show more ...

[AsmParser] Remove typed pointer auto-detection

IR is now always parsed in opaque pointer mode, unless
-opaque-pointers=0 is explicitly given. There is no automatic
detection of typed pointers anymo

[AsmParser] Remove typed pointer auto-detection

IR is now always parsed in opaque pointer mode, unless
-opaque-pointers=0 is explicitly given. There is no automatic
detection of typed pointers anymore.

The -opaque-pointers=0 option is added to any remaining IR tests
that haven't been migrated yet.

Differential Revision: https://reviews.llvm.org/D141912

show more ...

[LV] Disable runtime unrolling for vectorized loops.

This patch adds metadata to disable runtime unrolling to the vectorized
loop. If runtime unrolling/interleaving is considered profitable, LV
will

[LV] Disable runtime unrolling for vectorized loops.

This patch adds metadata to disable runtime unrolling to the vectorized
loop. If runtime unrolling/interleaving is considered profitable, LV
will interleave the loop directly. There should be no need to perform
runtime unrolling at a later stage.

Note that we already add metadata to disable runtime unrolling to the
scalar loop after vectorization.

The additional unrolling unnecessarily increases code size and compile
time. In addition to that we have several bug reports of unncessary
runtime unrolling for vectorized loops, e.g. PR40961

Compile-time improvements:

NewPM-O3: -1.04%
NewPM-ReleaseThinLTO: -0.59%
NewPM-ReleaseLTO-g: -0.97%

https://llvm-compile-time-tracker.com/compare.php?from=ce1be13a868d0f8afa367975558c1a6175cce33a&to=78bc2e67f22e9e10e61cdb6cdac4bb857d95eb1b&stat=instructions:u

Fixes #40306.

Reviewed By: lebedev.ri, nikic

Differential Revision: https://reviews.llvm.org/D115261

show more ...

[LV] Do not LoopSimplify/LCSSA after generating main vector loop.

At the moment LV runs LoopSimplify and reconstructs LCSSA form after
generating the main vector loop and before generating the epilo

[LV] Do not LoopSimplify/LCSSA after generating main vector loop.

At the moment LV runs LoopSimplify and reconstructs LCSSA form after
generating the main vector loop and before generating the epilogue
vector loop.

In practice, this adds a new exit block for the scalar loop because the
middle block now also branches to the original exit block of the scalar
loop. It also requires adding a new LCSSA phi in the newly created exit
block.

This complicates things when modeling exit values in VPlan, because we
would need to update the VPlan for the epilogue loop to update the newly
created LCSSA phi node.

But none of that should be necessary, as all analysis requiring
loop-simplify form is already done at this point and LCSSA form of the
original loop is not broken.

Reviewed By: bmahjour

Differential Revision: https://reviews.llvm.org/D125810

show more ...

[LV] Regenerate check lines for some tests.

Make sure the auto-generated check lines are up-to-date for some files,
to reduce the test diff in upcoming changes

Revert "[NFCI] Regenerate SROA/LoopVectorize test checks"

This reverts commit 14e3450fb57305aa9ff3e9e60687b458e43835c9.

[NFCI] Regenerate LoopVectorize test checks

Recommit "[VPlan] Introduce recipe to build scalar steps."

This reverts the revert commit ff93260bf6bddfbad1fa65c4d5184988885b900f.

The underlying issue causing the PPC bot failures has been fixed

Recommit "[VPlan] Introduce recipe to build scalar steps."

This reverts the revert commit ff93260bf6bddfbad1fa65c4d5184988885b900f.

The underlying issue causing the PPC bot failures has been fixed in
cbaac1473403 and a corresponding test case has been added in
ad2cad1c521c.

Original message:

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
induction variable needed.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D115953

show more ...

Revert "[VPlan] Introduce recipe to build scalar steps."

This reverts commit 49b23f451cf713036c99573a35daed308d2ac894.

This appears to break some PPC build bots. Revert while I investigate.

[VPlan] Introduce recipe to build scalar steps.

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
in

[VPlan] Introduce recipe to build scalar steps.

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
induction variable needed.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D115953

show more ...

[LV] Restore metadata to disable runtime unrolling for epilogue loop.

After d4a8fc3a87a1 LV stopped adding metadata to disable runtime
unrolling to the vectorized epilogue loop. This was missed beca

[LV] Restore metadata to disable runtime unrolling for epilogue loop.

After d4a8fc3a87a1 LV stopped adding metadata to disable runtime
unrolling to the vectorized epilogue loop. This was missed because
278aa65cc495 removed the relevant test coverage.

This patch fixes that by adding the relevant metadata after
vector loop generation.

show more ...

[SCEVExpander] Only check overflow if it is needed.

9345ab3a4550 updated generateOverflowCheck to skip creating checks that
always evaluate to false. This in turn means that we only need to check
fo

[SCEVExpander] Only check overflow if it is needed.

9345ab3a4550 updated generateOverflowCheck to skip creating checks that
always evaluate to false. This in turn means that we only need to check
for overflows if the result of the multiplication is actually used.

Sink the Or for the overflow check into ComputeEndCheck, so it is only
created when there's an actual check.

show more ...

[SCEVExpand] Only create required predicate checks.

Currently generateOverflowCheck always creates code for Step being
negative and positive, followed by a select at the end depending on
Step's sign

[SCEVExpand] Only create required predicate checks.

Currently generateOverflowCheck always creates code for Step being
negative and positive, followed by a select at the end depending on
Step's sign.

This patch updates the code to only create either the checks for step
being positive or negative, if the sign is known.

Follow-up to D116696.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D116747

show more ...

[SCEVExpand] Do not create redundant 'or false' for pred expansion.

This patch updates SCEVExpander::expandUnionPredicate to not create
redundant 'or false, x' instructions. While those are triviall

[SCEVExpand] Do not create redundant 'or false' for pred expansion.

This patch updates SCEVExpander::expandUnionPredicate to not create
redundant 'or false, x' instructions. While those are trivially
foldable, they can be easily avoided and hinder code that checks the
size/cost of the generated checks before further folds.

I am planning on look into a few other similar improvements to code
generated by SCEVExpander.

I remember a while ago @lebedev.ri working on doing some trivial folds
like that in IRBuilder itself, but there where concerns that such
changes may subtly break existing code.

Reviewed By: reames, lebedev.ri

Differential Revision: https://reviews.llvm.org/D116696

show more ...

Revert rest of `IRBuilderBase`'s short-circuiting folds

Upon further investigation and discussion,
this is actually the opposite direction from what we should be taking,
and this direction wouldn't

Revert rest of `IRBuilderBase`'s short-circuiting folds

Upon further investigation and discussion,
this is actually the opposite direction from what we should be taking,
and this direction wouldn't solve the motivational problem anyway.

Additionally, some more (polly) tests have escaped being updated.
So, let's just take a step back here.

This reverts commit f3190dedeef9da2109ea57e4cb372f295ff53b88.
This reverts commit 749581d21f2b3f53e4fca4eb8728c942d646893b.
This reverts commit f3df87d57e096143670e0fd396e81d43393a2dd2.
This reverts commit ab1dbcecd6f0969976fafd62af34730436ad5944.

show more ...

[IR] `IRBuilderBase::CreateOr()`: fix short-circuiting for constant on LHS

There is no guarantee that the constant is on RHS here,
we have to handle both cases.

Refs. https://reviews.llvm.org/D1093

[IR] `IRBuilderBase::CreateOr()`: fix short-circuiting for constant on LHS

There is no guarantee that the constant is on RHS here,
we have to handle both cases.

Refs. https://reviews.llvm.org/D109368#3089809

show more ...