[IR] Convert from nocapture to captures(none) (#123181)

This PR removes the old `nocapture` attribute, replacing it with the new
`captures` attribute introduced in #116990. This change is
intended

[IR] Convert from nocapture to captures(none) (#123181)

This PR removes the old `nocapture` attribute, replacing it with the new
`captures` attribute introduced in #116990. This change is
intended to be essentially NFC, replacing existing uses of `nocapture`
with `captures(none)` without adding any new analysis capabilities.
Making use of non-`none` values is left for a followup.

Some notes:
* `nocapture` will be upgraded to `captures(none)` by the bitcode
reader.
* `nocapture` will also be upgraded by the textual IR reader. This is to
make it easier to use old IR files and somewhat reduce the test churn in
this PR.
* Helper APIs like `doesNotCapture()` will check for `captures(none)`.
* MLIR import will convert `captures(none)` into an `llvm.nocapture`
attribute. The representation in the LLVM IR dialect should be updated
separately.

show more ...

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

[LV] Use SCEV to check if minimum iteration check is known. (#111310)

Use SCEV to check if the minimum iteration check (TC < Step) is known to
be false.

This is a first step towards addressing

[LV] Use SCEV to check if minimum iteration check is known. (#111310)

Use SCEV to check if the minimum iteration check (TC < Step) is known to
be false.

This is a first step towards addressing
https://github.com/llvm/llvm-project/issues/111098. To catch the exact
case from the issue, we need to do extra work to make sure the wrap
flags on the shl are preserved and used by SCEV.

Note that skeleton creation will be gradually moved to VPlan and this
simplification should be done as VPlan transform eventually. The current
plan is to move skeleton creation to VPlan starting from parts closest
to the parts already created by VPlan, starting with induction resume
value creation (started with
https://github.com/llvm/llvm-project/pull/110577), then memory and SCEV
checks and finally minimum iteration checks.

PR: https://github.com/llvm/llvm-project/pull/111310

show more ...

LV/test: improve a couple of tests, regen with UTC (#107225)

Add noalias, where applicable, to eliminate unnecessary memory check,
and regen with UTC.

[LLVM][VPlan] Pick more optimal initial value for VPBlend. (#104019)

By choosing an initial value whose mask is only used by the blend we can
remove the need for the mask entirely.

[VPlan] Remove unused first mask op from VPBlendRecipe. (#87770)

VPBlendRecipe does not use the first mask operand. Removing it allows
VPlan-based DCE to remove unused mask computations.

This al

[VPlan] Remove unused first mask op from VPBlendRecipe. (#87770)

VPBlendRecipe does not use the first mask operand. Removing it allows
VPlan-based DCE to remove unused mask computations.

This also fixes #87410, where unused Not VPInstructions are considered
having only their first lane demanded, but some of their operands
providing a vector value due to other users.

Fixes https://github.com/llvm/llvm-project/issues/87410

PR: https://github.com/llvm/llvm-project/pull/87770

show more ...

[LV] Create block in mask up-front if needed. (#76635)

At the moment, block and edge masks are created on demand, which means
that they are inserted at the point where they are demanded and then
c

[LV] Create block in mask up-front if needed. (#76635)

At the moment, block and edge masks are created on demand, which means
that they are inserted at the point where they are demanded and then
cached. It is possible that the mask for a block is looked up later at a
point that's not dominated by the point where the mask has been
inserted.

To avoid this, create masks up front on entry to the corresponding basic
block and leave it to VPlan simplification to remove unneeded masks.

Note that we need to create masks for all blocks, if any of the blocks
in the loop needs predication, as computing the mask of a block depends
on the masks of its predecessor.

Needed for #76090.

https://github.com/llvm/llvm-project/pull/76635

show more ...

[LoopVectorize] Enable integer Mul and Add as select reduction patterns

This patch vectorizes Phi node loop reductions for select's whos condition
comes from a floating-point comparison, with its op

[LoopVectorize] Enable integer Mul and Add as select reduction patterns

This patch vectorizes Phi node loop reductions for select's whos condition
comes from a floating-point comparison, with its operands being integers
for Add, Sub, and Mul reductions.

Example:

int foo(float *x, int n) {
int sum = 0;
for (int i=0; i<n; ++i) {
float elem = x[i];
if (elem > 0) {
sum += 2;
}
}
return sum;
}

This would previously fail to vectorize due to the integer reduction.

show more ...

Revert "[LoopVectorize] Enable integer Mul and Add as select reduction patterns"

This reverts commit f90103851f9a381bbf7ed6da250217577afd00d2.

[LoopVectorize] Enable integer Mul and Add as select reduction patterns

This patch vectorizes Phi node loop reductions for select's whos condition
comes from a floating-point comparison, with its op

[LoopVectorize] Enable integer Mul and Add as select reduction patterns

This patch vectorizes Phi node loop reductions for select's whos condition
comes from a floating-point comparison, with its operands being integers
for Add, Sub, and Mul reductions.

Example:

int foo(float *x, int n) {
int sum = 0;
for (int i=0; i<n; ++i) {
float elem = x[i];
if (elem > 0) {
sum += 2;
}
}
return sum;
}

Differential Revision: https://reviews.llvm.org/D141842

show more ...

[LoopVectorize] Convert some tests to opaque pointers (NFC)

[NFC] Port all runlines for LoopVectorize pass tests to -passes syntax

[LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask

This patch fixes pr48832 by correctly generating the mask when a poison value is involved.

Consider this CFG (whic

[LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask

This patch fixes pr48832 by correctly generating the mask when a poison value is involved.

Consider this CFG (which is a part of the input):

```
for.body: ; preds = %for.cond
br i1 true, label %cond.false, label %land.rhs

land.rhs: ; preds = %for.body
br i1 poison, label %cond.end, label %cond.false

cond.false: ; preds = %for.body, %land.rhs
br label %cond.end

cond.end: ; preds = %land.rhs, %cond.false
%cond = phi i32 [ 0, %cond.false ], [ 1, %land.rhs ]

```

The path for.body -> land.rhs -> cond.end should be taken when 'select i1 false, i1 poison, i1 false' holds (which means it's never taken); but VPRecipeBuilder::createEdgeMask was emitting 'and i1 false, poison' instead.
The former one successfully blocks poison propagation whereas the latter one doesn't, making the condition poison and thus causing the miscompilation.

SimplifyCFG has a similar bug (which didn't expose a real-world bug yet), and a patch for this is also ongoing (see https://reviews.llvm.org/D95026).

Reviewed By: bjope

Differential Revision: https://reviews.llvm.org/D95217

show more ...

[Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics

The vector reduction intrinsics started life as experimental ops, so backend support
was lacking. As part of promot

[Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics

The vector reduction intrinsics started life as experimental ops, so backend support
was lacking. As part of promoting them to 1st-class intrinsics, however, codegen
support was added/improved:
D58015
D90247

So I think it is safe to now remove this complication from IR.

Note that we still have an IR-level codegen expansion pass for these as discussed
in D95690. Removing that is another step in simplifying the logic. Also note that
x86 was already unconditionally forming reductions in IR, so there should be no
difference for x86.

I spot checked a couple of the tests here by running them through opt+llc and did
not see any asm diffs.

If we do find functional differences for other targets, it should be possible
to (at least temporarily) restore the shuffle IR with the ExpandReductions IR
pass.

Differential Revision: https://reviews.llvm.org/D96552

show more ...

Recommit #2: "[LV] Induction Variable does not remain scalar under tail-folding."

This was reverted because of a miscompilation. At closer inspection, the
problem was actually visible in a changed l

Recommit #2: "[LV] Induction Variable does not remain scalar under tail-folding."

This was reverted because of a miscompilation. At closer inspection, the
problem was actually visible in a changed llvm regression test too. This
one-line follow up fix/recommit will splat the IV, which is what we are trying
to avoid if unnecessary in general, if tail-folding is requested even if all
users are scalar instructions after vectorisation. Because with tail-folding,
the splat IV will be used by the predicate of the masked loads/stores
instructions. The previous version omitted this, which caused the
miscompilation. The original commit message was:

If tail-folding of the scalar remainder loop is applied, the primary induction
variable is splat to a vector and used by the masked load/store vector
instructions, thus the IV does not remain scalar. Because we now mark
that the IV does not remain scalar for these cases, we don't emit the vector IV
if it is not used. Thus, the vectoriser produces less dead code.

Thanks to Ayal Zaks for the direction how to fix this.

show more ...

Revert "Recommit "[LV] Induction Variable does not remain scalar under tail-folding.""

This reverts commit ae45b4dbe73ffde5fe3119835aa947d5a49635ed. It
causes miscompilations, test case on the maili

Revert "Recommit "[LV] Induction Variable does not remain scalar under tail-folding.""

This reverts commit ae45b4dbe73ffde5fe3119835aa947d5a49635ed. It
causes miscompilations, test case on the mailing list.

show more ...

Recommit "[LV] Induction Variable does not remain scalar under tail-folding."

With 3 llvm regr tests fixed/updated that I had missed.

Revert "[LV] Induction Variable does not remain scalar under tail-folding."

This reverts commit 617aa64c84146468b384453375d1d34f97eb57db.

while I investigate buildbot failures.

[LV] Induction Variable does not remain scalar under tail-folding.

If tail-folding of the scalar remainder loop is applied, the primary induction
variable is splat to a vector and used by the masked

[LV] Induction Variable does not remain scalar under tail-folding.

If tail-folding of the scalar remainder loop is applied, the primary induction
variable is splat to a vector and used by the masked load/store vector
instructions, thus the IV does not remain scalar. Because we now mark
that the IV does not remain scalar for these cases, we don't emit the vector IV
if it is not used. Thus, the vectoriser produces less dead code.

Thanks to Ayal Zaks for the direction how to fix this.

Differential Revision: https://reviews.llvm.org/D78911

show more ...

Revert "Temporarily Revert "Add basic loop fusion pass.""

The reversion apparently deleted the test/Transforms directory.

Will be re-reverting again.

llvm-svn: 358552

Add a new reduction pattern match

Adding a new reduction pattern match for vectorizing code similar
to TSVC s3111:

for (int i = 0; i < N; i++)
if (a[i] > b)
sum += a[i];

This patch adds supp

Add a new reduction pattern match

Adding a new reduction pattern match for vectorizing code similar
to TSVC s3111:

for (int i = 0; i < N; i++)
if (a[i] > b)
sum += a[i];

This patch adds support for fadd, fsub and fmull, as well as multiple
branches and different (but compatible) instructions (ex. add+sub) in
different branches.

The difference from the previous patch(https://reviews.llvm.org/D49168)
is as follows:
- Added check of fast-math property of fp-instruction to the
previous patch
- Fix/add some pattern for if-reduction.ll


Differential Revision: https://reviews.llvm.org/D54464

Patch by Takahiro Miyoshi <takahiro.miyoshi@linaro.org>
and Masakazu Ueno <masakazu.ueno@linaro.org>

llvm-svn: 347989

show more ...

[LV] Add a new reduction pattern match

Adding a new reduction pattern match for vectorizing code similar to TSVC s3111:

for (int i = 0; i < N; i++)
if (a[i] > b)
sum += a[i];

This patch adds

[LV] Add a new reduction pattern match

Adding a new reduction pattern match for vectorizing code similar to TSVC s3111:

for (int i = 0; i < N; i++)
if (a[i] > b)
sum += a[i];

This patch adds support for fadd, fsub and fmull, as well as multiple
branches and different (but compatible) instructions (ex. add+sub) in
different branches.

I have forwarded to trunk, added fsub and fmul functionality and
additional tests, but the credit goes to Takahiro, who did most of the
actual work.

Differential Revision: https://reviews.llvm.org/D49168

Patch by Takahiro Miyoshi <takahiro.miyoshi@linaro.org>.

llvm-svn: 344172

show more ...