[IA] Generalize the support for power-of-two (de)interleave intrinsics (#123863)

Previously, AArch64 used pattern matching to support
llvm.vector.(de)interleave of 2 and 4; RISC-V only supported
(

[IA] Generalize the support for power-of-two (de)interleave intrinsics (#123863)

Previously, AArch64 used pattern matching to support
llvm.vector.(de)interleave of 2 and 4; RISC-V only supported
(de)interleave of 2.

This patch consolidates the logics in these two targets by factoring out
the common factor calculations into the InterleaveAccess Pass.

show more ...

[InterleavedAccessPass]: Ensure that dead nodes get erased only once (#122643)

Use SmallSetVector instead of SmallVector to avoid duplication,
so that dead nodes get erased/deleted only once.

[CodeGen] Remove unused includes (NFC) (#115996)

Identified with misc-include-cleaner.

[InterleavedAccess] Use SmallVectorImpl references. NFC

Instead of repeating SmallVector size in multiple places.

[IA][AArch64]: Construct (de)interleave4 out of (de)interleave2 (#89276)

- [AArch64]: TargetLowering is updated to spot load/store (de)interleave4 like sequences using PatternMatch,
and emit equ

[IA][AArch64]: Construct (de)interleave4 out of (de)interleave2 (#89276)

- [AArch64]: TargetLowering is updated to spot load/store (de)interleave4 like sequences using PatternMatch,
and emit equivalent sve.ld4 and sve.st4 intrinsics.

show more ...

Move several vector intrinsics out of experimental namespace (#88748)

This patch is moving out following intrinsics:
* vector.interleave2/deinterleave2
* vector.reverse
* vector.splice

from th

Move several vector intrinsics out of experimental namespace (#88748)

This patch is moving out following intrinsics:
* vector.interleave2/deinterleave2
* vector.reverse
* vector.splice

from the experimental namespace.

All these intrinsics exist in LLVM for more than a year now, and are
widely used, so should not be considered as experimental.

show more ...

[AArch64] Add costs for LD3/LD4 shuffles.

Similar to #87934, this adds costs to the shuffles in a canonical LD3/LD4
pattern, which are represented in LLVM as deinterleaving-shuffle(load). This
likel

[AArch64] Add costs for LD3/LD4 shuffles.

Similar to #87934, this adds costs to the shuffles in a canonical LD3/LD4
pattern, which are represented in LLVM as deinterleaving-shuffle(load). This
likely has less effect at the moment than the ST3/ST4 costs as instcombine will
perform certain transforms without considering the cost.

show more ...

[NFC][RemoveDIs] Use iterators for insertion at various call-sites (#84736)

These are the last remaining "trivial" changes to passes that use
Instruction pointers for insertion. All of this should

[NFC][RemoveDIs] Use iterators for insertion at various call-sites (#84736)

These are the last remaining "trivial" changes to passes that use
Instruction pointers for insertion. All of this should be NFC, it's just
changing the spelling of how we identify a position.

In one or two locations, I'm also switching uses of getNextNode etc to
using std::next with iterators. This too should be NFC.

---------

Merged by: Stephen Tozer <stephen.tozer@sony.com>

show more ...

[CodeGen] Port `InterleavedAccess` to new pass manager (#74904)

[InterleavedAccessPass] Avoid optimizing load instructions if it has dead binop users (#71339)

If a load instruction qualifies to be optimized by InterleavedAccess
Pass, but also has a dead binop i

[InterleavedAccessPass] Avoid optimizing load instructions if it has dead binop users (#71339)

If a load instruction qualifies to be optimized by InterleavedAccess
Pass, but also has a dead binop instruction, this will lead to a crash.

Binop instruction will not be deleted, because normally it would be
deleted through its' users, but it has none. Later on deleting a load
instruction will fail because it still has uses.

show more ...

[AArch64][CodeGen] Lower (de)interleave2 intrinsics to ld2/st2

The InterleavedAccess pass currently matches (de)interleaving
shufflevector instructions with loads or stores, and calls into
target lo

[AArch64][CodeGen] Lower (de)interleave2 intrinsics to ld2/st2

The InterleavedAccess pass currently matches (de)interleaving
shufflevector instructions with loads or stores, and calls into
target lowering to generate ldN or stN instructions.

Since we can't use shufflevector for scalable vectors (besides a
splat with zeroinitializer), we have interleave2 and deinterleave2
intrinsics. This patch extends InterleavedAccess to recognize those
intrinsics and if possible replace them with ld2/st2.

Reviewed By: paulwalker-arm

Differential Revision: https://reviews.llvm.org/D146218

show more ...

Fix uninitialized scalar members in CodeGen

This change fixes some static code analysis warnings.

Reviewed By: LuoYuanke

Differential Revision: https://reviews.llvm.org/D148811

[InterleaveAccess] Check that binop shuffles have an undef second operand

It is expected that shuffles that we hoist through binops only have a single
vector operand, the other being undef/poison. T

[InterleaveAccess] Check that binop shuffles have an undef second operand

It is expected that shuffles that we hoist through binops only have a single
vector operand, the other being undef/poison. The checks for
isDeInterleaveMaskOfFactor check that all the elements come from inside the
first vector, but with non-canonical shuffles the second operand could still
have a value. Add a quick check to make sure it is UndefValue as expected, to
make sure we don't run into problems with BinOpShuffles not using BinOps.

Fixes #61749

Differential Revision: https://reviews.llvm.org/D147306

show more ...

[RISCV][NFC] Share interleave mask checking logic

This adds two new methods to ShuffleVectorInst, isInterleave and
isInterleaveMask, so that the logic to check if a shuffle mask is an
interleave can

[RISCV][NFC] Share interleave mask checking logic

This adds two new methods to ShuffleVectorInst, isInterleave and
isInterleaveMask, so that the logic to check if a shuffle mask is an
interleave can be shared across the TTI, codegen and the interleaved
access pass.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D145971

show more ...

[CodeGen] Qualify auto variables in for loops (NFC)

[InterleaveAccessPass] Handle multi-use binop shuffles

D89489 added some logic to the interleaved access pass to attempt to
undo the folding of shuffles into binops, that instcombine performs. If
ea

[InterleaveAccessPass] Handle multi-use binop shuffles

D89489 added some logic to the interleaved access pass to attempt to
undo the folding of shuffles into binops, that instcombine performs. If
early-cse is run too, the binops may be commoned into a single operation
with multiple shuffle uses. It is still profitable reverse the transform
though, so long as all the uses are shuffles.

Differential Revision: https://reviews.llvm.org/D129419

show more ...

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-in

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681

show more ...

Cleanup includes: DebugInfo & CodeGen

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121332

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https:/

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169

show more ...

Cleanup codegen includes

after: 1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169

Mark CFG as preserved in TypePromotion and InterleaveAccess passes

Neither of these passes modify the CFG, allowing us to preserve DomTree
and LoopInfo across them by using setPreservesCFG.

Differe

Mark CFG as preserved in TypePromotion and InterleaveAccess passes

Neither of these passes modify the CFG, allowing us to preserve DomTree
and LoopInfo across them by using setPreservesCFG.

Differential Revision: https://reviews.llvm.org/D110161

show more ...

[InterleaveAccess] Copy fast math flags when adjusting binary operators in interleave access pass

The Interleave Access pass will convert shuffle(binop(load, load)) to
binop(shuffle(load), shuffle(l

[InterleaveAccess] Copy fast math flags when adjusting binary operators in interleave access pass

The Interleave Access pass will convert shuffle(binop(load, load)) to
binop(shuffle(load), shuffle(load)), in order to create more
interleaving load patterns (VLD2/3/4) that might have been messed up by
instcombine. As shown in D104247 we were missing copying IR flags to the
new instruction though, which should just be kept the same as the
original instruction.

Differential Revision: https://reviews.llvm.org/D104255

show more ...

[llvm] Use append_range (NFC)

[CodeGen] Update transformations to use poison for shufflevector/insertelem's initial vector elem

This patch is a part of D93817 and makes transformations in CodeGen use poison for shufflevector/ins

[CodeGen] Update transformations to use poison for shufflevector/insertelem's initial vector elem

This patch is a part of D93817 and makes transformations in CodeGen use poison for shufflevector/insertelem's initial vector element.

The change in CodeGenPrepare.cpp is fine because the mask of shufflevector should be always zero.
It doesn't touch the second element (which is poison).

The change in InterleavedAccessPass.cpp is also fine becauses the mask is of the form <a, a+m, a+2m, .., a+km> where a+km is smaller than
the size of the first vector operand.
This is guaranteed by the caller of replaceBinOpShuffles, which is lowerInterleavedLoad.
It calls isDeInterleaveMask and isDeInterleaveMaskOfFactor to check the mask is the desirable form.
isDeInterleaveMask has the check that a+km is smaller than the vector size.
To check my understanding, I added an assertion & added a test to show that this optimization doesn't fire in such case.

Reviewed By: spatel

Differential Revision: https://reviews.llvm.org/D94056

show more ...

[InterleavedAccess] Return correct 'modified' status.

Both tryReplaceExtracts and replaceBinOpShuffles may modify the IR, even
if no interleaved loads are generated, but currently the pass pretends

[InterleavedAccess] Return correct 'modified' status.

Both tryReplaceExtracts and replaceBinOpShuffles may modify the IR, even
if no interleaved loads are generated, but currently the pass pretends
no changes were made.

This patch updates the pass to return true if either of the functions
made any changes. In case of tryReplaceExtracts, changes are made if
there are any Extracts and true is returned.

`replaceBinOpShuffles` always makes changes if BinOpShuffles is not empty.
It also always returned true, so I went ahead and change it to just
`replaceBinOpShuffles`.

Fixes PR48208.

Reviewed By: SjoerdMeijer

Differential Revision: https://reviews.llvm.org/D93997

show more ...