[llvm] Fix crash caused by reprocessing complex reductions (#122077)

If a complex pattern had the shape of both a complex->complex reduction
and a complex->single reduction, the matching would recog

[llvm] Fix crash caused by reprocessing complex reductions (#122077)

If a complex pattern had the shape of both a complex->complex reduction
and a complex->single reduction, the matching would recognise both and
deem the graph a valid transformation. Preventing this reprocessing
results in only one of these matching, meaning that in the case of an
invalid graph, we don't try to transform it anyway.

show more ...

Complex deinterleaving/single reductions build fix Reapply "Add support for single reductions in ComplexDeinterleavingPass (#112875)" (#120441)

This reverts commit 76714be5fd4ace66dd9e19ce706c2e214

Complex deinterleaving/single reductions build fix Reapply "Add support for single reductions in ComplexDeinterleavingPass (#112875)" (#120441)

This reverts commit 76714be5fd4ace66dd9e19ce706c2e2149dd5716, fixing the
build failure that caused the revert.

The failure stemmed from the complex deinterleaving pass identifying a
series of add operations as a "complex to single reduction", so when it
tried to transform this erroneously identified pattern, it faulted. The
fix applied is to ensure that complex numbers (or patterns that match
them) are used throughout, by checking if there is a deinterleave node
amidst the graph.

show more ...

Revert "Add support for single reductions in ComplexDeinterleavingPass (#112875)"

This reverts commit b3eede5e1fa7ab742b86e9be22db7bccd2505b8a.

This has been breaking most AArch64 stage2 builds for

Revert "Add support for single reductions in ComplexDeinterleavingPass (#112875)"

This reverts commit b3eede5e1fa7ab742b86e9be22db7bccd2505b8a.

This has been breaking most AArch64 stage2 builds for 4+ hours,
reverting to get the bots back to green.

https://lab.llvm.org/buildbot/#/builders/41/builds/4172
https://lab.llvm.org/buildbot/#/builders/4/builds/4281
https://lab.llvm.org/buildbot/#/builders/199/builds/263
https://lab.llvm.org/buildbot/#/builders/198/builds/334
https://lab.llvm.org/buildbot/#/builders/143/builds/4276
https://lab.llvm.org/buildbot/#/builders/17/builds/4725

show more ...

Add support for single reductions in ComplexDeinterleavingPass (#112875)

The Complex Deinterleaving pass assumes that all values emitted will
result in complex numbers, this patch aims to remove th

Add support for single reductions in ComplexDeinterleavingPass (#112875)

The Complex Deinterleaving pass assumes that all values emitted will
result in complex numbers, this patch aims to remove that assumption and
adds support for emitting just the real or imaginary components, not
both.

show more ...

[CodeGen] Remove unused includes (NFC) (#115996)

Identified with misc-include-cleaner.

Move several vector intrinsics out of experimental namespace (#88748)

This patch is moving out following intrinsics:
* vector.interleave2/deinterleave2
* vector.reverse
* vector.splice

from th

Move several vector intrinsics out of experimental namespace (#88748)

This patch is moving out following intrinsics:
* vector.interleave2/deinterleave2
* vector.reverse
* vector.splice

from the experimental namespace.

All these intrinsics exist in LLVM for more than a year now, and are
widely used, so should not be considered as experimental.

show more ...

[NFC][RemoveDIs] Always use iterators for inserting PHIs

It's becoming potentially unsafe to insert a PHI instruction using a plain
Instruction pointer. Switch all the remaining sites that create an

[NFC][RemoveDIs] Always use iterators for inserting PHIs

It's becoming potentially unsafe to insert a PHI instruction using a plain
Instruction pointer. Switch all the remaining sites that create and insert
PHIs to use iterators instead. For example, the code in
ComplexDeinterleavingPass.cpp is definitely at-risk of mixing PHIs and
debug-info.

show more ...

[ComplexDeinterleaving] Use MapVector to fix codegen non-determinism.

[CodeGen] Fix incorrect insertion point selection for reduction nodes in ComplexDeinterleavingPass

When replacing ComplexDeinterleavingPass::ReductionOperation, we can do it
either from the Real or

[CodeGen] Fix incorrect insertion point selection for reduction nodes in ComplexDeinterleavingPass

When replacing ComplexDeinterleavingPass::ReductionOperation, we can do it
either from the Real or Imaginary part. The correct way is to take whichever
is later in the BasicBlock, but before the patch, we just always took the
Real part.

Fixes https://github.com/llvm/llvm-project/issues/65044

Differential Revision: https://reviews.llvm.org/D159209

show more ...

[CodeGen] Improve speed of ComplexDeinterleaving pass

Cache all results of running `identifyNode`, even those that do not identify
potential complex operations. This patch prevents ComplexDeinterlea

[CodeGen] Improve speed of ComplexDeinterleaving pass

Cache all results of running `identifyNode`, even those that do not identify
potential complex operations. This patch prevents ComplexDeinterleaving pass
from repeatedly trying to identify Nodes for the same pair of instructions.

Fixes https://github.com/llvm/llvm-project/issues/64379

Differential Revision: https://reviews.llvm.org/D156916

show more ...

[CodeGen] Extend ComplexDeinterleaving pass to recognise patterns using integer types

AArch64 introduced CMLA and CADD instructions as part of SVE2. This
change allows to generate such instructions

[CodeGen] Extend ComplexDeinterleaving pass to recognise patterns using integer types

AArch64 introduced CMLA and CADD instructions as part of SVE2. This
change allows to generate such instructions when this architecture
feature is available.

Differential Revision: https://reviews.llvm.org/D153808

show more ...

[CodeGen] Fix incorrectly detected reduction bug in ComplexDeinterleaving pass

Using ACLE intrinsics, it is possible to create a loop that the
deinterleaving pass incorrectly classified as a reducti

[CodeGen] Fix incorrectly detected reduction bug in ComplexDeinterleaving pass

Using ACLE intrinsics, it is possible to create a loop that the
deinterleaving pass incorrectly classified as a reduction loop.
For example, for fixed-width vectors the loop was like below:

vector.body:
%a = phi <4 x float> [ %init.a, %entry ], [ %updated.a, %vector.body ]
%b = phi <4 x float> [ %init.b, %entry ], [ %updated.b, %vector.body ]
...
; Does not depend on %a or %b:
%updated.a = ...
%updated.b = ...

Differential Revision: https://reviews.llvm.org/D154598

show more ...

[CodeGen] Add support for Splats in ComplexDeinterleaving pass

This commit allows generating of complex number intrinsics for expressions
with constants or loops invariants, which are represented as

[CodeGen] Add support for Splats in ComplexDeinterleaving pass

This commit allows generating of complex number intrinsics for expressions
with constants or loops invariants, which are represented as splats.
For instance, after vectorizing loops in the following code snippets,
the ComplexDeinterleaving pass will be able to generate complex number
intrinsics:

```
complex<> x = ...;
for (int i = 0; i < N; ++i)
c[i] = a[i] * b[i] * x;
```

or

```
for (int i = 0; i < N; ++i)
c[i] = a[i] * b[i] * (11.0 + 3.0i);
```

Differential Revision: https://reviews.llvm.org/D153355

show more ...

[CodeGen] Refactor ComplexDeinterleaving to run identification on Values instead of Instructions

This change will make it easier to add identification of complex constants
in future patches.

Differ

[CodeGen] Refactor ComplexDeinterleaving to run identification on Values instead of Instructions

This change will make it easier to add identification of complex constants
in future patches.

Differential Revision: https://reviews.llvm.org/D153446

show more ...

[NFC]Fix possibly derefer nullptr in ComplexDeinterleavingPass.cpp

Fix static analyzer reports issue, add assert to avoid analyzer report.

Reviewed By: igor.kirillov

Differential Revision: https:/

[NFC]Fix possibly derefer nullptr in ComplexDeinterleavingPass.cpp

Fix static analyzer reports issue, add assert to avoid analyzer report.

Reviewed By: igor.kirillov

Differential Revision: https://reviews.llvm.org/D153942

show more ...

Fix the ComplexDeinterleaving bug when handling mixed reductions.

Add a missing check that ensures that ComplexDeinterleaving for reduction
is only analyzed for Real and Imaginary Instructions of th

Fix the ComplexDeinterleaving bug when handling mixed reductions.

Add a missing check that ensures that ComplexDeinterleaving for reduction
is only analyzed for Real and Imaginary Instructions of the same type.

Differential Revision: https://reviews.llvm.org/D153862

show more ...

Revert "Revert "[CodeGen] Extend reduction support in ComplexDeinterleaving pass to support predication""

Adds the capability to recognize SelectInst that appear in the IR.
These instructions are ge

Revert "Revert "[CodeGen] Extend reduction support in ComplexDeinterleaving pass to support predication""

Adds the capability to recognize SelectInst that appear in the IR.
These instructions are generated during scalable vectorization for reduction
and when the code contains conditions inside the loop body or when
"-prefer-predicate-over-epilogue=predicate-dont-vectorize" is set.

Differential Revision: https://reviews.llvm.org/D152558

This reverts commit ab09654832dba5cef8baa6400fdfd3e4d1495624.

Reason: Reapplying after removing unnecessary default case in switch expression.

show more ...

Revert "[CodeGen] Extend reduction support in ComplexDeinterleaving pass to support predication"

ComplexDeinterleavingPass.cpp:1849:3: error: default label in switch which covers all enumeration val

Revert "[CodeGen] Extend reduction support in ComplexDeinterleaving pass to support predication"

ComplexDeinterleavingPass.cpp:1849:3: error: default label in switch which covers all enumeration values

This reverts commit 116953b82130df1ebd817b3587b16154f659c013.

show more ...

[CodeGen] Extend reduction support in ComplexDeinterleaving pass to support predication

Adds the capability to recognize SelectInst that appear in the IR.
These instructions are generated during sca

[CodeGen] Extend reduction support in ComplexDeinterleaving pass to support predication

Adds the capability to recognize SelectInst that appear in the IR.
These instructions are generated during scalable vectorization for reduction
and when the code contains conditions inside the loop body or when
"-prefer-predicate-over-epilogue=predicate-dont-vectorize" is set.

Differential Revision: https://reviews.llvm.org/D152558

show more ...

[CodeGen] Fix a warning

This patch fixes:

llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp:1790:3: error:
default label in switch which covers all enumeration values
[-Werror,-Wcovered-switch-d

[CodeGen] Fix a warning

This patch fixes:

llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp:1790:3: error:
default label in switch which covers all enumeration values
[-Werror,-Wcovered-switch-default]

show more ...

[CodeGen] Add support for reductions in ComplexDeinterleaving pass

This commit enhances the ComplexDeinterleaving pass to handle unordered
reductions in simple one-block vectorized loops, supporting

[CodeGen] Add support for reductions in ComplexDeinterleaving pass

This commit enhances the ComplexDeinterleaving pass to handle unordered
reductions in simple one-block vectorized loops, supporting both
SVE and Neon architectures.

Differential Revision: https://reviews.llvm.org/D152022

show more ...

[CodeGen] Improve handling -Ofast generated code by ComplexDeinterleaving pass

Code generated with -Ofast and -O3 -ffp-contract=fast (add
-ffinite-math-only to enable vectorization) can differ signi

[CodeGen] Improve handling -Ofast generated code by ComplexDeinterleaving pass

Code generated with -Ofast and -O3 -ffp-contract=fast (add
-ffinite-math-only to enable vectorization) can differ significantly.
Code compiled with -O3 can be deinterleaved using patterns as the
instruction order is preserved. However, with the -Ofast flag, there
can be multiple changes in the computation sequence, and even the real
and imaginary parts may not be calculated in parallel.
For more details, refer to
llvm/test/CodeGen/AArch64/complex-deinterleaving-*-fast.ll and
llvm/test/CodeGen/AArch64/complex-deinterleaving-*-contract.ll tests.
This patch implements a more general approach and enables handling most
-Ofast cases.

Differential Revision: https://reviews.llvm.org/D148558

show more ...

[CodeGen] Refactor IR generation functions to use IRBuilder in ComplexDeinterleaving pass

This patch updates several functions in LLVM's IR generation code to accept
an IRBuilder object as an argume

[CodeGen] Refactor IR generation functions to use IRBuilder in ComplexDeinterleaving pass

This patch updates several functions in LLVM's IR generation code to accept
an IRBuilder object as an argument, rather than an Instruction that indicates
the insertion point for new instructions.
This change is necessary to handle sophisticated -Ofast optimization cases
from D148558 where it's unclear which instructions should be used as the
insertion point for new operations.

Differential Revision: https://reviews.llvm.org/D148703

show more ...

[llvm] Reduce ComplexDeinterleavingPass.h includes

Remove the unnecessary `"llvm/IR/PatternMatch.h"` include directive from
`ComplexDeinterleavingPass.h` and move it to the corresponding source
file

[llvm] Reduce ComplexDeinterleavingPass.h includes

Remove the unnecessary `"llvm/IR/PatternMatch.h"` include directive from
`ComplexDeinterleavingPass.h` and move it to the corresponding source
file.

Add missing includes that were transitively included by this header to 3
other source files.

This reduces the total number of preprocessing tokens across the LLVM
source files in `lib` from (roughly) 1,964,876,961 to 1,935,091,611 - a
reduction of ~1.52%. This should result in a small improvement in
compilation time.

show more ...

Revert "[llvm] Reduce ComplexDeinterleavingPass.h includes"

This reverts commit 058ca5c07106d38ad66e3ec4972a613a64e88151.