Fix typo "tranpose" (#124929)

[mlir][vector] Disable `BreakDownVectorBitCast` for scalable vectors (#122725)

`BreakDownVectorBitCast` leverages
* `vector.extract_strided_slices` + `vector.insert_strided_slices`

As these Op

[mlir][vector] Disable `BreakDownVectorBitCast` for scalable vectors (#122725)

`BreakDownVectorBitCast` leverages
* `vector.extract_strided_slices` + `vector.insert_strided_slices`

As these Ops do not support extracting scalable sub-vectors (i.e.
extracting/inserting a fraction of a scalable dim), it's best to bail
out.

show more ...

[mlir][NFC] Avoid using braced initializer lists to call a constructor. (#123714)

In the LLVM style guide, we prefer not using braced initializer lists to
call a constructor. Also, we prefer using

[mlir][NFC] Avoid using braced initializer lists to call a constructor. (#123714)

In the LLVM style guide, we prefer not using braced initializer lists to
call a constructor. Also, we prefer using an equal before the open curly
brace if we use a braced initializer list when initializing a variable.

See

https://llvm.org/docs/CodingStandards.html#do-not-use-braced-initializer-lists-to-call-a-constructor
for more details.

The style guide does not explain the reason well. There is an article
from abseil, which mentions few benefits. E.g., we can avoid the most
vexing parse, etc. See https://abseil.io/tips/88 for more details.

Signed-off-by: hanhanW <hanhan0912@gmail.com>

show more ...

[mlir][IR][NFC] Move free-standing functions to `MemRefType` (#123465)

Turn free-standing `MemRefType`-related helper functions in
`BuiltinTypes.h` into member functions.

[MemRef] Migrate away from PointerUnion::{is,get} (NFC) (#120382)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dy

[MemRef] Migrate away from PointerUnion::{is,get} (NFC) (#120382)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[MLIR] Fix `BubbleDownVectorBitCastForExtract` crash on non-static index (#116518)

Previously the patch was not expecting to handle non-static index, when
the index is a non constant value it will

[MLIR] Fix `BubbleDownVectorBitCastForExtract` crash on non-static index (#116518)

Previously the patch was not expecting to handle non-static index, when
the index is a non constant value it will crash.

This patch is to make sure it return gracefully instead of crashing.

show more ...

[mlir][vector] Add a new TD Op for patterns leveraging ShapeCastOp (#110525)

Adds a new Transform Dialect Op that collects patters for dropping unit
dims from various Ops:
* `transform.apply_pat

[mlir][vector] Add a new TD Op for patterns leveraging ShapeCastOp (#110525)

Adds a new Transform Dialect Op that collects patters for dropping unit
dims from various Ops:
* `transform.apply_patterns.vector.drop_unit_dims_with_shape_cast`.

It excludes patterns for vector.transfer Ops - these are collected
under:
* `apply_patterns.vector.rank_reducing_subview_patterns`,

and use ShapeCastOp _and_ SubviewOp to reduce the rank (and to eliminate
unit dims).

This new TD Ops allows us to test the "ShapeCast folder" pattern in
isolation. I've extracted the only test that I could find for that
folder from "vector-transforms.mlir" and moved it to a dedicated file:
"shape-cast-folder.mlir". I also added a test case with scalable
vectors.

Changes in VectorTransforms.cpp are not needed (added a comment with
a TODO + ordered the patterns alphabetically). I am Including them here
to avoid a separate PR.

show more ...

[mlir][vector] Add pattern for dropping unit dims from for loops (#109585)

This adds a pattern for dropping unit dims from the iter_args of scf.for
ops using vector.shape_cast. This composes with t

[mlir][vector] Add pattern for dropping unit dims from for loops (#109585)

This adds a pattern for dropping unit dims from the iter_args of scf.for
ops using vector.shape_cast. This composes with the other patterns for
dropping unit dims from elementwise ops and transposes.

show more ...

[mlir][vector] Group re-order patterns together (#102856)

Group all patterns that re-order vector.transpose and vector.broadcast
Ops (*) under `populateSinkVectorOpsPatterns`. These patterns are
n

[mlir][vector] Group re-order patterns together (#102856)

Group all patterns that re-order vector.transpose and vector.broadcast
Ops (*) under `populateSinkVectorOpsPatterns`. These patterns are
normally used to "sink" redundant Vector Ops, hence grouping together.
Example:

```mlir
%at = vector.transpose %a, [1, 0]: vector<4x2xf32> to vector<2x4xf32>
%bt = vector.transpose %b, [1, 0]: vector<4x2xf32> to vector<2x4xf32>
%r = arith.addf %at, %bt : vector<2x4xf32>
```
would get converted to:
```mlir
%0 = arith.addf %a, %b : vector<4x2xf32>
%r = vector.transpose %0, [1, 0] : vector<2x4xf32>
```

This patch also moves all tests for these patterns so that all of them
are:
* run under one test-flag: `test-vector-sink-patterns`,
* located in one file: "vector-sink.mlir".

To facilitate this change:
* `-test-sink-vector-broadcast` is renamed as
`test-vector-sink-patterns`,
* "sink-vector-broadcast.mlir" is renamed as "vector-sink.mlir",
* tests for `ReorderCastOpsOnBroadcast` and
`ReorderElementwiseOpsOnTranspose` patterns are moved from
"vector-reduce-to-contract.mlir" to "vector-sink.mlir",
* `ReorderElementwiseOpsOnTranspose` patterns are removed from
`populateVectorReductionToContractPatterns` and added to (newly
created) `populateSinkVectorOpsPatterns`,
* `ReorderCastOpsOnBroadcast` patterns are removed from
`populateVectorReductionToContractPatterns` - these are already
present in `populateSinkVectorOpsPatterns`.

This should allow us better layering and more straightforward testing.
For the latter, the goal is to be able to easily identify which pattern
a particular test is exercising (especially when it's a specific
pattern).

NOTES FOR DOWNSTREAM USERS

In order to preserve the current functionality, please make sure to add
* `populateSinkVectorOpsPatterns`,

wherever you are using `populateVectorReductionToContractPatterns`.
Also, rename `populateSinkVectorBroadcastPatterns` as
`populateSinkVectorOpsPatterns`.

(*) I didn't notice any other re-order patterns.

show more ...

[mlir][vector] Add tests for `populateSinkVectorBroadcastPatterns` (1/n) (#102286)

Adds tests for scalable vectors in:
* sink-vector-broadcast.mlir

This test file excercises patterns grouped u

[mlir][vector] Add tests for `populateSinkVectorBroadcastPatterns` (1/n) (#102286)

Adds tests for scalable vectors in:
* sink-vector-broadcast.mlir

This test file excercises patterns grouped under
`populateSinkVectorBroadcastPatterns`, which includes:
* `ReorderElementwiseOpsOnBroadcast`,
* `ReorderCastOpsOnBroadcast`.

Right now there are only tests for the former. However, I've noticed
that "vector-reduce-to-contract.mlir" contains tests for the latter and
I've left a few TODOs to group these tests back together in one file.

Additionally, added some helpful `notifyMatchFailure` messages in
`ReorderElementwiseOpsOnBroadcast`.

show more ...

[mlir][vector] Handle corner cases in DropUnitDimsFromTransposeOp. (#102518)

https://github.com/llvm/llvm-project/commit/da8778e499d8049ac68c2e152941a38ff2bc9fb2
breaks the lowering of vector.trans

[mlir][vector] Handle corner cases in DropUnitDimsFromTransposeOp. (#102518)

https://github.com/llvm/llvm-project/commit/da8778e499d8049ac68c2e152941a38ff2bc9fb2
breaks the lowering of vector.transpose that all the dimensions are unit
dimensions. The revision fixes the issue and adds a test.

---------

Signed-off-by: hanhanW <hanhan0912@gmail.com>

show more ...

[mlir][vector] Fix return of `DropUnitDimsFromTransposeOp` pattern (#102478)

This accidentally returned `failure()` (rather than `success()`) when it
applied.

[mlir][vector] Add pattern to drop unit dims from vector.transpose (#102017)

Example:

BEFORE:
```mlir
%transpose = vector.transpose %vector, [3, 0, 1, 2]
: vector<1x1x4x[4]xf32> to vector<[4

[mlir][vector] Add pattern to drop unit dims from vector.transpose (#102017)

Example:

BEFORE:
```mlir
%transpose = vector.transpose %vector, [3, 0, 1, 2]
: vector<1x1x4x[4]xf32> to vector<[4]x1x1x4xf32>
```

AFTER:
```mlir
%dropDims = vector.shape_cast %vector
: vector<1x1x4x[4]xf32> to vector<4x[4]xf32>
%transpose = vector.transpose %0, [1, 0]
: vector<4x[4]xf32> to vector<[4]x4xf32>
%restoreDims = vector.shape_cast %transpose
: vector<[4]x4xf32> to vector<[4]x1x1x4xf32>
```

show more ...

[mlir][vector][nfc] Simplify `in_bounds` attr update (#100334)

Since the `in_bounds` attribute is mandatory, there's no need for logic
like this (`readOp.getInBounds()` is guaranteed to return a no

[mlir][vector][nfc] Simplify `in_bounds` attr update (#100334)

Since the `in_bounds` attribute is mandatory, there's no need for logic
like this (`readOp.getInBounds()` is guaranteed to return a non-empty
ArrayRef):

```cpp
ArrayAttr inBoundsAttr = readOp.getInBounds()
? rewriter.getArrayAttr( readOp.getInBoundsAttr().getValue().drop_back(dimsToDrop))
: ArrayAttr();
```

Instead, we can do this:
```cpp
ArrayAttr inBoundsAttr = rewriter.getArrayAttr(
readOp.getInBoundsAttr().getValue().drop_back(dimsToDrop));
```

This is a small follow-up for #97049 - this change should've been
included there.

show more ...

[MLIR][Vector] Generalize DropUnitDimFromElementwiseOps to non leading / trailing dimensions. (#98455)

Generalizes DropUnitDimFromElementwiseOps to support inner unit
dimensions.
This change stems

[MLIR][Vector] Generalize DropUnitDimFromElementwiseOps to non leading / trailing dimensions. (#98455)

Generalizes DropUnitDimFromElementwiseOps to support inner unit
dimensions.
This change stems from improving lowering of contractionOps for Arm SME.
Where we end up with inner unit dimensions on MulOp, BroadcastOp and
TransposeOp, preventing the generation of outerproducts.
discussed
[here](https://discourse.llvm.org/t/on-improving-arm-sme-lowering-resilience-in-mlir/78543/17?u=nujaa).

Fix after : https://github.com/llvm/llvm-project/pull/97652 showed an
unhandled edge case when all dimensions are one. The generated target
VectorType would be `vector<f32>` which is apparently not supported by
the mulf.
In case all dimensions are dropped, the target vectorType is
vector<1xf32>

---------

Co-authored-by: Benjamin Maxwell <macdue@dueutil.tech>

show more ...

[mlir][vector] Restrict DropInnerMostUnitDimsTransfer{Read|Write} (#96218)

Restrict `DropInnerMostUnitDimsTransfer{Read|Write}` so that it fails
when one of the indices to be dropped could be != 0

[mlir][vector] Restrict DropInnerMostUnitDimsTransfer{Read|Write} (#96218)

Restrict `DropInnerMostUnitDimsTransfer{Read|Write}` so that it fails
when one of the indices to be dropped could be != 0 and "out of bounds":

```mlir
func.func @negative_example(%arg0: memref<16x1xf32>, %arg1: vector<8x1xf32>, %idx_1: index, %idx_2: index) {
vector.transfer_write %arg1, %arg0[%idx_1, %idx_2] {in_bounds = [true, false]} : vector<8x1xf32>, memref<16x1xf32>
return
}
```

This is an edge case that could represent an out-of-bounds access,
though that will depend on the actual value of %i. Importantly, without
this change it would be transformed as follows:

```mlir
func.func @negative_example(%arg0: memref<16x1xf32>, %arg1: vector<8x1xf32>, %arg2: index, %arg3: index) {
%subview = memref.subview %arg0[0, 0] [16, 1] [1, 1] : memref<16x1xf32> to memref<16xf32, strided<[1]>>
%0 = vector.shape_cast %arg1 : vector<8x1xf32> to vector<8xf32>
vector.transfer_write %0, %subview[%arg2] {in_bounds = [true]} : vector<8xf32>, memref<16xf32, strided<[1]>>
return
}
```

This is incorrect - `%idx_2` is ignored and the "out of bounds" flags is
not propagated. Hence the extra restriction to avoid such cases.

NOTE: This is a follow-up for: #94904

show more ...

Revert "[MLIR][Vector] Generalize DropUnitDimFromElementwiseOps to non leading / trailing dimensions." (#97652)

Reverts llvm/llvm-project#92934 because it breaks some lowering. To
repro: `mlir-opt

Revert "[MLIR][Vector] Generalize DropUnitDimFromElementwiseOps to non leading / trailing dimensions." (#97652)

Reverts llvm/llvm-project#92934 because it breaks some lowering. To
repro: `mlir-opt -test-vector-transfer-flatten-patterns ~/repro.mlir`

```mlir
func.func @unit_dim_folding(%arg0: vector<1x1xf32>) -> vector<1x1xf32> {
%cst = arith.constant dense<0.000000e+00> : vector<1x1xf32>
%0 = arith.mulf %arg0, %cst : vector<1x1xf32>
return %0 : vector<1x1xf32>
}
```

show more ...

mlir/LogicalResult: move into llvm (#97309)

This patch is part of a project to move the Presburger library into
LLVM.

[mlir][vector] Generalize folding of ext-contractionOp to other types. (#96593)

Many state of the art models and quantization operations are now
directly working on vector.contract on integers.

[mlir][vector] Generalize folding of ext-contractionOp to other types. (#96593)

Many state of the art models and quantization operations are now
directly working on vector.contract on integers.

This commit enables generalizes ext-contraction folding S.T we can emit
more performant vector.contracts on codegen pipelines.

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

show more ...

[mlir][vector] Add ElementwiseToOuterproduct (#93664)

1D multi-reduction are lowered to arith which can prevent some
optimisations. I propose `ElementwiseToOuterproduct` matching a series of
ops t

[mlir][vector] Add ElementwiseToOuterproduct (#93664)

1D multi-reduction are lowered to arith which can prevent some
optimisations. I propose `ElementwiseToOuterproduct` matching a series of
ops to generate `vector.outerproduct`.
As part of some `ElementwiseToVectorOpsPatterns`, it could allow to fuse
other elementwiseOps to vector dialect.
Originally discussed
https://discourse.llvm.org/t/on-improving-arm-sme-lowering-resilience-in-mlir/78543/24.

quote @MacDue
```
%lhsBcast = vector.broadcast %lhsCast : vector<[4]xf32> to vector<[4]x[4]xf32>
%lhsT = vector.transpose %lhsBcast, [1, 0] : vector<[4]x[4]xf32> to vector<[4]x[4]xf32>
%rhsBcast = vector.broadcast %rhs : vector<[4]xf32> to vector<[4]x[4]xf32>
%mul = arith.mulf %lhsT, %rhsBcast : vector<[4]x[4]xf32>
```

Can be rewritten as:

```
%mul = vector.outerproduct $lhs, $rhs : vector<[4]xf32>, vector<[4]xf32>
```

---------

Co-authored-by: Han-Chung Wang <hanhan0912@gmail.com>

show more ...

[mlir][vector] Update tests for collapse 3/n (nfc) (#94906)

The main goal of this PR (and subsequent PRs), is to add more tests with
scalable vectors to:
* vector-transfer-collapse-inner-most-di

[mlir][vector] Update tests for collapse 3/n (nfc) (#94906)

The main goal of this PR (and subsequent PRs), is to add more tests with
scalable vectors to:
* vector-transfer-collapse-inner-most-dims.mlir

There's quite a few cases to consider, hence this is split into multiple
PRs. In this PR, the very first test for `vector.transfer_write` is
complemented with all the possible combinations:
* scalable (rather than fixed) unit trailing dim,
* dynamic (rather than static) trailing dim in the source memref.

To this end, the following tests:
* `@leading_scalable_dimension_transfer_write`
`@trailing_scalable_one_dim_transfer_write`

are replaced with:
* `@drop_two_inner_most_dim_scalable_inner_dim` and
`@negative_scalable_unit_dim`,

respectively. In addition:
* "_for_transfer_write" is removed from function names (to reduce
noise).

In addition, to maintain consistency between the tests for `xfer_read`
and `xfer_write`, 2 negative tests for `xfer_read` are also renamed.
This is to follow the suggestion made during the review of this PR.

Extra comments in "VectorTransforms.cpp" are added to better
document the limitations related to scalable vectors and which tests
added here excercise.

This is a follow-up for: #94490 and #94604

NOTE: This PR is limited to tests for `vector.transfer_write`.

show more ...

[MLIR][Vector] Generalize DropUnitDimFromElementwiseOps to non leading / trailing dimensions. (#92934)

Generalizes `DropUnitDimFromElementwiseOps` to support inner unit
dimensions.
This change ste

[MLIR][Vector] Generalize DropUnitDimFromElementwiseOps to non leading / trailing dimensions. (#92934)

Generalizes `DropUnitDimFromElementwiseOps` to support inner unit
dimensions.
This change stems from improving lowering of contractionOps for Arm SME.
Where we end up with inner unit dimensions on MulOp, BroadcastOp and
TransposeOp, preventing the generation of outerproducts.
discussed
[here](https://discourse.llvm.org/t/on-improving-arm-sme-lowering-resilience-in-mlir/78543/17?u=nujaa).

---------

Co-authored-by: Benjamin Maxwell <macdue@dueutil.tech>

show more ...

[mlir][vector] Restrict DropInnerMostUnitDimsTransferRead (#94904)

Restrict `DropInnerMostUnitDimsTransferRead` so that it fails when one
of the indices to be dropped could be != 0, e.g.

```mlir

[mlir][vector] Restrict DropInnerMostUnitDimsTransferRead (#94904)

Restrict `DropInnerMostUnitDimsTransferRead` so that it fails when one
of the indices to be dropped could be != 0, e.g.

```mlir
func.func @negative_example(%A: memref<16x1xf32>, %i:index, %j:index) -> (vector<8x1xf32>) {
%f0 = arith.constant 0.0 : f32
%1 = vector.transfer_read %A[%i, %j], %f0 : memref<16x1xf32>, vector<8x1xf32>
return %1 : vector<8x1xf32>
}
```

This is an edge case that could represent an out-of-bounds access,
though that will depend on the actual value of `%j`. Importantly,
_without this change_ it would be transformed as follows:
```mlir
func.func @negative_example(%arg0: memref<16x1xf32>, %arg1: index, %arg2: index) -> vector<8x1xf32> {
%cst = arith.constant 0.000000e+00 : f32
%subview = memref.subview %arg0[0, 0] [16, 1] [1, 1] : memref<16x1xf32> to memref<16xf32, strided<[1]>>
%0 = vector.transfer_read %subview[%arg1], %cst : memref<16xf32, strided<[1]>>, vector<8xf32>
%1 = vector.shape_cast %0 : vector<8xf32> to vector<8x1xf32>
return %1 : vector<8x1xf32>
}
```

This is incorrect - `%arg2` is ignored. Hence the extra restriction to
avoid such cases.

NOTE: This PR is limited to tests for `vector.transfer_read`.

show more ...

[mlir][vector] Fix scalability issues in drop innermost unit dims transfer patterns (#92402)

Previously, these rewrites would drop scalable dimensions and treated
`[1]` (scalable one dim) as a unit

[mlir][vector] Fix scalability issues in drop innermost unit dims transfer patterns (#92402)

Previously, these rewrites would drop scalable dimensions and treated
`[1]` (scalable one dim) as a unit dimension. This patch propagates
scalable dimensions and ensures `[1]` is not treated as a unit
dimension.

show more ...

[mlir][Vector] Fix crash in drop unit dims (#87104)

An `arith.select` may have a scalar condition and true/false vector values.