[mlir][IR][NFC] Move free-standing functions to `MemRefType` (#123465)

Turn free-standing `MemRefType`-related helper functions in
`BuiltinTypes.h` into member functions.

[MemRef] Migrate away from PointerUnion::{is,get} (NFC) (#120382)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dy

[MemRef] Migrate away from PointerUnion::{is,get} (NFC) (#120382)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[mlir][IR] Add `Block::isReachable` helper function (#114928)

Add a new helper function `isReachable` to `Block`. This function
traverses all successors of a block to determine if another block is

[mlir][IR] Add `Block::isReachable` helper function (#114928)

Add a new helper function `isReachable` to `Block`. This function
traverses all successors of a block to determine if another block is
reachable from the current block.

This functionality has been reimplemented in multiple places in MLIR.
Possibly additional copies in downstream projects. Therefore, moving it
to a common place.

show more ...

[MLIR][Vector] Update Transfer{Read|Write}DropUnitDimsPattern patterns (#112394)

Updates `TransferWriteDropUnitDimsPattern` and
`TransferReadDropUnitDimsPattern` to inherit from
`MaskableOpRewrite

[MLIR][Vector] Update Transfer{Read|Write}DropUnitDimsPattern patterns (#112394)

Updates `TransferWriteDropUnitDimsPattern` and
`TransferReadDropUnitDimsPattern` to inherit from
`MaskableOpRewritePattern` so that masked versions of
xfer_read/xfer_write Ops are also supported:

```mlir
%v = vector.mask %mask {
vector.transfer_read %arg[%c0, %c0, %c0, %c0], %cst :
memref<1x1x3x2xi8, strided<[6, 6, 2, 1], offset: ?>>, vector<3x2xi8>
} : vector<3x2xi1> -> vector<3x2xi8>
```

show more ...

[mlir][vector] Add all view-like ops to transfer flow opt (#110521)

`vector.transfer_*` folding and forwarding currently does not take into
account reshaping view-like memref ops (expand and collap

[mlir][vector] Add all view-like ops to transfer flow opt (#110521)

`vector.transfer_*` folding and forwarding currently does not take into
account reshaping view-like memref ops (expand and collapse shape),
leading to potentially invalid store folding or value forwarding. This
patch adds tracking for those (and other) view-like ops. It is still
possible to design operations that alias memrefs without being a view
(e.g. memref in the iter_args of an `scf.for`), so these patterns may
still need revisiting in the future.

show more ...

[mlir][vector] Refactor vector-transfer-flatten.mlir (nfc) (1/n) (#95743)

The main goal of this and subsequent PRs is to unify and categorize
tests in:
* vector-transfer-flatten.mlir

This s

[mlir][vector] Refactor vector-transfer-flatten.mlir (nfc) (1/n) (#95743)

The main goal of this and subsequent PRs is to unify and categorize
tests in:
* vector-transfer-flatten.mlir

This should make it easier to identify the edge cases being tested (and
how they differ), remove duplicates and to add tests for scalable
vectors.

The main contributions of this PR:
* split tests that covered `xfer_read` + `xfer_write` into separate
tests (majority of the existing tests check _one_ xfer Op at a time),
* organise tests for `xfer_read` and `xfer_write` into separate
groups (separate with a big bold comment).

Note, all tests (i.e. test cases) are preserved and some new tests are
added. Deletions that you will see in `git diff` correspond to
`xfer_write` and `xfer_read` Ops being extracted to separate functions
(so that there's one xfer Op per function). In particular, the number of
test functions has grown from 26 to 30.

In addition, this PR unifies the tests so that:
* input variable names are consistent (e.g. make sure that the input
memref is always `arg`)
* CHECK lines use similar indentations
* 2 x tabs are always used for function arguments, 1 x tab for
function body

Finally, changes in "VectorTransferOpTransforms.cpp" are merely meant to
unify comments and logic between
* `FlattenContiguousRowMajorTransferWritePattern` and
* `FlattenContiguousRowMajorTransferReadPattern`.

show more ...

[mlir][vector] Improve flattening vector.transfer_write ops. (#94051)

We can flatten the transfer ops even when the collapsed indices are not
zeros. We can compute it. It is already supported in
v

[mlir][vector] Improve flattening vector.transfer_write ops. (#94051)

We can flatten the transfer ops even when the collapsed indices are not
zeros. We can compute it. It is already supported in
vector.transfer_read cases. The revision refactors the logic and reuse
it in transfer_write cases.

show more ...

[mlir][vector] Teach `TransferOptimization` to look through trivial aliases (#87805)

This allows `TransferOptimization` to eliminate and forward stores that
are to trivial aliases (rather than just

[mlir][vector] Teach `TransferOptimization` to look through trivial aliases (#87805)

This allows `TransferOptimization` to eliminate and forward stores that
are to trivial aliases (rather than just to identical memref values).

A trivial aliases is (currently) defined as:

1. A `memref.cast`
2. A `memref.subview` with a zero offset and unit strides
3. A chain of 1 and 2

show more ...

[mlir][Vector] Fix bug in vector xfer op flattening transformation (#81964)

It looks like the affine map generated to compute the indices of the
collapsed dimensions used the wrong dim size. For in

[mlir][Vector] Fix bug in vector xfer op flattening transformation (#81964)

It looks like the affine map generated to compute the indices of the
collapsed dimensions used the wrong dim size. For indices `[idx0][idx1]`
we computed the collapsed index as `idx0*size0 + idx1` instead of
`idx0*size1 + idx1`. This led to correctness issues in convolution tests
when enabling this transformation internally.

show more ...

[mlir][Vector] Add vector bitwidth target to xfer op flattening (#81966)

This PR adds an optional bitwidth parameter to the vector xfer op
flattening transformation so that the flattening doesn't h

[mlir][Vector] Add vector bitwidth target to xfer op flattening (#81966)

This PR adds an optional bitwidth parameter to the vector xfer op
flattening transformation so that the flattening doesn't happen if the
trailing dimension of the read/writen vector is larger than this
bitwidth (i.e., we are already able to fill at least one vector register
with that size).

show more ...

[mlir][vector] Add pattern to drop unit dim from elementwise(a, b)) (#74817)

For vectors with either leading or trailing unit dim, replaces:

elementwise(a, b)

with:

sc_a = shape_cas

[mlir][vector] Add pattern to drop unit dim from elementwise(a, b)) (#74817)

For vectors with either leading or trailing unit dim, replaces:

elementwise(a, b)

with:

sc_a = shape_cast(a)
sc_b = shape_cast(b)
res = elementwise(sc_a, sc_b)
return shape_cast(res)

The newly inserted shape_cast Ops fold (before elementwise Op) and then
restore (after elementwise Op) the unit dim. Vectors `a` and `b` are
required to be rank > 1.

Example:
```mlir
%mul = arith.mulf %B_row, %A_row : vector<1x[4]xf32>
%cast = vector.shape_cast %mul : vector<1x[4]xf32> to vector<[4]xf32>
```

gets converted to:

```mlir
%B_row_sc = vector.shape_cast %B_row : vector<1x[4]xf32> to vector<[4]xf32>
%A_row_sc = vector.shape_cast %A_row : vector<1x[4]xf32> to vector<[4]xf32>
%mul = arith.mulf %B_row_sc, %A_row_sc : vector<[4]xf32>
%mul_sc = vector.shape_cast %mul : vector<[4]xf32> to vector<1x[4]xf32>
%cast = vector.shape_cast %mul_sc : vector<1x[4]xf32> to vector<[4]xf32>
```

In practice, the bottom 2 shape_cast(s) will be folded away.

show more ...

[mlir][Vector] Update patterns for flattening vector.xfer Ops (2/N) (#73523)

Updates patterns for flattening `vector.transfer_read` by relaxing the
requirement that the "collapsed" indices are all

[mlir][Vector] Update patterns for flattening vector.xfer Ops (2/N) (#73523)

Updates patterns for flattening `vector.transfer_read` by relaxing the
requirement that the "collapsed" indices are all zero. This enables
collapsing cases like this one:

```mlir
%2 = vector.transfer_read %arg4[%c0, %arg0, %arg1, %c0] ... :
memref<1x43x4x6xi32>, vector<1x2x6xi32>
```

Previously only the following case would be consider for collapsing
(all indices are 0):

```mlir
%2 = vector.transfer_read %arg4[%c0, %c0, %c0, %c0] ... :
memref<1x43x4x6xi32>, vector<1x2x6xi32>
```

Also adds some new comments and renames the `firstContiguousInnerDim`
parameter as `firstDimToCollapse` (the latter better matches the actual
meaning).

Similar updates for `vector.transfer_write` will be implemented in a
follow-up patch.

show more ...

[mlir][Vector] Update patterns for flattening vector.xfer Ops (1/N) (#73522)

Updates "flatten vector" patterns to support more cases, namely Ops that
read/write vectors with leading unit dims. For

[mlir][Vector] Update patterns for flattening vector.xfer Ops (1/N) (#73522)

Updates "flatten vector" patterns to support more cases, namely Ops that
read/write vectors with leading unit dims. For example:

```mlir
%0 = vector.transfer_read %arg0[%c0, %c0, %c0, %c0] ... :
memref<5x4x3x2xi8, strided<[24, 6, 2, 1], offset: ?>>, vector<1x1x2x2xi8>
```

Currently, the `vector.transfer_read` above would not be flattened. With
this
change, it will be rewritten as follows:
```mlir
%collapse_shape = memref.collapse_shape %arg0 [[0, 1, 2, 3]] :
memref<5x4x3x2xi8, strided<[24, 6, 2, 1], offset: ?>>
into memref<120xi8, strided<[1], offset: ?>>
%0 = vector.transfer_read %collapse_shape[%c0] ... :
memref<120xi8, strided<[1], offset: ?>>, vector<4xi8>
%1 = vector.shape_cast %0 : vector<4xi8> to vector<1x1x2x2xi8>
```

`hasMatchingInnerContigousShape` is generalised and renamed as
`isContiguousSlice` to better match the updated functionality. A few
test names are updated to better highlight what case is being exercised.

show more ...

[mlir][vector] Fix unit dim dropping pattern for masked writes (#74038)

This does the same as #72142 for vector.transfer_write. Previously the
pattern would silently drop the mask.

[mlir][vector] Extend TransferReadDropUnitDimsPattern to support partially-static memrefs (#72142)

This patch extends TransferReadDropUnitDimsPattern to support dropping
unit dims from partially-st

[mlir][vector] Extend TransferReadDropUnitDimsPattern to support partially-static memrefs (#72142)

This patch extends TransferReadDropUnitDimsPattern to support dropping
unit dims from partially-static memrefs, for example:

%v = vector.transfer_read %base[%c0, %c0], %pad {in_bounds = [true, true]} :
memref<?x1xi8, strided<[?, ?], offset: ?>>, vector<[16]x1xi8>

Is rewritten as:

%dim0 = memref.dim %base, %c0 : memref<?x1xi8, strided<[?, ?], offset: ?>>
%subview = memref.subview %base[0, 0] [%dim0, 1] [1, 1] :
memref<?x1xi8, strided<[?, ?], offset: ?>> to memref<?xi8, #map1>
%v = vector.transfer_read %subview[%c0], %pad {in_bounds = [true]}
: memref<?xi8, #map1>, vector<[16]xi8>

Scalable vectors are now also supported, the scalable dims were being
dropped when creating the rank-reduced vector type. The xfer op can also
have a mask of type 'vector.create_mask', which gets rewritten as long
as the mask of the unit dim is a constant of 1.

show more ...

[mlir][vector] Enable transfer op hoisting with dynamic indices (#68500)

Recent changes (https://github.com/llvm/llvm-project/pull/66930)
disabled vector transfer ops hoisting with view-like interm

[mlir][vector] Enable transfer op hoisting with dynamic indices (#68500)

Recent changes (https://github.com/llvm/llvm-project/pull/66930)
disabled vector transfer ops hoisting with view-like intermediate ops.
The recommended way is to fold subview ops into transfer op indices
before invoking hoisting. That would mean now we see transfer op indices
involving dynamic values, instead of static constant values before with
subview ops. Therefore hoisting won't kick in anymore. This breaks
downstream users.

To fix it, this commit enables hoisting transfer ops with dynamic
indices by using `ValueBoundsConstraintSet` to prove ranges are disjoint
in `isDisjointTransferIndices`. Given that utility is used in many
places including op folders, right now we introduce a flag to it and
only set as true for "heavy" transforms in hoisting and load-store
forwarding.

show more ...

[mlir][Vector] Add support for Value indices to vector.extract/insert

`vector.extract/insert` ops only support constant indices. This PR is
extending them so that arbitrary values can be used instea

[mlir][Vector] Add support for Value indices to vector.extract/insert

`vector.extract/insert` ops only support constant indices. This PR is
extending them so that arbitrary values can be used instead.

This work is part of the RFC: https://discourse.llvm.org/t/rfc-psa-remove-vector-extractelement-and-vector-insertelement-ops-in-favor-of-vector-extract-and-vector-insert-ops

Differential Revision: https://reviews.llvm.org/D155034

show more ...

[mlir][vector] Refine vector.transfer_read hoisting/forwarding (#65770)

Make sure that when analysing a `vector.transfer_read` that's a
candidate for either hoisting or store-to-load forwarding,
`me

[mlir][vector] Refine vector.transfer_read hoisting/forwarding (#65770)

Make sure that when analysing a `vector.transfer_read` that's a
candidate for either hoisting or store-to-load forwarding,
`memref.collapse_shape` Ops are correctly included in the alias
analysis. This is done by either
* making sure that relevant users are taken into account, or
* source Ops are correctly identified.

show more ...

[mlir][vector] Use DenseI64ArrayAttr for ExtractOp/InsertOp positions

`DenseI64ArrayAttr` provides a better API than `I64ArrayAttr`. E.g., accessors returning `ArrayRef<int64_t>` (instead of `ArrayA

[mlir][vector] Use DenseI64ArrayAttr for ExtractOp/InsertOp positions

`DenseI64ArrayAttr` provides a better API than `I64ArrayAttr`. E.g., accessors returning `ArrayRef<int64_t>` (instead of `ArrayAttr`) are generated.

Differential Revision: https://reviews.llvm.org/D156684

show more ...

[mlir][NFC] Use `getConstantIntValue` instead of casting to `ConstantIndexOp`

`getConstantIntValue` extracts constant values from all constant-like ops, not just `arith::ConstantIndexOp`.

Different

[mlir][NFC] Use `getConstantIntValue` instead of casting to `ConstantIndexOp`

`getConstantIntValue` extracts constant values from all constant-like ops, not just `arith::ConstantIndexOp`.

Differential Revision: https://reviews.llvm.org/D154356

show more ...

Reland "[mlir][Vector] Extend xfer drop unit dim patterns"

This reverts commit 76d71f3792b2b1864992446f7b1028b026dccd11.

[mlir][Vector] Prevent vector-to-scalar xfer patterns from triggering on sub-vectors

Patterns that convert extract(transfer_read) into a scalar load where
incorrectly triggering for cases where a su

[mlir][Vector] Prevent vector-to-scalar xfer patterns from triggering on sub-vectors

Patterns that convert extract(transfer_read) into a scalar load where
incorrectly triggering for cases where a sub-vector instead of a scalar
was extracted.

Reviewed By: nicolasvasilache, hanchung, awarzynski

Differential Revision: https://reviews.llvm.org/D151862

show more ...

Revert "[mlir][Vector] Extend xfer drop unit dim patterns"

This reverts commit a53cd03deac5e6272e9dae88a90cd51410d312d5.

This commit is exposing some implementation gaps in other patterns.
Revertin

Revert "[mlir][Vector] Extend xfer drop unit dim patterns"

This reverts commit a53cd03deac5e6272e9dae88a90cd51410d312d5.

This commit is exposing some implementation gaps in other patterns.
Reverting for now.

show more ...

[mlir][Vector] Extend xfer drop unit dim patterns

This patch extends the transfer drop unit dim patterns to support cases where the vector shape should also be reduced
(e.g., transfer_read(memref<1x

[mlir][Vector] Extend xfer drop unit dim patterns

This patch extends the transfer drop unit dim patterns to support cases where the vector shape should also be reduced
(e.g., transfer_read(memref<1x4x1xf32>, vector<1x4x1xf32>) -> transfer_read(memref<4xf32>, vector<4xf32>).

Reviewed By: hanchung, pzread

Differential Revision: https://reviews.llvm.org/D151007

show more ...

[mlir][Vector] Extend xfer_read(extract)->scalar load to support multiple uses

This patch extends the vector.extract(vector.transfer_read) -> scalar
load patterns to support vector.transfer_read wit

[mlir][Vector] Extend xfer_read(extract)->scalar load to support multiple uses

This patch extends the vector.extract(vector.transfer_read) -> scalar
load patterns to support vector.transfer_read with multiple uses. For
now, we check that all the uses are vector.extract operations.
Supporting multiple uses is predicated under a flag.

Reviewed By: hanchung

Differential Revision: https://reviews.llvm.org/D150812

show more ...