[mlir][vector] Add a new TD Op for patterns leveraging ShapeCastOp (#110525)

Adds a new Transform Dialect Op that collects patters for dropping unit
dims from various Ops:
* `transform.apply_pat

[mlir][vector] Add a new TD Op for patterns leveraging ShapeCastOp (#110525)

Adds a new Transform Dialect Op that collects patters for dropping unit
dims from various Ops:
* `transform.apply_patterns.vector.drop_unit_dims_with_shape_cast`.

It excludes patterns for vector.transfer Ops - these are collected
under:
* `apply_patterns.vector.rank_reducing_subview_patterns`,

and use ShapeCastOp _and_ SubviewOp to reduce the rank (and to eliminate
unit dims).

This new TD Ops allows us to test the "ShapeCast folder" pattern in
isolation. I've extracted the only test that I could find for that
folder from "vector-transforms.mlir" and moved it to a dedicated file:
"shape-cast-folder.mlir". I also added a test case with scalable
vectors.

Changes in VectorTransforms.cpp are not needed (added a comment with
a TODO + ordered the patterns alphabetically). I am Including them here
to avoid a separate PR.

show more ...

[mlir][vector] Populate sink patterns in apply_patterns.vector.reduction_to_contract (#104754)

This restores the functionality to before:
https://github.com/llvm/llvm-project/commit/42944da5ba7617b

[mlir][vector] Populate sink patterns in apply_patterns.vector.reduction_to_contract (#104754)

This restores the functionality to before:
https://github.com/llvm/llvm-project/commit/42944da5ba7617bbc02f341e9ef401c325310a73
This fixes a buildbot failure:
https://lab.llvm.org/buildbot/#/builders/143/builds/1487

show more ...

[mlir] Support DialectRegistry extension comparison (#101119)

`PassManager::run` loads the dependent dialects for each pass into the
current context prior to invoking the individual passes. If the

[mlir] Support DialectRegistry extension comparison (#101119)

`PassManager::run` loads the dependent dialects for each pass into the
current context prior to invoking the individual passes. If the
dependent dialect is already loaded into the context, this should be a
no-op. However, if there are extensions registered in the
`DialectRegistry`, the dependent dialects are unconditionally registered
into the context.

This poses a problem for dynamic pass pipelines, however, because they
will likely be executing while the context is in an immutable state
(because of the parent pass pipeline being run).

To solve this, we'll update the extension registration API on
`DialectRegistry` to require a type ID for each extension that is
registered. Then, instead of unconditionally registered dialects into a
context if extensions are present, we'll check against the extension
type IDs already present in the context's internal `DialectRegistry`.
The context will only be marked as dirty if there are net-new extension
types present in the `DialectRegistry` populated by
`PassManager::getDependentDialects`.

Note: this PR removes the `addExtension` overload that utilizes
`std::function` as the parameter. This is because `std::function` is
copyable and potentially allocates memory for the contained function so
we can't use the function pointer as the unique type ID for the
extension.

Downstream changes required:
- Existing `DialectExtension` subclasses will need a type ID to be
registered for each subclass. More details on how to register a type ID
can be found here:
https://github.com/llvm/llvm-project/blob/8b68e06731e0033ed3f8d6fe6292ae671611cfa1/mlir/include/mlir/Support/TypeID.h#L30
- Existing uses of the `std::function` overload of `addExtension` will
need to be refactored into dedicated `DialectExtension` classes with
associated type IDs. The attached `std::function` can either be inlined
into or called directly from `DialectExtension::apply`.

---------

Co-authored-by: Mehdi Amini <joker.eph@gmail.com>

show more ...

[mlir][vector] Add ElementwiseToOuterproduct (#93664)

1D multi-reduction are lowered to arith which can prevent some
optimisations. I propose `ElementwiseToOuterproduct` matching a series of
ops t

[mlir][vector] Add ElementwiseToOuterproduct (#93664)

1D multi-reduction are lowered to arith which can prevent some
optimisations. I propose `ElementwiseToOuterproduct` matching a series of
ops to generate `vector.outerproduct`.
As part of some `ElementwiseToVectorOpsPatterns`, it could allow to fuse
other elementwiseOps to vector dialect.
Originally discussed
https://discourse.llvm.org/t/on-improving-arm-sme-lowering-resilience-in-mlir/78543/24.

quote @MacDue
```
%lhsBcast = vector.broadcast %lhsCast : vector<[4]xf32> to vector<[4]x[4]xf32>
%lhsT = vector.transpose %lhsBcast, [1, 0] : vector<[4]x[4]xf32> to vector<[4]x[4]xf32>
%rhsBcast = vector.broadcast %rhs : vector<[4]xf32> to vector<[4]x[4]xf32>
%mul = arith.mulf %lhsT, %rhsBcast : vector<[4]x[4]xf32>
```

Can be rewritten as:

```
%mul = vector.outerproduct $lhs, $rhs : vector<[4]xf32>, vector<[4]xf32>
```

---------

Co-authored-by: Han-Chung Wang <hanhan0912@gmail.com>

show more ...

[mlir][vector] Add support for unrolling vector.bitcast ops. (#94064)

The revision unrolls vector.bitcast like:

```mlir
%0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
```

to

[mlir][vector] Add support for unrolling vector.bitcast ops. (#94064)

The revision unrolls vector.bitcast like:

```mlir
%0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
```

to

```mlir
%cst = arith.constant dense<0> : vector<2x2xi64>
%0 = vector.extract %arg0[0] : vector<4xi32> from vector<2x4xi32>
%1 = vector.bitcast %0 : vector<4xi32> to vector<2xi64>
%2 = vector.insert %1, %cst [0] : vector<2xi64> into vector<2x2xi64>
%3 = vector.extract %arg0[1] : vector<4xi32> from vector<2x4xi32>
%4 = vector.bitcast %3 : vector<4xi32> to vector<2xi64>
%5 = vector.insert %4, %2 [1] : vector<2xi64> into vector<2x2xi64>
```

The scalable vector is not supported because of the limitation of
`vector::createUnrollIterator`. The targetRank could mismatch the final
rank during unrolling; there is no direct way to query what the final
rank is from the object.

show more ...

[mlir][vector] Add Vector-dialect interleave-to-shuffle pattern, enable in VectorToSPIRV (#92012)

This is the second attempt at merging #91800, which bounced due to a
linker error apparently caused

[mlir][vector] Add Vector-dialect interleave-to-shuffle pattern, enable in VectorToSPIRV (#92012)

This is the second attempt at merging #91800, which bounced due to a
linker error apparently caused by an undeclared dependency.
`MLIRVectorToSPIRV` needed to depend on `MLIRVectorTransforms`. In fact
that was a preexisting issue already flagged by the tool in
https://discourse.llvm.org/t/ninja-can-now-check-for-missing-cmake-dependencies-on-generated-files/74344.

Context: https://github.com/iree-org/iree/issues/17346.

Test IREE integrate showing it's fixing the problem it's intended to
fix, i.e. it allows IREE to drop its local revert of
https://github.com/llvm/llvm-project/pull/89131:

https://github.com/iree-org/iree/pull/17359

This is added to VectorToSPIRV because SPIRV doesn't currently handle
`vector.interleave` (see motivating context above).

This is limited to 1D, non-scalable vectors.

show more ...

Revert "[mlir][vector] Add Vector-dialect interleave-to-shuffle pattern, enable in VectorToSPIRV" (#92006)

Reverts llvm/llvm-project#91800

Reason: https://lab.llvm.org/buildbot/#/builders/268/bui

Revert "[mlir][vector] Add Vector-dialect interleave-to-shuffle pattern, enable in VectorToSPIRV" (#92006)

Reverts llvm/llvm-project#91800

Reason: https://lab.llvm.org/buildbot/#/builders/268/builds/13935

show more ...

[mlir][vector] Add Vector-dialect interleave-to-shuffle pattern, enable in VectorToSPIRV (#91800)

Context: https://github.com/iree-org/iree/issues/17346.

Test IREE integrate showing it's fixing t

[mlir][vector] Add Vector-dialect interleave-to-shuffle pattern, enable in VectorToSPIRV (#91800)

Context: https://github.com/iree-org/iree/issues/17346.

Test IREE integrate showing it's fixing the problem it's intended to
fix, i.e. it allows IREE to drop its local revert of
https://github.com/llvm/llvm-project/pull/89131:

https://github.com/iree-org/iree/pull/17359

This is added to VectorToSPIRV because SPIRV doesn't currently handle
`vector.interleave` (see motivating context above).

This is limited to 1D, non-scalable vectors.

show more ...

[mlir] split transform interfaces into a separate library (#85221)

Transform interfaces are implemented, direction or via extensions, in
libraries belonging to multiple other dialects. Those dialec

[mlir] split transform interfaces into a separate library (#85221)

Transform interfaces are implemented, direction or via extensions, in
libraries belonging to multiple other dialects. Those dialects don't
need to depend on the non-interface part of the transform dialect, which
includes the growing number of ops and transitive dependency footprint.

Split out the interfaces into a separate library. This in turn requires
flipping the dependency from the interface on the dialect that has crept
in because both co-existed in one library. The interface shouldn't
depend on the transform dialect either.

As a consequence of splitting, the capability of the interpreter to
automatically walk the payload IR to identify payload ops of a certain
kind based on the type used for the entry point symbol argument is
disabled. This is a good move by itself as it simplifies the interpreter
logic. This functionality can be trivially replaced by a
`transform.structured.match` operation.

show more ...

[mlir][VectorOps] Add unrolling for n-D vector.interleave ops (#80967)

This unrolls n-D vector.interleave ops like:

```mlir
vector.interleave %i, %j : vector<6x3xf32>
```

To a sequence of 1-

[mlir][VectorOps] Add unrolling for n-D vector.interleave ops (#80967)

This unrolls n-D vector.interleave ops like:

```mlir
vector.interleave %i, %j : vector<6x3xf32>
```

To a sequence of 1-D operations:
```mlir
%i_0 = vector.extract %i[0]
%j_0 = vector.extract %j[0]
%res_0 = vector.interleave %i_0, %j_0 : vector<3xf32>
vector.insert %res_0, %result[0] :
// ... repeated x6
```

The 1-D operations can then be directly lowered to LLVM.

Depends on: #80966

show more ...

[mlir][Vector] Add support for sub-byte transpose emulation (#80110)

This PR adds patterns to convert a sub-byte vector transpose into a
sequence of instructions that perform the transpose on i8 ve

[mlir][Vector] Add support for sub-byte transpose emulation (#80110)

This PR adds patterns to convert a sub-byte vector transpose into a
sequence of instructions that perform the transpose on i8 vector
elements. Whereas this rewrite may not lead to the absolute peak
performance, it should ensure correctness when dealing with sub-byte
transposes.

show more ...

[mlir][Vector] Add a rewrite pattern for better low-precision bitcast… (#66387)

…(trunci) expansion

This revision adds a rewrite for sequences of vector `bitcast(trunci)`
to use a more efficient

[mlir][Vector] Add a rewrite pattern for better low-precision bitcast… (#66387)

…(trunci) expansion

This revision adds a rewrite for sequences of vector `bitcast(trunci)`
to use a more efficient sequence of vector operations comprising
`shuffle` and `bitwise` ops.

Such patterns appear naturally when writing quantization /
dequantization functionality with the vector dialect.

The rewrite performs a simple enumeration of each of the bits in the
result vector and determines its provenance in the pre-trunci vector.
The enumeration is used to generate the proper sequence of `shuffle`,
`andi`, `ori` followed by an optional final `trunci`/`extui`.

The rewrite currently only applies to 1-D non-scalable vectors and bails
out if the final vector element type is not a multiple of 8. This is a
failsafe heuristic determined empirically: if the resulting type is not
an even number of bytes, further complexities arise that are not
improved by this pattern: the heavy lifting still needs to be done by
LLVM.

show more ...

[mlir][VectorOps] Don't drop scalable dims when lowering transfer_reads/writes (in VectorToLLVM)

This is a follow-on to D158753, and allows the lowering of a
transfer read/write of n-D vectors with

[mlir][VectorOps] Don't drop scalable dims when lowering transfer_reads/writes (in VectorToLLVM)

This is a follow-on to D158753, and allows the lowering of a
transfer read/write of n-D vectors with a single trailing scalable dimension
to primitive vector ops.

The final conversion to LLVM depends on D158517 and D158752, without
these patches type conversion will fail (or an assert is hit in the LLVM
backend) if the final IR contains an array of scalable vectors.

This patch adds `transform.apply_patterns.vector.lower_create_mask`
which allows the lowering of vector.create_mask/constant_mask to be
tested independently of --convert-vector-to-llvm.

Reviewed By: c-rhodes, awarzynski, dcaballe

Differential Revision: https://reviews.llvm.org/D159482

show more ...

[mlir][memref][transform] Add vector_to_llvm conversion patterns

These patterns are exposed via a new "apply_conversion_patterns" op.

Also provide a new type converter that converts from memref to

[mlir][memref][transform] Add vector_to_llvm conversion patterns

These patterns are exposed via a new "apply_conversion_patterns" op.

Also provide a new type converter that converts from memref to LLVM types. Conversion patterns that lower to LLVM are special: they require an `LLVMTypeConverter`; a normal `TypeConverter` is not enough. This revision also adds a new interface method to pattern descriptor ops to verify that the default type converter of the enclosing "apply_conversion_patterns" op is compatible with the set of patterns. At the moment, a simple `StringRef` is used. This can evolve to a richer type in the future if needed.

Differential Revision: https://reviews.llvm.org/D157369

show more ...

[mlir][transform] Expose vector patterns useful for cleaning up masked vectorization and add test to showcase composition

Differential Revision: https://reviews.llvm.org/D157047

[mlir][vector] Add transform.apply_patterns.vector.fold_arith_extension

This patch implements a transform op for the FoldArithExtIntoContractionOp
pattern. The pattern folds arith.extf into vector.c

[mlir][vector] Add transform.apply_patterns.vector.fold_arith_extension

This patch implements a transform op for the FoldArithExtIntoContractionOp
pattern. The pattern folds arith.extf into vector.contract for the
backends with native support for mixed-mode contractions.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D156484

show more ...

[mlir][vector] Remove duplicate tensor subset <-> vector transfer patterns

Remove patterns that fold tensor subset ops into vector transfer ops from the vector dialect. These patterns already exist

[mlir][vector] Remove duplicate tensor subset <-> vector transfer patterns

Remove patterns that fold tensor subset ops into vector transfer ops from the vector dialect. These patterns already exist in the tensor dialect.

Differential Revision: https://reviews.llvm.org/D154932

show more ...

[mlir][vector][transform] Expose tensor slice -> transfer folding patterns

Add a new transform op to populate patterns: ApplyFoldTensorSliceIntoTransferPatternsOp.

Differential Revision: https://re

[mlir][vector][transform] Expose tensor slice -> transfer folding patterns

Add a new transform op to populate patterns: ApplyFoldTensorSliceIntoTransferPatternsOp.

Differential Revision: https://reviews.llvm.org/D152531

show more ...

[mlir][vector][transform] Add ApplyCastAwayVectorLeadingOneDimPatternsOp

Expose the respective patterns as a transform op.

Differential Revision: https://reviews.llvm.org/D152346

[mlir][transform] Use separate ops instead of PatternRegistry

* Remove `transform::PatternRegistry`.
* Add a new op for each currently registered pattern set.
* Change names of vector dialect patter

[mlir][transform] Use separate ops instead of PatternRegistry

* Remove `transform::PatternRegistry`.
* Add a new op for each currently registered pattern set.
* Change names of vector dialect pattern selector ops, so that they are consistent with the remaining code base.
* Remove redundant `transform.vector.extract_address_computations` op.

Differential Revision: https://reviews.llvm.org/D152249

show more ...

[mlir][vector][transform] Register vector dialect patterns

Differential Revision: https://reviews.llvm.org/D151983

[mlir][Vector] Add a vector.materialize_masks transform operation

[mlir][Vector] Split transform.vector.lower_mask in 2 ops.

This gives us better control to lower masked operations independently of the create mask operations.
It is often useful to maintain high-le

[mlir][Vector] Split transform.vector.lower_mask in 2 ops.

This gives us better control to lower masked operations independently of the create mask operations.
It is often useful to maintain high-level mask information instead of lowering it too early to
too fine-grained form.

Differential Revision: https://reviews.llvm.org/D148162

show more ...

[mlir][Vector][Transforms] Improve the control over individual vector lowerings and transforms

This revision adds vector transform operations that allow us to better inspect the composition
of vario

[mlir][Vector][Transforms] Improve the control over individual vector lowerings and transforms

This revision adds vector transform operations that allow us to better inspect the composition
of various lowerings that were previously very opaque.

This commit is NFC in that it does not change patterns beyond adding `rewriter.notifyFailure` messages
and it does not change the tests beyond breaking them into pieces and using transforms instead of
throwaway opaque test passes.

Reviewed By: ftynse, springerm

Co-authored-by: Alex Zinenko <zinenko@google.com>

Differential Revision: https://reviews.llvm.org/D146755

show more ...

[mlir][Vector] NFC - Reorganize vector patterns

Vector dialect patterns have grown enormously in the past year to a point where they are now impenetrable.
Start reorganizing them towards finer-grain

[mlir][Vector] NFC - Reorganize vector patterns

Vector dialect patterns have grown enormously in the past year to a point where they are now impenetrable.
Start reorganizing them towards finer-grained control.

Differential Revision: https://reviews.llvm.org/D146736

show more ...