[mlir][Vector] Add an optional "masked" boolean array attribute to vector transfer operations

Summary:
Vector transfer ops semantic is extended to allow specifying a per-dimension `masked`
attribute

[mlir][Vector] Add an optional "masked" boolean array attribute to vector transfer operations

Summary:
Vector transfer ops semantic is extended to allow specifying a per-dimension `masked`
attribute. When the attribute is false on a particular dimension, lowering to LLVM emits
unmasked load and store operations.

Differential Revision: https://reviews.llvm.org/D80098

show more ...

[mlir] [VectorOps] Implement vector.create_mask lowering to LLVM IR

Summary:
First, compact implementation of lowering to LLVM IR. A bit more
challenging than the constant mask due to the dynamic in

[mlir] [VectorOps] Implement vector.create_mask lowering to LLVM IR

Summary:
First, compact implementation of lowering to LLVM IR. A bit more
challenging than the constant mask due to the dynamic indices, of course.
I like to hear if there are more efficient ways of doing this in LLVM,
but this for now at least gives us a functional reference implementation.

Reviewers: nicolasvasilache, ftynse, bkramer, reidtatge, andydavis1, mehdi_amini

Reviewed By: nicolasvasilache

Subscribers: mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, stephenneuendorffer, Joonsoo, grosul1, frgossen, Kayjukh, jurahul, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D79954

show more ...

[mlir][Vector] NFC - Rename vector.strided_slice into vector.extract_strided_slice

Differential Revision: https://reviews.llvm.org/D79734

[mlir] VectorToLLVM: propagate up from replaceTransferOp

In the Vector to LLVM conversion, the `replaceTransferOp` function calls
into a type converter that may fail and suppresses the status. Chang

[mlir] VectorToLLVM: propagate up from replaceTransferOp

In the Vector to LLVM conversion, the `replaceTransferOp` function calls
into a type converter that may fail and suppresses the status. Change
the function to return the failure status instead, Since it is called
from a pattern, the failure can be readily propagated to the rest of
infrastructure.

show more ...

[mlir][vector] set alignment when lowering transfer_read and transfer_write.

When emitting masked load / store, set alignment from data layout.

Differential Revision: https://reviews.llvm.org/D79246

[mlir] [VectorOps] Included i1 support for vector.print

Summary:
Added boolean support to vector.print.
Useful for upcoming "mask" tests.

Reviewers: ftynse, nicolasvasilache, andydavis1

Reviewed B

[mlir] [VectorOps] Included i1 support for vector.print

Summary:
Added boolean support to vector.print.
Useful for upcoming "mask" tests.

Reviewers: ftynse, nicolasvasilache, andydavis1

Reviewed By: andydavis1

Subscribers: mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, Joonsoo, grosul1, frgossen, Kayjukh, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D79198

show more ...

[mlir][vector] let transfer_read and transfer_write take non-zero addrspace.

Enhance lowering logic and tests so vector.transfer_read and
vector.transfer_write take memrefs on non-zero addrspaces.

[mlir][vector] let transfer_read and transfer_write take non-zero addrspace.

Enhance lowering logic and tests so vector.transfer_read and
vector.transfer_write take memrefs on non-zero addrspaces.

Differential Revision: https://reviews.llvm.org/D79023

show more ...

[mlir][VectorOps] Extend VectorTransfer lowering to n-D memref with minor identity map

Summary: This revision extends the lowering of vector transfers to work with n-D memref and 1-D vector where th

[mlir][VectorOps] Extend VectorTransfer lowering to n-D memref with minor identity map

Summary: This revision extends the lowering of vector transfers to work with n-D memref and 1-D vector where the permutation map is an identity on the most minor dimensions (1 for now).

Differential Revision: https://reviews.llvm.org/D78925

show more ...

[mlir] [VectorOps] Progressive lowering of vector.broadcast

Summary:
Rather than having a full, recursive, lowering of vector.broadcast
to LLVM IR, it is much more elegant to have a progressive lowe

[mlir] [VectorOps] Progressive lowering of vector.broadcast

Summary:
Rather than having a full, recursive, lowering of vector.broadcast
to LLVM IR, it is much more elegant to have a progressive lowering
of each vector.broadcast into a lower dimensional vector.broadcast,
until only elementary vector operations remain. This results
in more elegant, step-wise code, that is easier to understand.
Also makes some optimizations in the generated code.

Reviewers: nicolasvasilache, mehdi_amini, andydavis1, grosul1

Reviewed By: nicolasvasilache

Subscribers: mehdi_amini, rriddle, jpienaar, burmako, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, Joonsoo, grosul1, frgossen, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D78071

show more ...

[MLIR][NFC] applyPatternsGreedily -> applyPatternsAndFoldGreedily

Rename mlir::applyPatternsGreedily -> applyPatternsAndFoldGreedily. The
new name is a more accurate description of the method - it p

[MLIR][NFC] applyPatternsGreedily -> applyPatternsAndFoldGreedily

Rename mlir::applyPatternsGreedily -> applyPatternsAndFoldGreedily. The
new name is a more accurate description of the method - it performs
both, application of the specified patterns and folding of all ops in
the op's region irrespective of whether any patterns have been supplied.

Differential Revision: https://reviews.llvm.org/D77478

show more ...

[mlir][Vector] Add lowering of 1-D vector transfer_read/write to masked load/store

Summary:
This revision adds support to lower 1-D vector transfers to LLVM.
A mask of the vector length is created t

[mlir][Vector] Add lowering of 1-D vector transfer_read/write to masked load/store

Summary:
This revision adds support to lower 1-D vector transfers to LLVM.
A mask of the vector length is created that compares the base offset + linear index to the dim of the vector.
In each position where this does not overflow (i.e. offset + vector index < dim), the mask is set to 1.

A notable fact is that the lowering uses llvm.dialect_cast to allow writing code in the simplest form by targeting the simplest mix of vector and LLVM dialects and
letting other conversions kick in.

Differential Revision: https://reviews.llvm.org/D77703

show more ...

[mlir] Add a new RewritePattern::hasBoundedRewriteRecursion hook.

Summary: Some pattern rewriters, like dialect conversion, prohibit the unbounded recursion(or reapplication) of patterns on generate

[mlir] Add a new RewritePattern::hasBoundedRewriteRecursion hook.

Summary: Some pattern rewriters, like dialect conversion, prohibit the unbounded recursion(or reapplication) of patterns on generated IR. Most patterns are not written with recursive application in mind, so will generally explode the stack if uncaught. This revision adds a hook to RewritePattern, `hasBoundedRewriteRecursion`, to signal that the pattern can safely be applied to the generated IR of a previous application of the same pattern. This allows for establishing a contract between the pattern and rewriter that the pattern knows and can handle the potential recursive application.

Differential Revision: https://reviews.llvm.org/D77782

show more ...

[mlir][Pass] Update the PassGen to generate base classes instead of utilities

Summary:
This is much cleaner, and fits the same structure as many other tablegen backends. This was not done originally

[mlir][Pass] Update the PassGen to generate base classes instead of utilities

Summary:
This is much cleaner, and fits the same structure as many other tablegen backends. This was not done originally as the CRTP in the pass classes made it overly verbose/complex.

Differential Revision: https://reviews.llvm.org/D77367

show more ...

[mlir][Pass] Remove the use of CRTP from the Pass classes

This revision removes all of the CRTP from the pass hierarchy in preparation for using the tablegen backend instead. This creates a much cle

[mlir][Pass] Remove the use of CRTP from the Pass classes

This revision removes all of the CRTP from the pass hierarchy in preparation for using the tablegen backend instead. This creates a much cleaner interface in the C++ code, and naturally fits with the rest of the infrastructure. A new utility class, PassWrapper, is added to replicate the existing behavior for passes not suitable for using the tablegen backend.

Differential Revision: https://reviews.llvm.org/D77350

show more ...

[mlir][Pass][NFC] Replace usages of ModulePass with OperationPass<ModuleOp>

ModulePass doesn't provide any special utilities and thus doesn't give enough benefit to warrant a special pass class. Thi

[mlir][Pass][NFC] Replace usages of ModulePass with OperationPass<ModuleOp>

ModulePass doesn't provide any special utilities and thus doesn't give enough benefit to warrant a special pass class. This revision replaces all usages with the more general OperationPass.

Differential Revision: https://reviews.llvm.org/D77339

show more ...

[mlir][Pass] Add support for generating pass utilities via tablegen

This revision adds support for generating utilities for passes such as options/statistics/etc. that can be inferred from the table

[mlir][Pass] Add support for generating pass utilities via tablegen

This revision adds support for generating utilities for passes such as options/statistics/etc. that can be inferred from the tablegen definition. This removes additional boilerplate from the pass, and also makes it easier to remove the reliance on the pass registry to provide certain things(e.g. the pass argument).

Differential Revision: https://reviews.llvm.org/D76659

show more ...

[mlir][Pass] Move the registration of conversion passes to tablegen

This removes the need to statically register conversion passes, and also puts all of the conversions within one centralized file.

[mlir][Pass] Move the registration of conversion passes to tablegen

This removes the need to statically register conversion passes, and also puts all of the conversions within one centralized file.

Differential Revision: https://reviews.llvm.org/D76658

show more ...

[mlir][NFC] Replace all usages of PatternMatchResult with LogicalResult

This also replaces usages of matchSuccess/matchFailure with success/failure respectively.

Differential Revision: https://revi

[mlir][NFC] Replace all usages of PatternMatchResult with LogicalResult

This also replaces usages of matchSuccess/matchFailure with success/failure respectively.

Differential Revision: https://reviews.llvm.org/D76313

show more ...

[mlir][Vector] Mostly-NFC - Restructure options for lowering to LLVM Matrix Intrinsics

Summary:
This revision restructures the calling of vector transforms to make it more flexible to ask for loweri

[mlir][Vector] Mostly-NFC - Restructure options for lowering to LLVM Matrix Intrinsics

Summary:
This revision restructures the calling of vector transforms to make it more flexible to ask for lowering through LLVM matrix intrinsics.
This also makes sure we bail out in degenerate cases (i.e. 1) in which LLVM complains about not being able to scalarize.

Differential Revision: https://reviews.llvm.org/D76266

show more ...

[mlir][NFC] Renamed VectorOps to Vector

Summary: Renamed VectorOps to Vector to avoid the redundant Ops suffix.

Differential Revision: https://reviews.llvm.org/D76317

[mlir] Remove the PatternState class and simplify PatternMatchResult.

Summary: PatternState was a mechanism to pass state between the match and rewrite calls of a RewritePattern. With the rise of ma

[mlir] Remove the PatternState class and simplify PatternMatchResult.

Summary: PatternState was a mechanism to pass state between the match and rewrite calls of a RewritePattern. With the rise of matchAndRewrite, this class is unused and unnecessary. This revision removes PatternState and simplifies PatternMatchResult to just be a LogicalResult. A future revision will replace all usages of PatternMatchResult/matchSuccess/matchFailure with LogicalResult equivalents.

Differential Revision: https://reviews.llvm.org/D76202

show more ...

[mlir][Vector]Lower vector.contract to llvm.intr.matrix_multiply

Summary:
This revision adds lowering of vector.contract to llvm.intr.matrix_multiply.
Note that there is currently a mismatch between

[mlir][Vector]Lower vector.contract to llvm.intr.matrix_multiply

Summary:
This revision adds lowering of vector.contract to llvm.intr.matrix_multiply.
Note that there is currently a mismatch between the MLIR vector dialect which
expects row-major layout and the LLVM matrix intrinsics which expect column
major layout.

As a consequence, we currently only match a vector.contract with indexing maps
that express column-major matrix multiplication.
Other cases would require additional transposes and it is better to wait for
LLVM intrinsics to provide a per-operation attribute that would specify which
layout is expected.

A separate integration test, not submitted to MLIR core, has independently
verified that correct execution occurs on a 2x2x2 matrix multiplication.

Differential Revision: https://reviews.llvm.org/D76014

show more ...

[mlir] [VectorOps,LinAlg] Remove direct LLVM lowering for vector.broadcast

Summary:
The direct lowering of vector.broadcast into LLVM has been replaced by
progressive lowering into elementary vector

[mlir] [VectorOps,LinAlg] Remove direct LLVM lowering for vector.broadcast

Summary:
The direct lowering of vector.broadcast into LLVM has been replaced by
progressive lowering into elementary vector ops. This also required a
small refactoring of a llvm.mlir test that used a direct vector.broadcast
operator (just to define a matmul).

Reviewers: nicolasvasilache, andydavis1, rriddle

Reviewed By: nicolasvasilache

Subscribers: mehdi_amini, rriddle, jpienaar, burmako, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, Joonsoo, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D76143

show more ...

[mlir] [VectorOps] Progressively lower vector.outerproduct to LLVM

Summary:
This replaces the direct lowering of vector.outerproduct to LLVM with progressive lowering into elementary vectors ops to

[mlir] [VectorOps] Progressively lower vector.outerproduct to LLVM

Summary:
This replaces the direct lowering of vector.outerproduct to LLVM with progressive lowering into elementary vectors ops to avoid having the similar lowering logic at several places.

NOTE1: with the new progressive rule, the lowered llvm is slightly more elaborate than with the direct lowering, but the generated assembly is just as optimized; still if we want to stay closer to the original, we should add a "broadcast on extract" to shuffle rewrite (rather than special cases all the lowering steps)

NOTE2: the original outerproduct lowering code should now be removed but some linalg test work directly on vector and contain some dead code, so this requires another CL

Reviewers: nicolasvasilache, andydavis1

Reviewed By: nicolasvasilache, andydavis1

Subscribers: mehdi_amini, rriddle, jpienaar, burmako, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, Joonsoo, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D75956

show more ...

[mlir][Vector] Add a vector.matrix_multiply op on 1-D vectors

Summary: This op mirrors the llvm.intr counterpart and allows lowering + type conversions in a progressive fashion.

Differential Revisi

[mlir][Vector] Add a vector.matrix_multiply op on 1-D vectors

Summary: This op mirrors the llvm.intr counterpart and allows lowering + type conversions in a progressive fashion.

Differential Revision: https://reviews.llvm.org/D75775

show more ...