Fix typo "tranpose" (#124929)

[mlir][NFC] Minor cleanup around `ModuleOp` usage (#110498)

Use `moduleOp.getBody()` instead of `moduleOp.getBodyRegion().front()`.

[mlir][sparse] use a consistent order between [dis]assembleOp and sto… (#84079)

…rage layout.

[mlir][sparse] code cleanup (using inferred type to construct to_[buf… (#83361)

…fer] op).

Apply clang-tidy fixes for modernize-loop-convert in SparseGPUCodegen.cpp (NFC)

[mlir][sparse] Implement parsing n out of m (#79935)

1. Add parsing methods for block[n, m].
2. Encode n and m with the newly extended 64-bit LevelType enum.
3. Update 2:4 methods names/comments t

[mlir][sparse] Implement parsing n out of m (#79935)

1. Add parsing methods for block[n, m].
2. Encode n and m with the newly extended 64-bit LevelType enum.
3. Update 2:4 methods names/comments to n:m.

show more ...

[MLIR] Fix crash in AffineMap::replace for zero result maps (#80930)

Fix obvious bug in AffineMap::replace for the case of zero result maps.
Extend/complete inferExprsFromList to work with empty ex

[MLIR] Fix crash in AffineMap::replace for zero result maps (#80930)

Fix obvious bug in AffineMap::replace for the case of zero result maps.
Extend/complete inferExprsFromList to work with empty expression lists.

show more ...

[mlir][sparse] recognize NVidia 2:4 type for matmul (#76758)

This removes the temporary DENSE24 attribute and replaces it with proper
recognition of dense to 24 conversion. The compressionh will be

[mlir][sparse] recognize NVidia 2:4 type for matmul (#76758)

This removes the temporary DENSE24 attribute and replaces it with proper
recognition of dense to 24 conversion. The compressionh will be
performed on the device prior to performing the matrix mult. Note that
we no longer need to start with the linalg version, we can lift this to
the proper named linalg op. Also renames some files into more consistent
names.

show more ...

[mlir][SCF] `scf.parallel`: Make reductions part of the terminator (#75314)

This commit makes reductions part of the terminator. Instead of
`scf.yield`, `scf.reduce` now terminates the body of `scf

[mlir][SCF] `scf.parallel`: Make reductions part of the terminator (#75314)

This commit makes reductions part of the terminator. Instead of
`scf.yield`, `scf.reduce` now terminates the body of `scf.parallel` ops.
`scf.reduce` may contain an arbitrary number of reductions, with one
region per reduction.

Example:
```mlir
%init = arith.constant 0.0 : f32
%r:2 = scf.parallel (%iv) = (%lb) to (%ub) step (%step) init (%init, %init)
-> f32, f32 {
%elem_to_reduce1 = load %buffer1[%iv] : memref<100xf32>
%elem_to_reduce2 = load %buffer2[%iv] : memref<100xf32>
scf.reduce(%elem_to_reduce1, %elem_to_reduce2 : f32, f32) {
^bb0(%lhs : f32, %rhs: f32):
%res = arith.addf %lhs, %rhs : f32
scf.reduce.return %res : f32
}, {
^bb0(%lhs : f32, %rhs: f32):
%res = arith.mulf %lhs, %rhs : f32
scf.reduce.return %res : f32
}
}
```

`scf.reduce` operations can no longer be interleaved with other ops in
the body of `scf.parallel`. This simplifies the op and makes it possible
to assign the `RecursiveMemoryEffects` trait to `scf.reduce`. (This was
not possible before because the op was not a terminator, causing the op
to be DCE'd.)

show more ...

[mlir][SparseTensor][NFC] Remove `isNestedIn` helper function (#75729)

Use `Region::findAncestorBlockInRegion` instead of a custom IR
traversal.

[NFC][mlir][sparse] remove redundant parameter. (#75551)

[mlir][sparse] refactor utilities into transform/utils dir (#75250)

Separates actual transformation files from supporting utility files in
the transforms directory. Includes a bazel overlay fix for

[mlir][sparse] refactor utilities into transform/utils dir (#75250)

Separates actual transformation files from supporting utility files in
the transforms directory. Includes a bazel overlay fix for the build (as
well as a bit of cleanup of that file to be less verbose and more
flexible).

show more ...

[mlir][SparseTensor] Fix invalid IR in `ForallRewriter` pattern (#74547)

The `ForallRewriter` pattern used to generate invalid IR:
```
mlir/test/Dialect/SparseTensor/GPU/gpu_combi.mlir:0:0: error:

[mlir][SparseTensor] Fix invalid IR in `ForallRewriter` pattern (#74547)

The `ForallRewriter` pattern used to generate invalid IR:
```
mlir/test/Dialect/SparseTensor/GPU/gpu_combi.mlir:0:0: error: 'scf.for' op expects region #0 to have 0 or 1 blocks
mlir/test/Dialect/SparseTensor/GPU/gpu_combi.mlir:0:0: note: see current operation:
"scf.for"(%8, %2, %9) ({
^bb0(%arg5: index):
// ...
"scf.yield"() : () -> ()
^bb1(%10: index): // no predecessors
"scf.yield"() : () -> ()
}) : (index, index, index) -> ()
```

This commit fixes tests such as
`mlir/test/Dialect/SparseTensor/GPU/gpu_combi.mlir` when verifying the
IR after each pattern application (#74270).

show more ...

[mlir][sparsifier] fix `isAdmissibleBSR` (#72195)

Fixes https://github.com/llvm/llvm-project/issues/72194.

[mlir][sparse][gpu] cleanup GPUDataTransferStrategy (#71615)

The flag seems to be doing practically the same thing for zero cost and
pinned dma. In addition, the register host is not truly the righ

[mlir][sparse][gpu] cleanup GPUDataTransferStrategy (#71615)

The flag seems to be doing practically the same thing for zero cost and
pinned dma. In addition, the register host is not truly the right zero
cost mechanism according to Thomas. So we are simplifying the setup for
now, until we have a better definition for what to implement and test.

https://github.com/llvm/llvm-project/issues/64316

show more ...

Changed the phrase sparse-compiler to sparsifier in comments (#71578)

When the Powers That Be decided that the name "sparse compiler" should
be changed to "sparsifier", we negected to change some o

Changed the phrase sparse-compiler to sparsifier in comments (#71578)

When the Powers That Be decided that the name "sparse compiler" should
be changed to "sparsifier", we negected to change some of the comments
in the code; this pull request completes the name change.

show more ...

[mlir][sparse] support BSR for cuSPARSE (libgen path only) (#69646)

[mlir][sparse][gpu] add CSC to libgen GPU sparsification using cuSparse (#67713)

Add CSC, but also adds BSR as a future format. Coming soon!

[mlir][sparse] rename sparse_tensor.(un)pack to sparse_tensor.(dis)as… (#67717)

…semble

Pack/Unpack are overridden in many other places, rename the operations
to avoid confusion.

[mlir][sparse][gpu] free all buffers allocated for spGEMM (#66813)

Yup, a bit of an oversight ;-)

[mlir][Interfaces] `LoopLikeOpInterface`: Support ops with multiple regions (#66754)

This commit implements `LoopLikeOpInterface` on `scf.while`. This
enables LICM (and potentially other transforms

[mlir][Interfaces] `LoopLikeOpInterface`: Support ops with multiple regions (#66754)

This commit implements `LoopLikeOpInterface` on `scf.while`. This
enables LICM (and potentially other transforms) on `scf.while`.

`LoopLikeOpInterface::getLoopBody()` is renamed to `getLoopRegions` and
can now return multiple regions.

Also fix a bug in the default implementation of
`LoopLikeOpInterface::isDefinedOutsideOfLoop()`, which returned "false"
for some values that are defined outside of the loop (in a nested op, in
such a way that the value does not dominate the loop). This interface is
currently only used for LICM and there is no way to trigger this bug, so
no test is added.

show more ...

[mlir][sparse][gpu] minor code cleanup for sparse gpu ops

Consistent order of ops and related methods.
Also, renamed SpGEMMGetSizeOp to SpMatGetSizeOp
since this is a general utility for sparse matr

[mlir][sparse][gpu] minor code cleanup for sparse gpu ops

Consistent order of ops and related methods.
Also, renamed SpGEMMGetSizeOp to SpMatGetSizeOp
since this is a general utility for sparse matrices,
not specific to GEMM ops only.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D157922

show more ...

[mlir][sparse][gpu] sparsifier GPU libgen for SpGEMM in cuSparse

With working integration end-to-end test

Reviewed By: K-Wu

Differential Revision: https://reviews.llvm.org/D157652

[mlir][sparse][gpu] introduce flag that controls host to device copy strategies (regular dma default)

Differential Revision: https://reviews.llvm.org/D155352

[mlir][sparse][gpu] add 2:4 spmm prune_and_check flag

Differential Revision: https://reviews.llvm.org/D155909