[mlir][sparse_tensor] Implement bufferization interface for `foreach` (#85183)

This commit fixes a memory leak in `sparse_codegen_foreach.mlir`. The
bufferization inserted a copy for the operand of

[mlir][sparse_tensor] Implement bufferization interface for `foreach` (#85183)

This commit fixes a memory leak in `sparse_codegen_foreach.mlir`. The
bufferization inserted a copy for the operand of `sparse_tensor.foreach`
because it conservatively assumed that the op writes to the operand.

show more ...

[mlir][sparse] reuse tensor.insert operation to insert elements into … (#84987)

…a sparse tensor.

Changed the phrase sparse-compiler to sparsifier in comments (#71578)

When the Powers That Be decided that the name "sparse compiler" should
be changed to "sparsifier", we negected to change some o

Changed the phrase sparse-compiler to sparsifier in comments (#71578)

When the Powers That Be decided that the name "sparse compiler" should
be changed to "sparsifier", we negected to change some of the comments
in the code; this pull request completes the name change.

show more ...

[mlir][sparse] rename sparse_tensor.(un)pack to sparse_tensor.(dis)as… (#67717)

…semble

Pack/Unpack are overridden in many other places, rename the operations
to avoid confusion.

[mlir][sparse] Better error handling when bufferizing sparse_tensor ops

sparse_tensor ops cannot be bufferized with One-Shot Bufferize. (They can only be analyzed.) The sparse compiler does the actu

[mlir][sparse] Better error handling when bufferizing sparse_tensor ops

sparse_tensor ops cannot be bufferized with One-Shot Bufferize. (They can only be analyzed.) The sparse compiler does the actual lowering to memref. Produce a proper error message instead of crashing.

This fixes #61311.

Differential Revision: https://reviews.llvm.org/D158728

show more ...

[mlir][bufferization] Generalize getAliasingOpResults to getAliasingValues

This revision is needed to support bufferization of `cf.br`/`cf.cond_br`. It will also be useful for better analysis of loo

[mlir][bufferization] Generalize getAliasingOpResults to getAliasingValues

This revision is needed to support bufferization of `cf.br`/`cf.cond_br`. It will also be useful for better analysis of loop ops.

This revision generalizes `getAliasingOpResults` to `getAliasingValues`. An OpOperand can now not only alias with OpResults but also with BlockArguments. In the case of `cf.br` (will be added in a later revision): a `cf.br` operand will alias with the corresponding argument of the destination block.

If an op does not implement the `BufferizableOpInterface`, the analysis in conservative. It previously assumed that an OpOperand may alias with each OpResult. It now assumes that an OpOperand may alias with each OpResult and each BlockArgument of the entry block.

Differential Revision: https://reviews.llvm.org/D157957

show more ...

[mlir][sparse] make UnpackOp return the actual filled length of unpacked memory

This might simplify frontend implementation by avoiding recomputation for the same value.

Reviewed By: aartbik

Diffe

[mlir][sparse] make UnpackOp return the actual filled length of unpacked memory

This might simplify frontend implementation by avoiding recomputation for the same value.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D154244

show more ...

[mlir][sparse] extend unpack operation to unpack arbitrary encodings.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D151174

[mlir][sparse] extend unpack operation to support unpacking a batched COO type

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D149103

[mlir][sparse] extend pack operation to support packing a batched COO type

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148670

[mlir][sparse] Renaming "pointer/index" to "position/coordinate"

The old "pointer/index" names often cause confusion since these names clash with names of unrelated things in MLIR; so this change re

[mlir][sparse] Renaming "pointer/index" to "position/coordinate"

The old "pointer/index" names often cause confusion since these names clash with names of unrelated things in MLIR; so this change rectifies this by changing everything to use "position/coordinate" terminology instead.

In addition to the basic terminology, there have also been various conventions for making certain distinctions like: (1) the overall storage for coordinates in the sparse-tensor, vs the particular collection of coordinates of a given element; and (2) particular coordinates given as a `Value` or `TypedValue<MemRefType>`, vs particular coordinates given as `ValueRange` or similar. I have striven to maintain these distinctions
as follows:

* "p/c" are used for individual position/coordinate values, when there is no risk of confusion. (Just like we use "d/l" to abbreviate "dim/lvl".)

* "pos/crd" are used for individual position/coordinate values, when a longer name is helpful to avoid ambiguity or to form compound names (e.g., "parentPos"). (Just like we use "dim/lvl" when we need a longer form of "d/l".)

I have also used these forms for a handful of compound names where the old name had been using a three-letter form previously, even though a longer form would be more appropriate. I've avoided renaming these to use a longer form purely for expediency sake, since changing them would require a cascade of other renamings. They should be updated to follow the new naming scheme, but that can be done in future patches.

* "coords" is used for the complete collection of crd values associated with a single element. In the runtime library this includes both `std::vector` and raw pointer representations. In the compiler, this is used specifically for buffer variables with C++ type `Value`, `TypedValue<MemRefType>`, etc.

The bare form "coords" is discouraged, since it fails to make the dim/lvl distinction; so the compound names "dimCoords/lvlCoords" should be used instead. (Though there may exist a rare few cases where is is appropriate to be intentionally ambiguous about what coordinate-space the coords live in; in which case the bare "coords" is appropriate.)

There is seldom the need for the pos variant of this notion. In most circumstances we use the term "cursor", since the same buffer is reused for a 'moving' pos-collection.

* "dcvs/lcvs" is used in the compiler as the `ValueRange` analogue of "dimCoords/lvlCoords". (The "vs" stands for "`Value`s".) I haven't found the need for it, but "pvs" would be the obvious name for a pos-`ValueRange`.

The old "ind"-vs-"ivs" naming scheme does not seem to have been sustained in more recent code, which instead prefers other mnemonics (e.g., adding "Buf" to the end of the names for `TypeValue<MemRefType>`). I have cleaned up a lot of these to follow the "coords"-vs-"cvs" naming scheme, though haven't done an exhaustive cleanup.

* "positions/coordinates" are used for larger collections of pos/crd values; in particular, these are used when referring to the complete sparse-tensor storage components.

I also prefer to use these unabbreviated names in the documentation, unless there is some specific reason why using the abbreviated forms helps resolve ambiguity.

In addition to making this terminology change, this change also does some cleanup along the way:
* correcting the dim/lvl terminology in certain places.
* adding `const` when it requires no other code changes.
* miscellaneous cleanup that was entailed in order to make the proper distinctions. Most of these are in CodegenUtils.{h,cpp}

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D144773

show more ...

[mlir][sparse] implement lowering rules for sparse_tensor::unpack

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D143672

[mlir][bufferization] Improve aliasing OpOperand/OpResult property

`getAliasingOpOperands`/`getAliasingOpResults` now encodes OpOperand/OpResult, buffer relation and a degree of certainty. E.g.:
```

[mlir][bufferization] Improve aliasing OpOperand/OpResult property

`getAliasingOpOperands`/`getAliasingOpResults` now encodes OpOperand/OpResult, buffer relation and a degree of certainty. E.g.:
```
// aliasingOpOperands(%r) = {(%t, EQUIV, DEFINITE)}
// aliasingOpResults(%t) = {(%r, EQUIV, DEFINITE)}
%r = tensor.insert %f into %t[%idx] : tensor<?xf32>

// aliasingOpOperands(%r) = {(%t0, EQUIV, MAYBE), (%t1, EQUIV, MAYBE)}
// aliasingOpResults(%t0) = {(%r, EQUIV, MAYBE)}
// aliasingOpResults(%t1) = {(%r, EQUIV, MAYBE)}
%r = arith.select %c, %t0, %t1 : tensor<?xf32>
```

`BufferizableOpInterface::bufferRelation` is removed, as it is now part of `getAliasingOpOperands`/`getAliasingOpResults`.

This change allows for better analysis, in particular wrt. equivalence. This allows additional optimizations and better error checking (which is sometimes overly conservative). Examples:

* EmptyTensorElimination can eliminate `tensor.empty` inside `scf.if` blocks. This requires a modeling of equivalence: It is not a per-OpResult property anymore. Instead, it can be specified for each OpOperand and OpResult. This is important because `tensor.empty` may be eliminated only if all values on the SSA use-def chain to the final consumer (`tensor.insert_slice`) are equivalent.
* The detection of "returning allocs from a block" can be improved. (Addresses a TODO in `assertNoAllocsReturned`.) This allows us to bufferize IR such as "yielding a `tensor.extract_slice` result from an `scf.if` branch", which currently fails to bufferize because the alloc detection is too conservative.
* Better bufferization of loops. Aliases of the iter_arg can be yielded (even if they are not equivalent) without having to realloc and copy the entire buffer on each iteration.

The above-mentioned examples are not yet implemented with this change. This change just improves the BufferizableOpInterface, its implementations and related helper functions, so that better aliasing information is available for each op.

Differential Revision: https://reviews.llvm.org/D142129

show more ...

[mlir][sparse] implement bufferizableOpInterface for sparse_tensor.pack operation

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D143281

[mlir][bufferization][NFC] Rename getAliasingOpOperand/getAliasingOpResult

* `getAliasingOpOperand` => `getAliasingOpOperands`
* `getAliasingOpResult` => `getAliasingOpResults`

Also a few minor cod

[mlir][bufferization][NFC] Rename getAliasingOpOperand/getAliasingOpResult

* `getAliasingOpOperand` => `getAliasingOpOperands`
* `getAliasingOpResult` => `getAliasingOpResults`

Also a few minor code cleanups and better documentation.

Differential Revision: https://reviews.llvm.org/D142979

show more ...

[mlir][bufferization] Generalize and rename isMemoryWrite

The name of the method was confusing. It is bufferizesToMemoryWrite, but from the perspective of OpResults.

`bufferizesToMemoryWrite(OpResu

[mlir][bufferization] Generalize and rename isMemoryWrite

The name of the method was confusing. It is bufferizesToMemoryWrite, but from the perspective of OpResults.

`bufferizesToMemoryWrite(OpResult)` now supports ops with regions that do not have aliasing OpOperands (such as `scf.if`). These ops no longer need to implement `isMemoryWrite`.

Differential Revision: https://reviews.llvm.org/D141684

show more ...

[mlir][sparse] Implement BufferizableOpInterface for additional ops

The handling of unknown ops will be tightened in a subsequent change. All sparse_tensor ops should implement BufferizableOpInterfa

[mlir][sparse] Implement BufferizableOpInterface for additional ops

The handling of unknown ops will be tightened in a subsequent change. All sparse_tensor ops should implement BufferizableOpInterface, otherwise, they are treated as "unknown" and additional buffer allocs/copies may be inserted around them.

Differential Revision: https://reviews.llvm.org/D142005

show more ...

[mlir][sparse] fix bufferizableOpInterface for InsertOp

Address comments in D136372

Reviewed By: aartbik, springerm

Differential Revision: https://reviews.llvm.org/D136510

[mlir][sparse] attach bufferizableOpInterface to InsertOp

To fix D136286

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136372

[mlir][bufferization][NFC] Move sparse_tensor.release to bufferization dialect

This op used to belong to the sparse dialect, but there are use cases for dense bufferization as well. (E.g., when a te

[mlir][bufferization][NFC] Move sparse_tensor.release to bufferization dialect

This op used to belong to the sparse dialect, but there are use cases for dense bufferization as well. (E.g., when a tensor alloc is returned from a function and should be deallocated at the call site.) This change moves the op to the bufferization dialect, which now has an `alloc_tensor` and a `dealloc_tensor` op.

Differential Revision: https://reviews.llvm.org/D129985

show more ...

[mlir][sparse][bufferize] Implement BufferizableOpInterface

Only the analysis part of the interface is implemented. The bufferization itself is performed by the SparseTensorConversion pass.

Differe

[mlir][sparse][bufferize] Implement BufferizableOpInterface

Only the analysis part of the interface is implemented. The bufferization itself is performed by the SparseTensorConversion pass.

Differential Revision: https://reviews.llvm.org/D128138

show more ...