Fix typo "tranpose" (#124929)

[mlir][Vector] Add support for poison indices to `Extract/IndexOp` (#123488)

Following up on #122188, this PR adds support for poison indices to
`ExtractOp` and `InsertOp`. It also includes canonic

[mlir][Vector] Add support for poison indices to `Extract/IndexOp` (#123488)

Following up on #122188, this PR adds support for poison indices to
`ExtractOp` and `InsertOp`. It also includes canonicalization patterns
to turn extract/insert ops with poison indices into `ub.poison`.

show more ...

[mlir][vector] Canonicalize gathers/scatters with trivial offsets (#117939)

Canonicalize gathers/scatters with contiguous (i.e. [0, 1, 2, ...])
offsets into vector masked load/store ops.

[mlir][IR][NFC] Move free-standing functions to `MemRefType` (#123465)

Turn free-standing `MemRefType`-related helper functions in
`BuiltinTypes.h` into member functions.

[mlir][Vector] Support poison in `vector.shuffle` mask (#122188)

This PR extends the existing poison support in
https://mlir.llvm.org/docs/Dialects/UBOps/ by representing poison mask
values in `ve

[mlir][Vector] Support poison in `vector.shuffle` mask (#122188)

This PR extends the existing poison support in
https://mlir.llvm.org/docs/Dialects/UBOps/ by representing poison mask
values in `vector.shuffle`. Similar to LLVM (see
https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/IR/Instructions.h#L1884)
this requires defining an integer value (`-1`) to represent poison in
the `vector.shuffle` mask.

show more ...

[mlir] Guard sccp pass from crashing with different source type (#120656)

Vector::BroadCastOp expects the identical element type in folding. It
causes the crash if the different source type is give

[mlir] Guard sccp pass from crashing with different source type (#120656)

Vector::BroadCastOp expects the identical element type in folding. It
causes the crash if the different source type is given to the SCCP pass.
We need to guard the pass from crashing if the nonidentical element type
is given, but still compatible. (e.g. index vs integer type)

https://github.com/llvm/llvm-project/issues/120193

show more ...

[MemRef] Migrate away from PointerUnion::{is,get} (NFC) (#120382)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dy

[MemRef] Migrate away from PointerUnion::{is,get} (NFC) (#120382)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[mlir][Vector] Add support for 0-d shapes in extract-shape_cast folder (#116650)

The extract <-> shape cast folder was conservatively asserting and
failing on 0-d vectors. This pr fixes this.

This

[mlir][Vector] Add support for 0-d shapes in extract-shape_cast folder (#116650)

The extract <-> shape cast folder was conservatively asserting and
failing on 0-d vectors. This pr fixes this.

This pr also adds more tests for 0d cases and updates related tests to
better reflect what they test.

show more ...

[MLIR] Move warp_execute_on_lane_0 from vector to gpu (#116994)

Please see the related RFC here:
https://discourse.llvm.org/t/rfc-move-execute-on-lane-0-from-vector-to-gpu-dialect/82989.

This pa

[MLIR] Move warp_execute_on_lane_0 from vector to gpu (#116994)

Please see the related RFC here:
https://discourse.llvm.org/t/rfc-move-execute-on-lane-0-from-vector-to-gpu-dialect/82989.

This patch does exactly one thing - moves the op to gpu.

show more ...

[mlir][vector] Fix 0-d vector transfer mask inference (#116526)

When inferring the mask of a transfer operation that results in a single `i1` element,
we could represent it using either `vector<i1

[mlir][vector] Fix 0-d vector transfer mask inference (#116526)

When inferring the mask of a transfer operation that results in a single `i1` element,
we could represent it using either `vector<i1>` or vector<1xi1>. To avoid type mismatches,
this PR updates the mask inference logic to consistently generate `vector<1xi1>` for
these cases. We can enable 0-D masks if they are needed in the future.

See: https://github.com/llvm/llvm-project/issues/116197

show more ...

[mlir][Vector] Add vector.extract(vector.shuffle) folder (#115105)

This PR adds a folder for extracting an element from a vector shuffle.
It turns something like:

```
%shuffle = vector.shuff

[mlir][Vector] Add vector.extract(vector.shuffle) folder (#115105)

This PR adds a folder for extracting an element from a vector shuffle.
It turns something like:

```
%shuffle = vector.shuffle %a, %b [0, 8, 7, 15]
: vector<8xf32>, vector<8xf32>
%extract = vector.extract %shuffle[3] : f32 from vector<4xf32>
```

into:

```
%extract = vector.extract %b[7] : f32 from vector<8xf32>
```

show more ...

[mlir] IntegerRangeAnalysis: add support for vector type (#112292)

Treat integer range for vector type as union of ranges of individual
elements. With this semantics, most arith ops on vectors will

[mlir] IntegerRangeAnalysis: add support for vector type (#112292)

Treat integer range for vector type as union of ranges of individual
elements. With this semantics, most arith ops on vectors will work out
of the box, the only special handling needed for constants and vector
elements manipulation ops.

The end goal of these changes is to be able to optimize vectorized index
calculations.

show more ...

[MLIR][Vector] Add Lowering for vector.step (#113655)

Currently, the lowering for vector.step lives
under a folder. This is not ideal if we want
to do transformation on it and defer the
materiza

[MLIR][Vector] Add Lowering for vector.step (#113655)

Currently, the lowering for vector.step lives
under a folder. This is not ideal if we want
to do transformation on it and defer the
materizaliztion of the constants much later.

This commits adds a rewrite pattern that
could be used by using
`transform.structured.vectorize_children_and_apply_patterns`
transform dialect operation.

Moreover, the rewriter of vector.step is also
now used in -convert-vector-to-llvm pass where
it handles scalable and non-scalable types as
LLVM expects it.

As a consequence of removing the vector.step
lowering as its folder, linalg vectorization
will keep vector.step intact.

show more ...

[mlir][Vector] Fix vector.insert folder for scalar to 0-d inserts (#113828)

The current vector.insert folder tries to replace a scalar with a 0-rank
vector. This patch fixes this crash by not foldi

[mlir][Vector] Fix vector.insert folder for scalar to 0-d inserts (#113828)

The current vector.insert folder tries to replace a scalar with a 0-rank
vector. This patch fixes this crash by not folding unless they types of
the result and replacement are same.

show more ...

[mlir][vector] Remove unneeded mask restriction (#113742)

These were added when the only mapping was to LLVM.

[MLIR] Vector: turn the ExtractStridedSlice rewrite pattern from #111541 into a canonicalization (#111614)

This is a reasonable canonicalization because `extract` is more
constrained than `extract_

[MLIR] Vector: turn the ExtractStridedSlice rewrite pattern from #111541 into a canonicalization (#111614)

This is a reasonable canonicalization because `extract` is more
constrained than `extract_strided_slices`, so there is no loss of
semantics here, just lifting an op to a special-case higher/constrained
op. And the additional `shape_cast` is merely adding leading unit dims
to match the original result type.

Context: discussion on #111541. I wasn't sure how this would turn out,
but in the process of writing this PR, I discovered at least 2 bugs in
the pattern introduced in #111541, which shows the value of shared
canonicalization patterns which are exercised on a high number of
testcases.

---------

Signed-off-by: Benoit Jacob <jacob.benoit.1@gmail.com>

show more ...

[mlir][vector] Implement speculation for vector.transferx ops (#111533)

This patch implements speculation for
vector.transfer_read/vector.transfer_write ops, allowing these ops to
work with LICM.

[mlir][vector] Relax the requirements on broadcast dims (#99341)

NOTE: This is a follow-up for #97049 in which the `in_bounds` attribute
was made mandatory.

This PR updates the semantics of the

[mlir][vector] Relax the requirements on broadcast dims (#99341)

NOTE: This is a follow-up for #97049 in which the `in_bounds` attribute
was made mandatory.

This PR updates the semantics of the `in_bounds` attribute so that
broadcast dimensions are no longer required to be "in bounds".
Specifically, these xfer_read/xfer_write Ops become valid after this
change:

```mlir
%read = vector.transfer_read %A[%base1, %base2], %pad
{in_bounds = [false], permutation_map = affine_map<(d0, d1) -> (0)>}
{permutation_map = affine_map<(d0, d1) -> (0)>}
: memref<?x?xf32>, vector<9xf32>

vector.transfer_write %vec, %A[%base1, %base2],
{in_bounds = [false], permutation_map = affine_map<(d0, d1) -> (0)>}
{permutation_map = affine_map<(d0, d1) -> (0)>}
: vector<9xf32>, memref<?x?xf32>
```

Note that the value `false` merely means "may run out-of-bounds", i.e.,
the corresponding access can still be "in bounds". In fact, the folder
for xfer Ops is also updated (*) and will update the attribute value
corresponding to broadcast dims to `true` if all non-broadcast dims
are marked as "in bounds".

Note that this PR doesn't change any of the lowerings. The changes in
"SuperVectorize.cpp", "Vectorization.cpp" and "AffineMap.cpp" are simple
reverts of recent changes in #97049. Those were only meant to facilitate
making `in_bounds` mandatory and to work around the extra requirements
for broadcast dims (those requirements ere removed in this PR). All
changes in tests are also reverts of changes from #97049.

For context, here's a PR in which "broadcast" dims where forced to
always be "in-bounds":
* https://reviews.llvm.org/D102566

(*) See `foldTransferInBoundsAttribute`.

show more ...

[mlir][vector][nfc] Clean-up VectorOps.{h|cpp} (#109316)

[mlir][Vector] Verify that masked ops implement MaskableOpInterface (#108123)

This PR fixes a bug in `MaskOp::verifier` that allowed `vector.mask` to
mask operations that did not implement the Mask

[mlir][Vector] Verify that masked ops implement MaskableOpInterface (#108123)

This PR fixes a bug in `MaskOp::verifier` that allowed `vector.mask` to
mask operations that did not implement the MaskableOpInterface.

show more ...

[mlir][vector] Relax strides check for 1-element vector load/stores (#108998)

Single elememst vector load/stores are equivalent to scalar load/stores,
so they don't need memref to be contigious.

[mlir] 'dyn_cast' is deprecated: Use mlir::dyn_cast<U>() instead (NFC)

/llvm-project/mlir/lib/Dialect/Vector/IR/VectorOps.cpp:2923:29: error: 'dyn_cast' is deprecated: Use mlir::dyn_cast<U>() instea

[mlir] 'dyn_cast' is deprecated: Use mlir::dyn_cast<U>() instead (NFC)

/llvm-project/mlir/lib/Dialect/Vector/IR/VectorOps.cpp:2923:29: error: 'dyn_cast' is deprecated: Use mlir::dyn_cast<U>() instead [-Werror,-Wdeprecated-declarations]
2923 | if (auto intAttr = attr.dyn_cast<IntegerAttr>()) {
| ^
/llvm-project/mlir/include/mlir/IR/Attributes.h:184:14: note: 'dyn_cast' has been explicitly marked deprecated here
184 | U Attribute::dyn_cast() const {
| ^

show more ...

[mlir] Fix crash in `InsertOpConstantFolder` when vector.insert operand is from a llvm.mlir.constant op (#88314)

In cases where llvm.mlir.constant has an attribute with a different type than the ret

[mlir] Fix crash in `InsertOpConstantFolder` when vector.insert operand is from a llvm.mlir.constant op (#88314)

In cases where llvm.mlir.constant has an attribute with a different type than the returned type,
the folder use to create an incorrect DenseElementsAttr and crash.

Resolves #74236

show more ...

[mlir][vector] Use `DenseI64ArrayAttr` in vector.multi_reduction (#102637)

This prevents some unnecessary conversions to/from int64_t and
IntegerAttr.

[mlir][vector] Add mask elimination transform (#99314)

This adds a new transform `eliminateVectorMasks()` which aims at
removing scalable `vector.create_masks` that will be all-true at
runtime. It

[mlir][vector] Add mask elimination transform (#99314)

This adds a new transform `eliminateVectorMasks()` which aims at
removing scalable `vector.create_masks` that will be all-true at
runtime. It attempts to do this by simply pattern-matching the mask
operands (similar to some canonicalizations), if that does not lead to
an answer (is all-true? yes/no), then value bounds analysis will be used
to find the lower bound of the unknown operands. If the lower bound is
>= to the corresponding mask vector type dim, then that dimension of the
mask is all true.

Note that the pattern matching prevents expensive value-bounds analysis
in cases where the mask won't be all true.

For example:
```mlir
%mask = vector.create_mask %dynamicValue, %c2 : vector<8x4xi1>
```
From looking at `%c2` we can tell this is not going to be an all-true
mask, so we don't need to run the value-bounds analysis for
`%dynamicValue` (and can exit the transform early).

Note: Eliminating create_masks here means replacing them with all-true
constants (which will then lead to the masks folding away).

show more ...