[mlir] Fix conflict of user defined reserved functions with internal prototypes (#123378)

On lowering from `memref` to LLVM, `malloc` and other intrinsic
functions from `libc` will be declared in t

[mlir] Fix conflict of user defined reserved functions with internal prototypes (#123378)

On lowering from `memref` to LLVM, `malloc` and other intrinsic
functions from `libc` will be declared in the current module. User's
redefinition of these reserved functions will poison the internal
analysis with wrong prototype. This patch adds assertion on the found
function's type and reports if it mismatch with the intended type.

Related to #120950


---------

Co-authored-by: Luohao Wang <Luohaothu@users.noreply.github.com>

show more ...

[mlir][IR][NFC] Move free-standing functions to `MemRefType` (#123465)

Turn free-standing `MemRefType`-related helper functions in
`BuiltinTypes.h` into member functions.

[Conversion] Migrate away from PointerUnion::{is,get} (NFC) (#122421)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() an

[Conversion] Migrate away from PointerUnion::{is,get} (NFC) (#122421)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[mlir][Vector] Clean up `populateVectorToLLVMConversionPatterns` (#119975)

Clean up `populateVectorToLLVMConversionPatterns` so that it populates
only conversion patterns. All rewrite patterns that

[mlir][Vector] Clean up `populateVectorToLLVMConversionPatterns` (#119975)

Clean up `populateVectorToLLVMConversionPatterns` so that it populates
only conversion patterns. All rewrite patterns that do not lower to LLVM
should be populated into a separate greedy pattern rewrite.

The current combination of rewrite patterns and conversion patterns
triggered an edge case when merging the 1:1 and 1:N dialect conversions.

Depends on #119973.

show more ...

[mlir][Vector] Fix vector.extract lowering to llvm for 0-d vectors (#117731)

The current implementation of lowering to llvm for vector.extract
incorrectly assumes that if the number of indices is ze

[mlir][Vector] Fix vector.extract lowering to llvm for 0-d vectors (#117731)

The current implementation of lowering to llvm for vector.extract
incorrectly assumes that if the number of indices is zero, the operation
can be folded away. This PR removes this condition and relies on the
folder to do it instead.

This PR also unifies the logic for scalar extracts and slice extracts,
which as a side effect also enables vector.extract lowering for n-d
vector.extract with dynamic inner most dimension. (This was only
prevented by a conservative check in the old implementation)

show more ...

[MLIR][Vector] Add Lowering for vector.step (#113655)

Currently, the lowering for vector.step lives
under a folder. This is not ideal if we want
to do transformation on it and defer the
materiza

[MLIR][Vector] Add Lowering for vector.step (#113655)

Currently, the lowering for vector.step lives
under a folder. This is not ideal if we want
to do transformation on it and defer the
materizaliztion of the constants much later.

This commits adds a rewrite pattern that
could be used by using
`transform.structured.vectorize_children_and_apply_patterns`
transform dialect operation.

Moreover, the rewriter of vector.step is also
now used in -convert-vector-to-llvm pass where
it handles scalable and non-scalable types as
LLVM expects it.

As a consequence of removing the vector.step
lowering as its folder, linalg vectorization
will keep vector.step intact.

show more ...

Fix typo "instrinsic" (#112899)

[mlir][NFC] Mark type converter in `populate...` functions as `const` (#111250)

This commit marks the type converter in `populate...` functions as
`const`. This is useful for debugging.

Patterns

[mlir][NFC] Mark type converter in `populate...` functions as `const` (#111250)

This commit marks the type converter in `populate...` functions as
`const`. This is useful for debugging.

Patterns already take a `const` type converter. However, some
`populate...` functions do not only add new patterns, but also add
additional type conversion rules. That makes it difficult to find the
place where a type conversion was added in the code base. With this
change, all `populate...` functions that only populate pattern now have
a `const` type converter. Programmers can then conclude from the
function signature that these functions do not register any new type
conversion rules.

Also some minor cleanups around the 1:N dialect conversion
infrastructure, which did not always pass the type converter as a
`const` object internally.

show more ...

[mlir][vector] Support for extracting 1-element vectors in VectorExtractOpConversion (#107549)

This patch adds support for converting `vector.extract` that extract
1-element vectors into LLVM, fixi

[mlir][vector] Support for extracting 1-element vectors in VectorExtractOpConversion (#107549)

This patch adds support for converting `vector.extract` that extract
1-element vectors into LLVM, fixing a crash in such cases.
E.g., `vector.extract %1[0]: vector<1xf32> from vector<2xf32>`. Fix
#61372.

show more ...

[mlir][vector] Use `DenseI64ArrayAttr` for shuffle masks (#101163)

Follow on from #100997. This again removes from boilerplate conversions
to/from IntegerAttr and int64_t (otherwise, this is a NFC).

[mlir][vector] Propagate scalability to gather/scatter ptrs vector (#97584)

In convert-vector-to-llvm the first operand (vector of pointers holding
all memory addresses to read) to the masked.gathe

[mlir][vector] Propagate scalability to gather/scatter ptrs vector (#97584)

In convert-vector-to-llvm the first operand (vector of pointers holding
all memory addresses to read) to the masked.gather (and scatter)
intrinsic has a fixed vector type.

This may result in intrinsics where the scalable flag has been dropped:
```
%0 = llvm.intr.masked.gather %1, %2, %3 {alignment = 4 : i32}
: (!llvm.vec<4 x ptr>, vector<[4]xi1>, vector<[4]xi32>) -> vector<[4]xi32>
```
Fortunately the operand is overloaded on the result type so we end up
with the correct IR when lowering to LLVM, but this is still incorrect.
This patch fixes it by propagating scalability.

show more ...

[mlir][vector] Add vector.step operation (#96776)

This patch adds a new vector.step operation to the Vector dialect. It
produces a linear sequence of index values from 0 to N, where N is the
numbe

[mlir][vector] Add vector.step operation (#96776)

This patch adds a new vector.step operation to the Vector dialect. It
produces a linear sequence of index values from 0 to N, where N is the
number of elements in the result vector, and can be used to create
vectors of indices.

It supports both fixed-width and scalable vectors. For fixed the
canonical representation is `arith.constant dense<[0, .., N]>`. A
scalable step cannot be represented as a constant and is lowered to the
`llvm.experimental.stepvector` intrinsic [1].

This op enables scalable vectorization of linalg.index ops, see #96778. It can
also be used in the SparseVectorizer in-place of lower-level stepvector
intrinsic, see [2] (patch to follow).

[1] https://llvm.org/docs/LangRef.html#llvm-experimental-stepvector-intrinsic
[2] https://github.com/llvm/llvm-project/blob/acf675b63f9426e61aac2155e29280f7d21f9421/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp#L385-L388

show more ...

[mlir][vector] Add `vector.from_elements` op (#95938)

This commit adds a new operation to the vector dialect:
`vector.from_elements`

The op constructs a new vector from a given list of scalar va

[mlir][vector] Add `vector.from_elements` op (#95938)

This commit adds a new operation to the vector dialect:
`vector.from_elements`

The op constructs a new vector from a given list of scalar values. It is
similar to `tensor.from_elements`.
```mlir
%0 = vector.from_elements %a, %b, %c, %a, %a, %a : vector<2x3xf32>
```

Constructing a new vector from elements was tedious before this op
existed: a typical way was to define an `arith.constant ... :
vector<...>`, followed by a chain of `vector.insert`.

Folders/canonicalizations are added that can fold `vector.extract` ops
and convert the `vector.from_elements` op into a `vector.splat` op.

The LLVM lowering generates an `llvm.mlir.undef`, followed by a sequence
of scalar insertions in the form of `llvm.insertelement`. Only 0-D and
1-D vectors are currently supported in the LLVM lowering.

show more ...

[MLIR][VectorToLLVM] Handle scalable dim in createVectorLengthValue() (#93361)

LLVM's Vector Predication Intrinsics require an explicit vector length
parameter:
https://llvm.org/docs/LangRef.html#

[MLIR][VectorToLLVM] Handle scalable dim in createVectorLengthValue() (#93361)

LLVM's Vector Predication Intrinsics require an explicit vector length
parameter:
https://llvm.org/docs/LangRef.html#vector-predication-intrinsics.

For a scalable vector type, this should be caculated as VectorScaleOp
multiplied by base vector length, e.g.: for <[4]xf32> we should return:
vscale * 4.

show more ...

[mlir][vector] Add 1D vector.deinterleave lowering (#93042)

This patch implements the lowering of vector.deinterleave
for 1D vectors.

For fixed vector types, the operation is lowered to two
ll

[mlir][vector] Add 1D vector.deinterleave lowering (#93042)

This patch implements the lowering of vector.deinterleave
for 1D vectors.

For fixed vector types, the operation is lowered to two
llvm shufflevector operations. One for even indexed
elements and the other for odd indexed elements. A poison
operation is used to satisfy the parameters of the
shufflevector parameters.

For scalable vectors, the llvm vector.deinterleave2
intrinsic is used for lowering. As such the results
found by extraction and used to form the result
struct for the intrinsic.

show more ...

Move several vector intrinsics out of experimental namespace (#88748)

This patch is moving out following intrinsics:
* vector.interleave2/deinterleave2
* vector.reverse
* vector.splice

from th

Move several vector intrinsics out of experimental namespace (#88748)

This patch is moving out following intrinsics:
* vector.interleave2/deinterleave2
* vector.reverse
* vector.splice

from the experimental namespace.

All these intrinsics exist in LLVM for more than a year now, and are
widely used, so should not be considered as experimental.

show more ...

Switch member calls to `isa/dyn_cast/cast/...` to free function calls. (#89356)

This change cleans up call sites. Next step is to mark the member
functions deprecated.

See https://mlir.llvm.org/

Switch member calls to `isa/dyn_cast/cast/...` to free function calls. (#89356)

This change cleans up call sites. Next step is to mark the member
functions deprecated.

See https://mlir.llvm.org/deprecation and
https://discourse.llvm.org/t/preferred-casting-style-going-forward.

show more ...

[mlir][Vector] Fix n-D vector.extract/insert lowering to LLVM (#87591)

The lowering of n-D vector.extract/insert ops to LLVM is not supported
but if one of these accidentally reaches the vector-to-

[mlir][Vector] Fix n-D vector.extract/insert lowering to LLVM (#87591)

The lowering of n-D vector.extract/insert ops to LLVM is not supported
but if one of these accidentally reaches the vector-to-llvm conversion
patterns, we end up with a kind of puzzling crash. This PR fixes that
crash and gracefully bails out in those cases.

show more ...

[mlir][vector][print] do not append newline to printing pure strings (#83213)

Since the vector.print str provides no punctuation control, it is
slightly more flexible to let the client of this oper

[mlir][vector][print] do not append newline to printing pure strings (#83213)

Since the vector.print str provides no punctuation control, it is
slightly more flexible to let the client of this operation decide
whether there should be a trailing newline. This allows for printing
like

vector.print str "nse = "
vector.print %nse : index

as

nse = 42

show more ...

[mlir][VectorOps] Add conversion of 1-D vector.interleave ops to LLVM (#80966)

The 1-D case directly maps to LLVM intrinsics. The n-D case will be
handled by unrolling to 1-D first (in a later patc

[mlir][VectorOps] Add conversion of 1-D vector.interleave ops to LLVM (#80966)

The 1-D case directly maps to LLVM intrinsics. The n-D case will be
handled by unrolling to 1-D first (in a later patch).

Depends on: #80965

show more ...

[mlir][Vector] Add nontemporal attribute, mirroring memref (#76752)

Since vector loads and stores from scalar memrefs translate to
llvm.load/store, add the ability to tag said loads and stores as

[mlir][Vector] Add nontemporal attribute, mirroring memref (#76752)

Since vector loads and stores from scalar memrefs translate to
llvm.load/store, add the ability to tag said loads and stores as
nontemporal. This mirrors functionality available in memref.load/store.

show more ...

[mlir][vector][nfc] remove unused template parameter (#75931)

[mlir][vector][gpu] Align minf/maxf reduction kind names with arith (#75901)

This is to avoid confusion when dealing with reduction/combining kinds.
For example, see a recent PR comment:
https://g

[mlir][vector][gpu] Align minf/maxf reduction kind names with arith (#75901)

This is to avoid confusion when dealing with reduction/combining kinds.
For example, see a recent PR comment:
https://github.com/llvm/llvm-project/pull/75846#discussion_r1430722175.

Previously, they were picked to mostly mirror the names of the llvm
vector reduction intrinsics:
https://llvm.org/docs/LangRef.html#llvm-vector-reduce-fmin-intrinsic. In
isolation, it was not clear if `<maxf>` has `arith.maxnumf` or
`arith.maximumf` semantics. The new reduction kind names map 1:1 to
arith ops, which makes it easier to tell/look up their semantics.

Because both the vector and the gpu dialect depend on the arith dialect,
it's more natural to align names with those in arith than with the
lowering to llvm intrinsics.

Issue: https://github.com/llvm/llvm-project/issues/72354

show more ...

[mlir][LLVM] Support `immargs` in LLVM_IntrOpBase intrinsics (#73013)

This extends `LLVM_IntrOpBase` so that it can be passed a list of
`immArgPositions` and a list (of the same length) of `immArgA

[mlir][LLVM] Support `immargs` in LLVM_IntrOpBase intrinsics (#73013)

This extends `LLVM_IntrOpBase` so that it can be passed a list of
`immArgPositions` and a list (of the same length) of `immArgAttrNames`.
`immArgPositions` contains the positions of `immargs` on the LLVM IR
intrinsic, and `immArgAttrNames` maps those to a corresponding MLIR
attribute.

This allows modeling LLVM `immargs` as MLIR attributes, which is the
closest match semantically (and had already been done manually for the
LLVM dialect intrinsics).

This has two upsides:
* It's slightly easier to implement intrinsics with immargs now
(especially if they make use of other features, such as overloads)
* It clearly defines that `immargs` should map to attributes, before
there was no mention of `immargs` in LLVMOpBase.td, so implementing them
was unclear

This works with other features of the `LLVM_IntrOpBase`, so `immargs`
can be marked as overloaded too (which is used in some intrinsics).

As part of this patch (and to test correctness) existing intrinsics have
been updated to use these new parameters.

This also uncovered a few issues with the
`llvm.intr.vector.insert/extract` intrinsics. First, the argument order
for insert did not match the LLVM intrinsic, and secondly, both were
missing a mlirBuilder (so failed to import from LLVM IR). This is
corrected with this patch (and a test case added).

show more ...

[MLIR][VectorToLLVM] Remove typed pointer support (#71075)

This commit removes the support for lowering Vector to LLVM dialect with
typed pointers. Typed pointers have been deprecated for a while n

[MLIR][VectorToLLVM] Remove typed pointer support (#71075)

This commit removes the support for lowering Vector to LLVM dialect with
typed pointers. Typed pointers have been deprecated for a while now and
it's planned to soon remove them from the LLVM dialect.

Related PSA:
https://discourse.llvm.org/t/psa-removal-of-typed-pointers-from-the-llvm-dialect/74502

show more ...