Reapply "[mlir][python] allow DenseIntElementsAttr for index type (#118947)" (#124804)

This reapplies #118947 and adapts to nanobind.

Revert "[mlir][python] allow DenseIntElementsAttr for index type (#118947)"

This reverts commit 9dd762e8b10586e749b0ddf3542e5dccf8392395.

[mlir][python] allow DenseIntElementsAttr for index type (#118947)

Model the `IndexType` as `uint64_t` when converting to a python integer.

With the python bindings,
```python
DenseIntElement

[mlir][python] allow DenseIntElementsAttr for index type (#118947)

Model the `IndexType` as `uint64_t` when converting to a python integer.

With the python bindings,
```python
DenseIntElementsAttr(op.attributes["attr"])
```
used to `assert` when `attr` had `index` type like `dense<[1, 2, 3, 4]>
: vector<4xindex>`.

---------

Co-authored-by: Christopher McGirr <christopher.mcgirr@amd.com>
Co-authored-by: Tiago Trevisan Jost <tiago.trevisanjost@amd.com>

show more ...

[MLIR,Python] Support converting boolean numpy arrays to and from mlir attributes (unrevert) (#115481)

This PR re-introduces the functionality of
https://github.com/llvm/llvm-project/pull/113064, w

[MLIR,Python] Support converting boolean numpy arrays to and from mlir attributes (unrevert) (#115481)

This PR re-introduces the functionality of
https://github.com/llvm/llvm-project/pull/113064, which was reverted in
https://github.com/llvm/llvm-project/commit/0a68171b3c67503f7143856580f1b22a93ef566e
due to memory lifetime issues.

Notice that I was not able to re-produce the ASan results myself, so I
have not been able to verify that this PR really fixes the issue.

---

Currently it is unsupported to:
1. Convert a MlirAttribute with type i1 to a numpy array
2. Convert a boolean numpy array to a MlirAttribute

Currently the entire Python application violently crashes with a quite
poor error message https://github.com/pybind/pybind11/issues/3336

The complication handling these conversions, is that MlirAttribute
represent booleans as a bit-packed i1 type, whereas numpy represents
booleans as a byte array with 8 bit used per boolean.

This PR proposes the following approach:
1. When converting a i1 typed MlirAttribute to a numpy array, we can not
directly use the underlying raw data backing the MlirAttribute as a
buffer to Python, as done for other types. Instead, a copy of the data
is generated using numpy's unpackbits function, and the result is send
back to Python.
2. When constructing a MlirAttribute from a numpy array, first the
python data is read as a uint8_t to get it converted to the endianess
used internally in mlir. Then the booleans are bitpacked using numpy's
bitpack function, and the bitpacked array is saved as the MlirAttribute
representation.

show more ...

Revert "[MLIR,Python] Support converting boolean numpy arrays to and from mlir attributes (#113064)"

This reverts commit fb7bf7a5acc65be44fc546f282942b91472553b3. There is
an ASan issue here, see th

Revert "[MLIR,Python] Support converting boolean numpy arrays to and from mlir attributes (#113064)"

This reverts commit fb7bf7a5acc65be44fc546f282942b91472553b3. There is
an ASan issue here, see the discussion on
https://github.com/llvm/llvm-project/pull/113064.

show more ...

[MLIR,Python] Support converting boolean numpy arrays to and from mlir attributes (#113064)

Currently it is unsupported to:
1. Convert a `MlirAttribute` with type `i1` to a numpy array
2. Convert

[MLIR,Python] Support converting boolean numpy arrays to and from mlir attributes (#113064)

Currently it is unsupported to:
1. Convert a `MlirAttribute` with type `i1` to a numpy array
2. Convert a boolean numpy array to a `MlirAttribute`

Currently the entire Python application violently crashes with a quite
poor error message https://github.com/pybind/pybind11/issues/3336

The complication handling these conversions, is that `MlirAttribute`
represent booleans as a bit-packed `i1` type, whereas numpy represents
booleans as a byte array with 8 bit used per boolean.

This PR proposes the following approach:
1. When converting a `i1` typed `MlirAttribute` to a numpy array, we can
not directly use the underlying raw data backing the `MlirAttribute` as
a buffer to Python, as done for other types. Instead, a copy of the data
is generated using numpy's unpackbits function, and the result is send
back to Python.
2. When constructing a `MlirAttribute` from a numpy array, first the
python data is read as a `uint8_t` to get it converted to the endianess
used internally in mlir. Then the booleans are bitpacked using numpy's
bitpack function, and the bitpacked array is saved as the
`MlirAttribute` representation.

Please note that I am not sure if this approach is the desired solution.
I'd appreciate any feedback.

show more ...

[mlir][python] Add bindings for mlirDenseElementsAttrGet (#91389)

This change adds bindings for `mlirDenseElementsAttrGet` which accepts a
list of MLIR attributes and constructs a DenseElementsAttr

[mlir][python] Add bindings for mlirDenseElementsAttrGet (#91389)

This change adds bindings for `mlirDenseElementsAttrGet` which accepts a
list of MLIR attributes and constructs a DenseElementsAttr. This allows
for creating `DenseElementsAttr`s of types not natively supported by
Python (e.g. BF16) without requiring other dependencies (e.g. `numpy` +
`ml-dtypes`).

show more ...

[mlir] Add Python bindings for DenseResourceElementsAttr. (#66319)

Only construction and type casting are implemented. The method to create
is explicitly named "unsafe" and the documentation calls

[mlir] Add Python bindings for DenseResourceElementsAttr. (#66319)

Only construction and type casting are implemented. The method to create
is explicitly named "unsafe" and the documentation calls out what the
caller is responsible for. There really isn't a better way to do this
and retain the power-user feature this represents.

show more ...

[MLIR:Python] Make DenseElementsAttr.get() only request a buffer format if no explicit type was provided.

Not every NumPy type (e.g., the `ml_dtypes.bfloat16` NumPy extension
type) has a type in the

[MLIR:Python] Make DenseElementsAttr.get() only request a buffer format if no explicit type was provided.

Not every NumPy type (e.g., the `ml_dtypes.bfloat16` NumPy extension
type) has a type in the Python buffer protocol, so exporting such a
buffer with `PyBUF_FORMAT` may fail.

However, we don't care about the self-reported type of a buffer if the
user provides an explicit type. In the case that an explicit type is
provided, don't request the format from the buffer protocol, which
allows arrays whose element types are unknown to the buffer protocol to
be passed.

Reviewed By: jpienaar, ftynse

Differential Revision: https://reviews.llvm.org/D155209

show more ...

[mlir][python] Downcast attributes in more places

Update remaining `PyAttribute`-returning APIs to return `MlirAttribute` instead,
so that they go through the downcasting mechanism.

Reviewed By: ma

[mlir][python] Downcast attributes in more places

Update remaining `PyAttribute`-returning APIs to return `MlirAttribute` instead,
so that they go through the downcasting mechanism.

Reviewed By: makslevental

Differential Revision: https://reviews.llvm.org/D154462

show more ...

[NFC][Py Reformat] Reformat python files in mlir subdir

This is an ongoing series of commits that are reformatting our
Python code.

Reformatting is done with `black`.

If you end up having problems

[NFC][Py Reformat] Reformat python files in mlir subdir

This is an ongoing series of commits that are reformatting our
Python code.

Reformatting is done with `black`.

If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.

If you run into any problems, post to discourse about it and
we will try to help.

RFC Thread below:

https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style

Differential Revision: https://reviews.llvm.org/D150782

show more ...

[MLIR][python bindings] Add support for DenseElementsAttr of IndexType

Differential Revision: https://reviews.llvm.org/D149690

[mlir][python] Support buffer protocol for splat dense attributes

These can be made to work by setting the buffer strides to 0.

Reviewed By: stellaraccident

Differential Revision: https://reviews.

[mlir][python] Support buffer protocol for splat dense attributes

These can be made to work by setting the buffer strides to 0.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D147187

show more ...

Support retrieving the splat value from DenseElementsAttrs in Python

This is especially convenient when trying to resize the splat.

Reviewed By: jpienaar

Differential Revision: https://reviews.llv

Support retrieving the splat value from DenseElementsAttrs in Python

This is especially convenient when trying to resize the splat.

Reviewed By: jpienaar

Differential Revision: https://reviews.llvm.org/D146510

show more ...

[mlir][python] Temporarily disable test for converting unsupported DenseElementsAttr types to a buffer.

* Need to investigate the proper solution to https://github.com/pybind/pybind11/issues/3336 or

[mlir][python] Temporarily disable test for converting unsupported DenseElementsAttr types to a buffer.

* Need to investigate the proper solution to https://github.com/pybind/pybind11/issues/3336 or engineer something different.
* The attempt to produce an empty buffer_info as a workaround triggers asan/ubsan.
* Usage of this API does not arise naturally in practice yet, and it is more important to be asan/crash clean than have a solution right now.
* Switching back to raising an exception, even though that triggers terminate().

show more ...

[mlir] Extend C and Python API to support bulk loading of DenseElementsAttr.

* This already half existed in terms of reading the raw buffer backing a DenseElementsAttr.
* Documented the precise expe

[mlir] Extend C and Python API to support bulk loading of DenseElementsAttr.

* This already half existed in terms of reading the raw buffer backing a DenseElementsAttr.
* Documented the precise expectations of the buffer layout.
* Extended the Python API to support construction from bitcasted buffers, allowing construction of all primitive element types (even those that lack a compatible representation in Python).
* Specifically, the Python API can now load all integer types at all bit widths and all floating point types (f16, f32, f64, bf16).

Differential Revision: https://reviews.llvm.org/D111284

show more ...

Move MLIR python sources to mlir/python.

* NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends).
* Includes previously unsubmitted fi

Move MLIR python sources to mlir/python.

* NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends).
* Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup.

Differential Revision: https://reviews.llvm.org/D101493

show more ...