[mlir python] Port in-tree dialects to nanobind. (#119924)

This is a companion to #118583, although it can be landed independently
because since #117922 dialects do not have to use the same Python

[mlir python] Port in-tree dialects to nanobind. (#119924)

This is a companion to #118583, although it can be landed independently
because since #117922 dialects do not have to use the same Python
binding framework as the Python core code.

This PR ports all of the in-tree dialect and pass extensions to
nanobind, with the exception of those that remain for testing pybind11
support.

This PR also:
* removes CollectDiagnosticsToStringScope from NanobindAdaptors.h. This
was overlooked in a previous PR and it is duplicated in Diagnostics.h.

---------

Co-authored-by: Jacques Pienaar <jpienaar@google.com>

show more ...

[mlir python] Port Python core code to nanobind. (#120473)

Relands #118583, with a fix for Python 3.8 compatibility. It was not
possible to set the buffer protocol accessers via slots in Python 3.8

[mlir python] Port Python core code to nanobind. (#120473)

Relands #118583, with a fix for Python 3.8 compatibility. It was not
possible to set the buffer protocol accessers via slots in Python 3.8.

Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve MLIR
IR construction time from JAX.

For a complicated Google-internal LLM model in JAX, this change improves
the MLIR
lowering time by around 5s (out of around 30s), which is a significant
speedup for simply switching binding frameworks.

To a large extent, this is a mechanical change, for instance changing
`pybind11::` to `nanobind::`.

Notes:
* this PR needs Nanobind 2.4.0, because it needs a bug fix
(https://github.com/wjakob/nanobind/pull/806) that landed in that
release.
* this PR does not port the in-tree dialect extension modules. They can
be ported in a future PR.
* I removed the py::sibling() annotations from def_static and def_class
in `PybindAdapters.h`. These ask pybind11 to try to form an overload
with an existing method, but it's not possible to form mixed
pybind11/nanobind overloads this ways and the parent class is now
defined in nanobind. Better solutions may be possible here.
* nanobind does not contain an exact equivalent of pybind11's buffer
protocol support. It was not hard to add a nanobind implementation of a
similar API.
* nanobind is pickier about casting to std::vector<bool>, expecting that
the input is a sequence of bool types, not truthy values. In a couple of
places I added code to support truthy values during casting.
* nanobind distinguishes bytes (`nb::bytes`) from strings (e.g.,
`std::string`). This required nb::bytes overloads in a few places.

show more ...

Revert "[mlir python] Port Python core code to nanobind. (#118583)"

This reverts commit 41bd35b58bb482fd466aa4b13aa44a810ad6470f.

Breakage detected, rolling back.

[mlir python] Port Python core code to nanobind. (#118583)

Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve

[mlir python] Port Python core code to nanobind. (#118583)

Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve MLIR
IR construction time from JAX.

For a complicated Google-internal LLM model in JAX, this change improves
the MLIR
lowering time by around 5s (out of around 30s), which is a significant
speedup for simply switching binding frameworks.

To a large extent, this is a mechanical change, for instance changing
`pybind11::`
to `nanobind::`.

Notes:
* this PR needs Nanobind 2.4.0, because it needs a bug fix
(https://github.com/wjakob/nanobind/pull/806) that landed in that
release.
* this PR does not port the in-tree dialect extension modules. They can
be ported in a future PR.
* I removed the py::sibling() annotations from def_static and def_class
in `PybindAdapters.h`. These ask pybind11 to try to form an overload
with an existing method, but it's not possible to form mixed
pybind11/nanobind overloads this ways and the parent class is now
defined in nanobind. Better solutions may be possible here.
* nanobind does not contain an exact equivalent of pybind11's buffer
protocol support. It was not hard to add a nanobind implementation of a
similar API.
* nanobind is pickier about casting to std::vector<bool>, expecting that
the input is a sequence of bool types, not truthy values. In a couple of
places I added code to support truthy values during casting.
* nanobind distinguishes bytes (`nb::bytes`) from strings (e.g.,
`std::string`). This required nb::bytes overloads in a few places.

show more ...

Revert "Revert "[mlir python] Add nanobind support (#119232)

Reverts revert #118517 after (hopefully) fixing builders
(https://github.com/llvm/llvm-zorg/pull/328,
https://github.com/llvm/llvm-zor

Revert "Revert "[mlir python] Add nanobind support (#119232)

Reverts revert #118517 after (hopefully) fixing builders
(https://github.com/llvm/llvm-zorg/pull/328,
https://github.com/llvm/llvm-zorg/pull/327)

This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.

show more ...

Revert "[mlir python] Add nanobind support for standalone dialects." (#118517)

Reverts llvm/llvm-project#117922 because deps aren't met on some of the
post-commit build bots.

[mlir python] Add nanobind support for standalone dialects. (#117922)

This PR allows out-of-tree dialects to write Python dialect modules
using nanobind instead of pybind11.

It may make sense to

[mlir python] Add nanobind support for standalone dialects. (#117922)

This PR allows out-of-tree dialects to write Python dialect modules
using nanobind instead of pybind11.

It may make sense to migrate in-tree dialects and some of the ODS Python
infrastructure to nanobind, but that is a topic for a future change.

This PR makes the following changes:
* adds nanobind to the CMake and Bazel build systems. We also add
robin_map to the Bazel build, which is a dependency of nanobind.
* adds a PYTHON_BINDING_LIBRARY option to various CMake functions, such
as declare_mlir_python_extension, allowing users to select a Python
binding library.
* creates a fork of mlir/include/mlir/Bindings/Python/PybindAdaptors.h
named NanobindAdaptors.h. This plays the same role, using nanobind
instead of pybind11.
* splits CollectDiagnosticsToStringScope out of PybindAdaptors.h and
into a new header mlir/include/mlir/Bindings/Python/Diagnostics.h, since
it is code that is no way related to pybind11 or for that matter,
Python.
* changed the standalone Python extension example to have both pybind11
and nanobind variants.
* changed mlir/python/mlir/dialects/python_test.py to have both pybind11
and nanobind variants.

Notes:
* A slightly unfortunate thing that I needed to do in the CMake
integration was to use FindPython in addition to FindPython3, since
nanobind's CMake integration expects the Python_ names for variables.
Perhaps there's a better way to do this.

show more ...

[mlir] Add option to disable MLIR Python dev package configuration. (#117934)

Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which
gates doing package discovery and configuration fo

[mlir] Add option to disable MLIR Python dev package configuration. (#117934)

Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which
gates doing package discovery and configuration for Python dev packages
by MLIR (this was made opt-out to preserve compatibility with
find_package(MLIR) based uses which do not set the standard options).

The default Python setup that MLIR does has been a problem for
super-projects that include LLVM for a long time because it forces a
very specific package discovery mechanism that is not uniform in all
uses.

When reviewing #117922, I noted that this would effectively be a break
the world event for downstreams, forcing them to adapt their nanobind
dep to the exact way that MLIR does it. Adding the option to just
wholesale skip the built-in configuration heuristics at least gives us a
mechanism to tell downstreams to migrate to, giving them complete
control and not requiring packaging workarounds. This seemed a better
option than (once again) creating a situation where downstreams could
not integrate the dep change without doing tricky infra upgrades, and it
removes the burden from the author of that patch from needing to think
about how this affects super-projects that include MLIR (i.e. they can
just be told to do it themselves as needed vs being in a wedged state
and unable to upgrade).

show more ...

[mlir][python] Update minimal version of pybind11 to 2.10. (#117314)

This PR updates the minimal required version of pybind11 from 2.9.0 to
2.10.0. New new version is almost 2.5 years old, which is

[mlir][python] Update minimal version of pybind11 to 2.10. (#117314)

This PR updates the minimal required version of pybind11 from 2.9.0 to
2.10.0. New new version is almost 2.5 years old, which is half a year
less than the previous version. This change is necessary to support the
changes introduced in #115307, which does not compile with pybind11
v.2.9.

Signed-off-by: Ingo Müller <ingomueller@google.com>

show more ...

[CMake] Remove requirement of NumPy header files (#108465)

NumPy header files were required for building MLIR, however the NumPy
C-API is never used. In other words, NumPy is not a build time
depe

[CMake] Remove requirement of NumPy header files (#108465)

NumPy header files were required for building MLIR, however the NumPy
C-API is never used. In other words, NumPy is not a build time
dependency. `numpy`, the python package, is required at runtime for the
python bindings tests. In particular the file
`mlir/python/mlir/runtime/np_to_memref.py` and all tests which may use
it. This commit removes the build time dependency, but the runtime
dependency remains through the `requirements.txt` file.

Co-authored-by: Erick Ochoa <erick@ceci-nest-pas.me>

show more ...

[MLIR] Removes CMake work-arounds.

CMake older than 3.20.0 is no longer supported.
This removes work-arounds for no longer supported versions.

Reviewed By: mehdi_amini

Differential Revision: https

[MLIR] Removes CMake work-arounds.

CMake older than 3.20.0 is no longer supported.
This removes work-arounds for no longer supported versions.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D152101

show more ...

[mlir][python] Bump min pybind11 version to 2.9.0

2.9.0 was released on December 28, 2021, and some following changes
require at least this version.

Reviewed By: stellaraccident

Differential Revis

[mlir][python] Bump min pybind11 version to 2.9.0

2.9.0 was released on December 28, 2021, and some following changes
require at least this version.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D150247

show more ...

[mlir] Update Python CMake version requirement.

The minimum required version is now 3.19 due to the usage of some
more recent features. Update the version check and error message
accordingly. Also r

[mlir] Update Python CMake version requirement.

The minimum required version is now 3.19 due to the usage of some
more recent features. Update the version check and error message
accordingly. Also remove some logic that behaved differently before
3.18, since we can assume we are now on version 3.19+.

Reviewed By: stella.stamenova

Differential Revision: https://reviews.llvm.org/D130171

show more ...

[MLIR] [CMake/Python] Fix ignored Python3_EXECUTABLE

On Windows (at least), cmake ignores Python3_EXECUTABLE unless the
'Interpreter' component is being found. If the user is specifying a
different

[MLIR] [CMake/Python] Fix ignored Python3_EXECUTABLE

On Windows (at least), cmake ignores Python3_EXECUTABLE unless the
'Interpreter' component is being found. If the user is specifying a
different version than the latest installed (say, 3.8 vs 3.9) with the
Python3_EXECUTABLE, cmake was using a combination of the newest version
and the desired version. Mitigated by adding 'Interpreter' in the first
invocation like the second one.

show more ...

[mlir] Add option to control python search priming

This adds an option to configure the CMake python search priming
behaviour that was introduced in D118148. In some environments the
priming would c

[mlir] Add option to control python search priming

This adds an option to configure the CMake python search priming
behaviour that was introduced in D118148. In some environments the
priming would cause the "real" search to fail. The default behaviour is
unchanged, i.e. the search will be primed.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D120765

show more ...

[MLIR] Workaround for python detection problems.

Although cmake should be platform-independent, we've observed
that some aspects of Python detection don't work on all platforms,
even with recent ver

[MLIR] Workaround for python detection problems.

Although cmake should be platform-independent, we've observed
that some aspects of Python detection don't work on all platforms,
even with recent versions of cmake. This appears to be due to bugs
in the python detection logic, especially when the NumPy component
is required and not located within the python installation.
As a workaround, this patch first searches for "Development" before
searching for "Development.Module", which seems to workaround the
issue.

Differential Revision: https://reviews.llvm.org/D118148

show more ...

[mlir] Rework subclass construction in PybindAdaptors.h

The constructor function was being defined without indicating its "__init__"
name, which made it interpret it as a regular fuction rather than

[mlir] Rework subclass construction in PybindAdaptors.h

The constructor function was being defined without indicating its "__init__"
name, which made it interpret it as a regular fuction rather than a
constructor. When overload resolution failed, Pybind would attempt to print the
arguments actually passed to the function, including "self", which is not
initialized since the constructor couldn't be called. This would result in
"__repr__" being called with "self" referencing an uninitialized MLIR C API
object, which in turn would cause undefined behavior when attempting to print
in C++. Even if the correct name is provided, the mechanism used by
PybindAdaptors.h to bind constructors directly as "__init__" functions taking
"self" is deprecated by Pybind. The new mechanism does not seem to have access
to a fully-constructed "self" object (i.e., the constructor in C++ takes a
`pybind11::detail::value_and_holder` that cannot be forwarded back to Python).

Instead, redefine "__new__" to perform the required checks (there are no
additional initialization needed for attributes and types as they are all
wrappers around a C++ pointer). "__new__" can call its equivalent on a
superclass without needing "self".

Bump pybind11 dependency to 3.8.0, which is the first version that allows one
to redefine "__new__".

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D117646

show more ...

[mlir][python] Add a warning for cmake version < 3.15.

As discussed on discord, we have never actually been able to build with the project-wide published min version of 3.14.3. The buildbot that tes

[mlir][python] Add a warning for cmake version < 3.15.

As discussed on discord, we have never actually been able to build with the project-wide published min version of 3.14.3. The buildbot that tests the Python configuration is currently pinned to 3.19.1, and there are a number of non-version/policy controlled features that Python building relies on that makes it unreliable with older versions. Some of the issues are pretty fundamental and I don't know how to do them on the older version. I think that, as an optional feature, at least advertising the PSA as in this patch is a good middle ground until the next project-wide CMake version bump.

Also moves setup logic to a macro so that everyone can use it.

show more ...

[mlir] use STATUS instead of CHECK_* in MLIRDetectPythonEnv.cmake

CHECK_* directives for message() where added in Cmake 3.17, LLVM
requires 3.14 as minimum so they may not be intepreted correctly an

[mlir] use STATUS instead of CHECK_* in MLIRDetectPythonEnv.cmake

CHECK_* directives for message() where added in Cmake 3.17, LLVM
requires 3.14 as minimum so they may not be intepreted correctly and
just print "CHECK_*" into the message stream. Replace them with STATUS.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D91959

show more ...

[mlir][Python] Fix the last remaining instance of PYTHON_EXECUTABLE.

* Was causing auto-detect of pybind11 to fail on clean configure.

Differential Revision: https://reviews.llvm.org/D92043

[mlir][Python] Support finding pybind11 from the python environment.

* Makes `pip install pybind11` do the right thing with no further config.
* Since we now require a version of pybind11 greater th

[mlir][Python] Support finding pybind11 from the python environment.

* Makes `pip install pybind11` do the right thing with no further config.
* Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable.
* Also adds the version spec to find_package so it will skip older versions that may be lying around.
* Tested the full matrix of old system install, no system install, pip install and no pip install.

Differential Revision: https://reviews.llvm.org/D91903

show more ...