[mlir] Migrate away from PointerUnion::{is,get} (NFC) (#122591)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_

[mlir] Migrate away from PointerUnion::{is,get} (NFC) (#122591)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[mlir][IR][NFC] Cleanup insertion point API usage (#115415)

Use `setInsertionPointToStart` / `setInsertionPointToEnd` when possible.

[MLIR] Erase location of folded constants (#75415)

Follow up to the discussion from #75258, and serves as an alternate
solution for #74670.

Set the location to Unknown for deduplicated / moved /

[MLIR] Erase location of folded constants (#75415)

Follow up to the discussion from #75258, and serves as an alternate
solution for #74670.

Set the location to Unknown for deduplicated / moved / materialized
constants by OperationFolder. This makes sure that the folded constants
don't end up with an arbitrary location of one of the original ops that
became it, and that hoisted ops don't confuse the stepping order.

show more ...

[mlir] Require folders to produce Values of same type (#75887)

This commit adds extra assertions to `OperationFolder` and `OpBuilder`
to ensure that the types of the folded SSA values match with th

[mlir] Require folders to produce Values of same type (#75887)

This commit adds extra assertions to `OperationFolder` and `OpBuilder`
to ensure that the types of the folded SSA values match with the result
types of the op. There used to be checks that discard the folded results
if the types do not match. This commit makes these checks stricter and
turns them into assertions.

Discarding folded results with the wrong type (without failing
explicitly) can hide bugs in op folders. Two such bugs became apparent
in MLIR (and some more in downstream projects) and are fixed with this
change.

Note: The existing type checks were introduced in
https://reviews.llvm.org/D95991.

Migration guide: If you see failing assertions (`folder produced value
of incorrect type`; make sure to run with assertions enabled!), run with
`-debug` or dump the operation right before the failing assertion. This
will point you to the op that has the broken folder. A common mistake is
a mismatch between static/dynamic dimensions (e.g., input has a static
dimension but folded result has a dynamic dimension).

show more ...

Revert "[MLIR] Fuse locations of merged constants (#74670)"

This reverts commit 87e2e89019ec4405fa47c3b4585be4e67473b590.
and its follow-ups 0d1490f09f23bf204b714c3c6ba5e0aaf4eeed9a (#75218)
and 6fe

Revert "[MLIR] Fuse locations of merged constants (#74670)"

This reverts commit 87e2e89019ec4405fa47c3b4585be4e67473b590.
and its follow-ups 0d1490f09f23bf204b714c3c6ba5e0aaf4eeed9a (#75218)
and 6fe3cd54670cae52dae92a38a6d28f450fe8f321 (#75312).

We observed significant OOM/timeout issues due to #74670 to quite a few
services including google-research/swirl-lm. The follow-up #75218 and
#75312 do not address the issue. Perhaps this is worth more
investigation.

show more ...

[MLIR][NFC] Add fast path to fused loc flattening. (#75312)

This is a follow-up on [PR
75218](https://github.com/llvm/llvm-project/pull/75218) that avoids
reconstructing a fused loc in the `Flatte

[MLIR][NFC] Add fast path to fused loc flattening. (#75312)

This is a follow-up on [PR
75218](https://github.com/llvm/llvm-project/pull/75218) that avoids
reconstructing a fused loc in the `FlattenFusedLocationRecursively`
helper when there has been no change.

show more ...

[MLIR] Flatten fused locations when merging constants. (#75218)

[PR 74670](https://github.com/llvm/llvm-project/pull/74670) added
support for merging locations at constant folding time. We have
di

[MLIR] Flatten fused locations when merging constants. (#75218)

[PR 74670](https://github.com/llvm/llvm-project/pull/74670) added
support for merging locations at constant folding time. We have
discovered that in some cases, the number of locations grows so big as
to cause a compilation process to OOM. In that case, many of the
locations end up appearing several times in nested fused locations.

We add here a helper that always flattens fused locations in order to
eliminate duplicates in the case of nested fused locations.

show more ...

[MLIR] Fuse locations of merged constants (#74670)

When merging constants by the operation folder, the location of the op
that remains should be updated to track the new meaning of this op. This
w

[MLIR] Fuse locations of merged constants (#74670)

When merging constants by the operation folder, the location of the op
that remains should be updated to track the new meaning of this op. This
way we do not lose track of all possible source locations that the
constant op came from, and the final location of the op is less reliant
on the order of folding. This will also help debuggers understand how to
step these instructions.

This PR introduces a helper for operation folder to fuse another
location into the location of an op. When an op is deduplicated, fuse
the location of the op to be removed into the op that is retained. The
retained op now represents both original ops.

The FusedLoc will have a string metadata to help understand the reason
for the location fusion (motivated by the
[example](https://github.com/llvm/llvm-project/blob/71be8f3c23497e28c86f1135f564b16106d8d6fb/mlir/include/mlir/IR/BuiltinLocationAttributes.td#L130)
in the docstring of FusedLoc).

show more ...

[mlir] Remove some code duplication between `Builders.cpp` and `FoldUtils.cpp`

Also update the documentation of `Operation::fold`, which did not take into account in-place foldings.

Differential Re

[mlir] Remove some code duplication between `Builders.cpp` and `FoldUtils.cpp`

Also update the documentation of `Operation::fold`, which did not take into account in-place foldings.

Differential Revision: https://reviews.llvm.org/D155691

show more ...

[mlir][IR] Implement proper folder for `IsCommutative` trait

Commutative ops were previously folded with a special rule in `OperationFolder`. This change turns the folding into a proper `OpTrait` fo

[mlir][IR] Implement proper folder for `IsCommutative` trait

Commutative ops were previously folded with a special rule in `OperationFolder`. This change turns the folding into a proper `OpTrait` folder.

Differential Revision: https://reviews.llvm.org/D155687

show more ...

[mlir] Move casting calls from methods to function calls

The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionali

[mlir] Move casting calls from methods to function calls

The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.

Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.

Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443

Implementation:
This patch updates all remaining uses of the deprecated functionality in
mlir/. This was done with clang-tidy as described below and further
modifications to GPUBase.td and OpenMPOpsInterfaces.td.

Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.

```
ninja -C $BUILD_DIR clang-tidy

run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix

rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```

Differential Revision: https://reviews.llvm.org/D151542

show more ...

[mlir][Transforms][NFC] Improve builder/listener API of OperationFolder

The constructor of `OperationFolder` takes a listener. Therefore, the remaining API should not take any builder/rewriters. Thi

[mlir][Transforms][NFC] Improve builder/listener API of OperationFolder

The constructor of `OperationFolder` takes a listener. Therefore, the remaining API should not take any builder/rewriters. This could lead to double notifications in case a listener is attached to the builder/rewriter.

As an internal cleanup, `OperationFolder` now has an `IRRewriter` instead of a `RewriterBase::Listener`. In most cases, `OperationFolder` no longer has to notify/deal with listeners. This is done by the rewriter.

Differential Revision: https://reviews.llvm.org/D146134

show more ...

[mlir][IR] Use Listener for IR callbacks in OperationFolder

Remove the IR modification callbacks from `OperationFolder`. Instead, an optional `RewriterBase::Listener` can be specified.
* `processGen

[mlir][IR] Use Listener for IR callbacks in OperationFolder

Remove the IR modification callbacks from `OperationFolder`. Instead, an optional `RewriterBase::Listener` can be specified.
* `processGeneratedConstants` => `notifyOperationCreated`
* `preReplaceAction` => `notifyOperationReplaced`

This simplifies the GreedyPatternRewriterDriver because we no longer need special handling for IR modifications due to op folding.

A folded operation is now enqueued on the GreedyPatternRewriteDriver's worklist if it was modified in-place. (There may be new patterns that apply after folding.)

Also fixes a bug in `TestOpInPlaceFold::fold`. The folder could previously be applied over and over and did not return a "null" OpFoldResult if the IR was not modified. (This is similar to a pattern that returns `success` without modifying IR; it can trigger an infinite loop in the GreedyPatternRewriteDriver.)

Differential Revision: https://reviews.llvm.org/D144463

show more ...

Reland [GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

Reland Note: Adds a fix to properly mark a commutative operation as folded if we change the or

Reland [GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

Reland Note: Adds a fix to properly mark a commutative operation as folded if we change the order
of its operands. This was uncovered by the fact that we no longer re-process constants.

This avoids accidentally reversing the order of constants during successive
application, e.g. when running the canonicalizer. This helps reduce the number
of iterations, and also avoids unnecessary changes to input IR.

Fixes #51892

Differential Revision: https://reviews.llvm.org/D122692

show more ...

Revert "[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down"

This reverts commit 59bbc7a0851b6e0054bb3ed47df0958822f08880.

This exposes an issue breakin

Revert "[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down"

This reverts commit 59bbc7a0851b6e0054bb3ed47df0958822f08880.

This exposes an issue breaking the contract of
`applyPatternsAndFoldGreedily` where we "converge" without applying
remaining patterns.

show more ...

[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

This avoids accidentally reversing the order of constants during successive
application, e.g. when ru

[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

This avoids accidentally reversing the order of constants during successive
application, e.g. when running the canonicalizer. This helps reduce the number
of iterations, and also avoids unnecessary changes to input IR.

Fixes #51892

Differential Revision: https://reviews.llvm.org/D122692

show more ...

Fix clang-tidy issues in mlir/ (NFC)

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D115956

[MLIR] Replace std ops with arith dialect ops

Precursor: https://reviews.llvm.org/D110200

Removed redundant ops from the standard dialect that were moved to the
`arith` or `math` dialects.

Renamed

[MLIR] Replace std ops with arith dialect ops

Precursor: https://reviews.llvm.org/D110200

Removed redundant ops from the standard dialect that were moved to the
`arith` or `math` dialects.

Renamed all instances of operations in the codebase and in tests.

Reviewed By: rriddle, jpienaar

Differential Revision: https://reviews.llvm.org/D110797

show more ...

[Canonicalize] Don't call isBeforeInBlock in OperationFolder::tryToFold.

This patch (e4635e6328c8) fixed a bug where a newly generated/reused
constant wouldn't dominate a folded operation. It did s

[Canonicalize] Don't call isBeforeInBlock in OperationFolder::tryToFold.

This patch (e4635e6328c8) fixed a bug where a newly generated/reused
constant wouldn't dominate a folded operation. It did so by calling
isBeforeInBlock to move the constant around on demand. This introduced
a significant compile time regression, because "isBeforeInBlock" is
O(n) in the size of a block the first time it is called, and the cache
is invalidated any time canonicalize changes something big in the block.

This fixes LLVM PR51738 and this CIRCT issue:
https://github.com/llvm/circt/issues/1700

This does affect the order of constants left in the top of a block,
I staged in the testsuite changes in rG42431b8207a5.

Differential Revision: https://reviews.llvm.org/D109454

show more ...

[mlir][FoldUtils] Ensure the created constant dominates the replaced op

This revision fixes a bug where an operation would get replaced with
a pre-existing constant that didn't dominate it. This can

[mlir][FoldUtils] Ensure the created constant dominates the replaced op

This revision fixes a bug where an operation would get replaced with
a pre-existing constant that didn't dominate it. This can occur when
a pattern inserts operations to be folded at the beginning of the
constants insertion block. This revision fixes the bug by moving the
existing constant before the replaced operation in such cases. This is
fine because if a constant didn't already exist, a new one would have
been inserted before this operation anyways.

Differential Revision: https://reviews.llvm.org/D108498

show more ...

[GreedyPatternRewriter] Introduce a config object that allows controlling internal parameters. NFC.

This exposes the iterations and top-down processing as flags, and also
allows controlling whether

[GreedyPatternRewriter] Introduce a config object that allows controlling internal parameters. NFC.

This exposes the iterations and top-down processing as flags, and also
allows controlling whether region simplification is desirable for a client.
This allows deleting some duplicated entrypoints to
applyPatternsAndFoldGreedily.

This also deletes the Constant Preprocessing pass, which isn't worth it
on balance.

All defaults are all kept the same, so no one should see a behavior change.

Differential Revision: https://reviews.llvm.org/D102988

show more ...

Merge with mainline.

Differential Revision: https://reviews.llvm.org/D102636

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit 361b7d125b438cda13fa45f13790767a62252be9 by Chris
Lattner <clattner@nondot.org>

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit 361b7d125b438cda13fa45f13790767a62252be9 by Chris
Lattner <clattner@nondot.org> dated Fri Mar 19 21:22:15 2021 -0700.

The change to the greedy rewriter driver picking a different order was
made without adequate analysis of the trade-offs and experimentation. A
change like this has far reaching consequences on transformation
pipelines, and a major impact upstream and downstream. For eg., one
can’t be sure that it doesn’t slow down a large number of cases by small
amounts or create other issues. More discussion here:
https://llvm.discourse.group/t/speeding-up-canonicalize/3015/25

Reverting this so that improvements to the traversal order can be made
on a clean slate, in bigger steps, and higher bar.

Differential Revision: https://reviews.llvm.org/D99329

show more ...

[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants.

This reapplies b5d9a3c / https://reviews.llvm.org/D98609 with a one line fix in
processExistingConstants to

[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants.

This reapplies b5d9a3c / https://reviews.llvm.org/D98609 with a one line fix in
processExistingConstants to skip() when erasing a constant we've already seen.

Original commit message:

1) Change the canonicalizer to walk the function in top-down order instead of
bottom-up order. This composes well with the "top down" nature of constant
folding and simplification, reducing iterations and re-evaluation of ops in
simple cases.
2) Explicitly enter existing constants into the OperationFolder table before
canonicalizing. Previously we would "constant fold" them and rematerialize
them, wastefully recreating a bunch fo constants, which lead to pointless
memory traffic.

Both changes together provide a 33% speedup for canonicalize on some mid-size
CIRCT examples.

One artifact of this change is that the constants generated in normal pattern
application get inserted at the top of the function as the patterns are applied.
Because of this, we get "inverted" constants more often, which is an aethetic
change to the IR but does permute some testcases.

Differential Revision: https://reviews.llvm.org/D99006

show more ...

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit b5d9a3c92358349d5444ab28de8ab5b2bee33a01.

The commit introduced a memory error

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit b5d9a3c92358349d5444ab28de8ab5b2bee33a01.

The commit introduced a memory error in canonicalization/operation
walking that is exposed when compiled with ASAN. It leads to crashes in
some "release" configurations.

show more ...