[mlir][SCF] Fix dynamic loop pipeline peeling for num_stages > total_iters (#112418)

When pipelining an `scf.for` with dynamic loop bounds, the epilogue
ramp-down must align with the prologue when

[mlir][SCF] Fix dynamic loop pipeline peeling for num_stages > total_iters (#112418)

When pipelining an `scf.for` with dynamic loop bounds, the epilogue
ramp-down must align with the prologue when num_stages >
total_iterations.

For example:
```
scf.for (0..ub) {
load(i)
add(i)
store(i)
}
```
When num_stages=3 the pipeline follows:
```
load(0) - add(0) - scf.for (0..ub-2) - store(ub-2)
load(1) - - add(ub-1) - store(ub-1)

```
The trailing `store(ub-2)`, `i=ub-2`, must align with the ramp-up for
`i=0` when `ub < num_stages-1`, so the index `i` should be `max(0,
ub-2)` and each subsequent index is an increment. The predicate must
also handle this scenario, so it becomes `predicate[0] =
total_iterations > epilogue_stage`.

show more ...

[mlir][scf] Fix loop iteration calculation for negative step in LoopPipelining (#110035)

This fixes loop iteration count calculation if the step is
a negative value, where we should adjust the

[mlir][scf] Fix loop iteration calculation for negative step in LoopPipelining (#110035)

This fixes loop iteration count calculation if the step is
a negative value, where we should adjust the added
delta from `step-1` to `step+1` when doing the ceil div.

show more ...

[SCF] Fixed epilogue predicates in loop pipelining (#108964)

The computed loop iteration is zero based, so only check it is less than
zero. This fixes the case when lower bound is not zero.

[MLIR][SCF] Loop pipelining fails on failed predication (no assert) (#107442)

The SCFLoopPipelining allows predication on peeled or loop ops. When the
predicationFn returns a nullptr this signifies

[MLIR][SCF] Loop pipelining fails on failed predication (no assert) (#107442)

The SCFLoopPipelining allows predication on peeled or loop ops. When the
predicationFn returns a nullptr this signifies the op type is
unsupported and the pipeliner fails except in `emitPrologue` where it
asserts.

This patch fixes handling in the prologue to gracefully fail.

show more ...

[MLIR][SCF] Add support for loop pipeline peeling for dynamic loops. (#106436)

Allow speculative execution and predicate results per stage.

[SCF][PIPELINE] Handle the case when values from the peeled prologue may escape out of the loop (#105755)

Previously the values in the peeled prologue that weren't treated with
the `predicateFn` we

[SCF][PIPELINE] Handle the case when values from the peeled prologue may escape out of the loop (#105755)

Previously the values in the peeled prologue that weren't treated with
the `predicateFn` were passed to the loop body without any other
predication. If those values are later used outside of the loop body,
they may be incorrect if the num iterations is smaller than num stages -
1. We need similar masking for those, as is done in the main loop body,
using already existing predicates.

show more ...

mlir/MathExtras: consolidate with llvm/MathExtras (#95087)

This patch is part of a project to move the Presburger library into
LLVM.

[MLIR][LoopPipelining] Improve schedule verifier, so it checks also operands of nested operations (#88450)

`verifySchedule` was not looking at the operands of nested operations,
which caused incorr

[MLIR][LoopPipelining] Improve schedule verifier, so it checks also operands of nested operations (#88450)

`verifySchedule` was not looking at the operands of nested operations,
which caused incorrect schedule to be allowed in some cases, potentially
leading to crash during expansion.
There is also minor fix in `cloneAndUpdateOperands` in the pipeline
expander that prevents double visit of the cloned op. This one has no
functional impact, so no test for it.

show more ...

[MLIR][SCF] Add checks to verify that the pipeliner schedule is correct. (#77083)

Add a check to validate that the schedule passed to the pipeliner
transformation is valid and won't cause the pipel

[MLIR][SCF] Add checks to verify that the pipeliner schedule is correct. (#77083)

Add a check to validate that the schedule passed to the pipeliner
transformation is valid and won't cause the pipeliner to break SSA.

This checks that the for each operation in the loop operations are
scheduled after their operands.

show more ...

[mlir] Fix loop pipelining when the operand of `yield` is not defined in the loop body (#75423)

[MLIR][SCF] Add support for pipelining dynamic loops (#74350)

Support loops without static boundaries. Since the number of iteration
is not known we need to predicate prologue and epilogue in case

[MLIR][SCF] Add support for pipelining dynamic loops (#74350)

Support loops without static boundaries. Since the number of iteration
is not known we need to predicate prologue and epilogue in case the
number of iterations is smaller than the number of stages.

This patch includes work from @chengjunlu

show more ...

[MLIR][SCF] Handle more cases in pipelining transform (#74007)

-Fix case where an op is scheduled in stage 0 and used with a distance
of 1
-Fix case where we don't peel the epilogue and a value no

[MLIR][SCF] Handle more cases in pipelining transform (#74007)

-Fix case where an op is scheduled in stage 0 and used with a distance
of 1
-Fix case where we don't peel the epilogue and a value not part of the
last stage is used outside the loop.

show more ...

[mlir][Interfaces] `LoopLikeOpInterface`: Add helpers to query tied inits/iter_args (#70408)

The `LoopLikeOpInterface` already has interface methods to query inits
and iter_args. This commit adds h

[mlir][Interfaces] `LoopLikeOpInterface`: Add helpers to query tied inits/iter_args (#70408)

The `LoopLikeOpInterface` already has interface methods to query inits
and iter_args. This commit adds helper functions to query tied
init/iter_arg pairs and removes the corresponding functions for
`scf::ForOp`.

show more ...

[mlir][nvgpu] add simple pipelining for shared memory copies

Add a simple transform operation to the NVGPU extension that performs
software pipelining of copies to shared memory. The functionality i

[mlir][nvgpu] add simple pipelining for shared memory copies

Add a simple transform operation to the NVGPU extension that performs
software pipelining of copies to shared memory. The functionality is
extremely minimalistic in this version and only supports copies from
global to shared memory inside an `scf.for` loop with either
`vector.transfer` or `nvgpu.device_async_copy` operations when
pipelining preconditions are already satisfied in the IR. This is the
minimally useful version that uses the more general loop pipeliner in an
NVGPU-specific way. Further extensions and orthogonalizations will be
necessary.

This required a change to the loop pipeliner itself to properly
propagate errors should the predicate generator fail.

This is loosely inspired from the vesion in IREE, but has less unsafe
assumptions and more principled way of communicating decisions.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D155223

show more ...

[mlir][scf] NFC - Add debug information to scf pipelining

[mlir] Move casting calls from methods to function calls

The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionali

[mlir] Move casting calls from methods to function calls

The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.

Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.

Caveats include:
- This clang-tidy script probably has more problems.
- This only touches C++ code, so nothing that is being generated.

Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443

Implementation:
This first patch was created with the following steps. The intention is
to only do automated changes at first, so I waste less time if it's
reverted, and so the first mass change is more clear as an example to
other teams that will need to follow similar steps.

Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
4. Some changes have been deleted for the following reasons:
- Some files had a variable also named cast
- Some files had not included a header file that defines the cast
functions
- Some files are definitions of the classes that have the casting
methods, so the code still refers to the method instead of the
function without adding a prefix or removing the method declaration
at the same time.

```
ninja -C $BUILD_DIR clang-tidy

run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix

rm -rf $BUILD_DIR/tools/mlir/**/*.inc

git restore mlir/lib/IR mlir/lib/Dialect/DLTI/DLTI.cpp\
mlir/lib/Dialect/Complex/IR/ComplexDialect.cpp\
mlir/lib/**/IR/\
mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp\
mlir/lib/Dialect/Vector/Transforms/LowerVectorMultiReduction.cpp\
mlir/test/lib/Dialect/Test/TestTypes.cpp\
mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp\
mlir/test/lib/Dialect/Test/TestAttributes.cpp\
mlir/unittests/TableGen/EnumsGenTest.cpp\
mlir/test/python/lib/PythonTestCAPI.cpp\
mlir/include/mlir/IR/
```

Differential Revision: https://reviews.llvm.org/D150123

show more ...

[mlir] Use Use *{Set,Map}::contains (NFC)

[mlir][Transform] NFC - Various API cleanups and use RewriterBase in lieu of PatternRewriter

Depends on: D145685

Differential Revision: https://reviews.llvm.org/D145977

[mlir][scf] Fix bug in software pipeliner and simplify the logic

Fix bug when pipelining while interleaving stages. Re-do the logic to
only consider cloned operands when updating the use-def chain.

[mlir][scf] Fix bug in software pipeliner and simplify the logic

Fix bug when pipelining while interleaving stages. Re-do the logic to
only consider cloned operands when updating the use-def chain.

Differential Revision: https://reviews.llvm.org/D145598

show more ...

[mlir] Add operations to BlockAndValueMapping and rename it to IRMapping

The patch adds operations to `BlockAndValueMapping` and renames it to `IRMapping`. When operations are cloned, old operations

[mlir] Add operations to BlockAndValueMapping and rename it to IRMapping

The patch adds operations to `BlockAndValueMapping` and renames it to `IRMapping`. When operations are cloned, old operations are mapped to the cloned operations. This allows mapping from an operation to a cloned operation. Example:

```
Operation *opWithRegion = ...
Operation *opInsideRegion = &opWithRegion->front().front();

IRMapping map
Operation *newOpWithRegion = opWithRegion->clone(map);
Operation *newOpInsideRegion = map.lookupOrNull(opInsideRegion);
```

Migration instructions:
All includes to `mlir/IR/BlockAndValueMapping.h` should be replaced with `mlir/IR/IRMapping.h`. All uses of `BlockAndValueMapping` need to be renamed to `IRMapping`.

Reviewed By: rriddle, mehdi_amini

Differential Revision: https://reviews.llvm.org/D139665

show more ...

[mlir][arith] Change dialect name from Arithmetic to Arith

Suggested by @lattner in https://discourse.llvm.org/t/rfc-define-precise-arith-semantics/65507/22.

Tested with:
`ninja check-mlir check-ml

[mlir][arith] Change dialect name from Arithmetic to Arith

Suggested by @lattner in https://discourse.llvm.org/t/rfc-define-precise-arith-semantics/65507/22.

Tested with:
`ninja check-mlir check-mlir-integration check-mlir-mlir-spirv-cpu-runner check-mlir-mlir-vulkan-runner check-mlir-examples`

and `bazel build --config=generic_clang @llvm-project//mlir:all`.

Reviewed By: lattner, Mogball, rriddle, jpienaar, mehdi_amini

Differential Revision: https://reviews.llvm.org/D134762

show more ...

[mlir][SCF] Fix loop pipelining unable to handle ops with regions

This change allows the SCF LoopPipelining transform to handle ops with
nested regions within the pipelined `scf.for` body. The op an

[mlir][SCF] Fix loop pipelining unable to handle ops with regions

This change allows the SCF LoopPipelining transform to handle ops with
nested regions within the pipelined `scf.for` body. The op and nested
regions are treated as a single unit from the transform's perspective.
This change also makes explicit the requirement that only ops whose
parent Block is the loop body Block are allowed to be scheduled by the
caller.

Reviewed By: ThomasRaoux, nicolasvasilache

Differential Revision: https://reviews.llvm.org/D133965

show more ...

[MLIR] Update pass declarations to new autogenerated files

The patch introduces the required changes to update the pass declarations and definitions to use the new autogenerated files and allow drop

[MLIR] Update pass declarations to new autogenerated files

The patch introduces the required changes to update the pass declarations and definitions to use the new autogenerated files and allow dropping the old infrastructure.

Reviewed By: mehdi_amini, rriddle

Differential Review: https://reviews.llvm.org/D132838

show more ...

Revert "[MLIR] Update pass declarations to new autogenerated files"

This reverts commit 2be8af8f0e0780901213b6fd3013a5268ddc3359.

[MLIR] Update pass declarations to new autogenerated files

The patch introduces the required changes to update the pass declarations and definitions to use the new autogenerated files and allow drop

[MLIR] Update pass declarations to new autogenerated files

The patch introduces the required changes to update the pass declarations and definitions to use the new autogenerated files and allow dropping the old infrastructure.

Reviewed By: mehdi_amini, rriddle

Differential Review: https://reviews.llvm.org/D132838

show more ...