[LoopUnroll] Add RuntimeUnrollMultiExit to loop unroll options (NFC) (#124462)

Add an extra knob to RuntimeUnrollMultiExit to let backends control
whether to allow multi-exit unrolling on a per-loo

[LoopUnroll] Add RuntimeUnrollMultiExit to loop unroll options (NFC) (#124462)

Add an extra knob to RuntimeUnrollMultiExit to let backends control
whether to allow multi-exit unrolling on a per-loop basis.

This gives backends more fine-grained control on deciding if multi-exit
unrolling is profitable for a given loop and uarch. Similar to
4226e0a0c75.

PR: https://github.com/llvm/llvm-project/pull/124462

show more ...

[TTI] Add SCEVExpansionBudget to loop unrolling options. (#118316)

Add an extra know to UnrollingPreferences to let backends control the
maximum budget for SCEV expansions.

This gives backends m

[TTI] Add SCEVExpansionBudget to loop unrolling options. (#118316)

Add an extra know to UnrollingPreferences to let backends control the
maximum budget for SCEV expansions.

This gives backends more fine-grained control on the cost of the runtime
checks for runtime unrolling.

PR: https://github.com/llvm/llvm-project/pull/118316

show more ...

[Utils] Remove unused includes (NFC) (#114748)

Identified with misc-include-cleaner.

[Loops] Use forgetLcssaPhiWithNewPredecessor() in more places

Use the more aggressive invalidation method in a number of places
that add incoming values to lcssa phi nodes. It is likely that
it's po

[Loops] Use forgetLcssaPhiWithNewPredecessor() in more places

Use the more aggressive invalidation method in a number of places
that add incoming values to lcssa phi nodes. It is likely that
it's possible to construct cases with incorrect SCEV preservation
similar to https://github.com/llvm/llvm-project/issues/109333 for
these.

show more ...

[Transforms] Construct SmallVector with ArrayRef (NFC) (#101851)

[IR] Add getDataLayout() helpers to BasicBlock and Instruction (#96902)

This is a helper to avoid writing `getModule()->getDataLayout()`. I
regularly try to use this method only to remember it does

[IR] Add getDataLayout() helpers to BasicBlock and Instruction (#96902)

This is a helper to avoid writing `getModule()->getDataLayout()`. I
regularly try to use this method only to remember it doesn't exist...

`getModule()->getDataLayout()` is also a common (the most common?)
reason why code has to include the Module.h header.

show more ...

[LoopUnroll] Use poison instead of undef for another preheader value

[LoopUnroll] Use poison instead of undef for preheader value

[llvm-project] Fix typo "seperate" (#95373)

[LoopUnroll] Consider convergence control tokens when unrolling (#91715)

- There is no restriction on a loop with controlled convergent
operations when
the relevant tokens are defined and used w

[LoopUnroll] Consider convergence control tokens when unrolling (#91715)

- There is no restriction on a loop with controlled convergent
operations when
the relevant tokens are defined and used within the loop.

- When a token defined outside a loop is used inside (also called a loop
convergence heart), unrolling is allowed only in the absence of
remainder or
runtime checks.

- When a token defined inside a loop is used outside, such a loop is
said to be
"extended". This loop can only be unrolled by also duplicating the
extended part
lying outside the loop. Such unrolling is disabled for now.

- Clean up loop hearts: When unrolling a loop with a heart, duplicating
the
heart will introduce multiple static uses of a convergence control token
in a
cycle that does not contain its definition. This violates the static
rules for
tokens, and needs to be cleaned up into a single occurrence of the
intrinsic.

- Spell out the initializer for UnrollLoopOptions to improve
readability.


Original implementation [D85605] by Nicolai Haehnle
<nicolai.haehnle@amd.com>.

show more ...

[RemoveDIs] Change remapDbgVariableRecord to remapDbgRecord (#91456)

We need to remap any DbgRecord, not just DbgVariableRecords.

This is the followup to #91447.

Co-authored-by: PietroGhg <pie

[RemoveDIs] Change remapDbgVariableRecord to remapDbgRecord (#91456)

We need to remap any DbgRecord, not just DbgVariableRecords.

This is the followup to #91447.

Co-authored-by: PietroGhg <pietro.ghiglio@codeplay.com>

show more ...

[RemoveDIs][NFC] Rename DPValue -> DbgVariableRecord (#85216)

This is the major rename patch that prior patches have built towards.
The DPValue class is being renamed to DbgVariableRecord, which re

[RemoveDIs][NFC] Rename DPValue -> DbgVariableRecord (#85216)

This is the major rename patch that prior patches have built towards.
The DPValue class is being renamed to DbgVariableRecord, which reflects
the updated terminology for the "final" implementation of the RemoveDI
feature. This is a pure string substitution + clang-format patch. The
only manual component of this patch was determining where to perform
these string substitutions: `DPValue` and `DPV` are almost exclusively
used for DbgRecords, *except* for:

- llvm/lib/target, where 'DP' is used to mean double-precision, and so
appears as part of .td files and in variable names. NB: There is a
single existing use of `DPValue` here that refers to debug info, which
I've manually updated.
- llvm/tools/gold, where 'LDPV' is used as a prefix for symbol
visibility enums.

Outside of these places, I've applied several basic string
substitutions, with the intent that they only affect DbgRecord-related
identifiers; I've checked them as I went through to verify this, with
reasonable confidence that there are no unintended changes that slipped
through the cracks. The substitutions applied are all case-sensitive,
and are applied in the order shown:

```
DPValue -> DbgVariableRecord
DPVal -> DbgVarRec
DPV -> DVR
```

Following the previous rename patches, it should be the case that there
are no instances of any of these strings that are meant to refer to the
general case of DbgRecords, or anything other than the DPValue class.
The idea behind this patch is therefore that pure string substitution is
correct in all cases as long as these assumptions hold.

show more ...

[RemoveDIs][NFC] Rename common interface functions for DPValues->DbgRecords (#84793)

As part of the effort to rename the DbgRecord classes, this patch
renames the widely-used functions that operate

[RemoveDIs][NFC] Rename common interface functions for DPValues->DbgRecords (#84793)

As part of the effort to rename the DbgRecord classes, this patch
renames the widely-used functions that operate on DbgRecords but refer
to DbgValues or DPValues in their names to refer to DbgRecords instead;
all such functions are defined in one of `BasicBlock.h`,
`Instruction.h`, and `DebugProgramInstruction.h`.

This patch explicitly does not change the names of any comments or
variables, except for where they use the exact name of one of the
renamed functions. The reason for this is reviewability; this patch can
be trivially examined to determine that the only changes are direct
string substitutions and any results from clang-format responding to the
changed line lengths. Future patches will cover renaming variables and
comments, and then renaming the classes themselves.

show more ...

[LoopUnroll] Fix missing sign extension

For integers larger than 64-bit, this would zero-extend a -1
value, instead of sign-extending it.

Fixes https://github.com/llvm/llvm-project/issues/80289.

[DebugInfo][RemoveDIs] Support cloning and remapping DPValues (#72546)

This patch adds support for CloneBasicBlock duplicating the DPValues
attached to instructions, and adds facilities to remap the

[DebugInfo][RemoveDIs] Support cloning and remapping DPValues (#72546)

This patch adds support for CloneBasicBlock duplicating the DPValues
attached to instructions, and adds facilities to remap them into their new
context. The plumbing to achieve this is fairly straightforwards and
mechanical.

I've also added illustrative uses to LoopUnrollRuntime, SimpleLoopUnswitch
and SimplifyCFG. The former only updates for the epilogue right now so I've
added CHECK lines just for the end of an unrolled loop (further updates
coming later). SimpleLoopUnswitch had no debug-info tests so I've added a
new one. The two modified parts of SimplifyCFG are covered by the two
modified SimplifyCFG tests.

These are scenarios where we have to do extra cloning for copying of
DPValues because they're no longer instructions, and remap them too.

show more ...

LoopUnrollRuntime: Add weights to all branches

Make sure every conditional branch constructed by `LoopUnrollRuntime`
code sets branch weights.

- Add new 1:127 weights for the conditional jumps chec

LoopUnrollRuntime: Add weights to all branches

Make sure every conditional branch constructed by `LoopUnrollRuntime`
code sets branch weights.

- Add new 1:127 weights for the conditional jumps checking whether the
whole (unrolled) loop should be skipped in the generated prolog or
epilog code.
- Remove `updateLatchBranchWeightsForRemainderLoop` function and just
add weights immediately when constructing the relevant branches. This
leads to simpler code and makes the code more obvious as every call
to `CreateCondBr` now has a `BranchWeights` parameter.
- Rework formula for epilogue latch weights, to assume equal
distribution of remainders and remove `assert` (as I was able to
reach this code when forcing small unroll factors on the commandline).

Differential Revision: https://reviews.llvm.org/D158642

show more ...

[NFC][RemoveDIs] Prefer iterator-insertion over instructions

Continuing the patch series to get rid of debug intrinsics [0], instruction
insertion needs to be done with iterators rather than instruc

[NFC][RemoveDIs] Prefer iterator-insertion over instructions

Continuing the patch series to get rid of debug intrinsics [0], instruction
insertion needs to be done with iterators rather than instruction pointers,
so that we can communicate information in the iterator class. This patch
adds an iterator-taking insertBefore method and converts various call sites
to take iterators. These are all sites where such debug-info needs to be
preserved so that a stage2 clang can be built identically; it's likely that
many more will need to be changed in the future.

At this stage, this is just changing the spelling of a few operations,
which will eventually become signifiant once the debug-info bearing
iterator is used.

[0] https://discourse.llvm.org/t/rfc-instruction-api-changes-needed-to-eliminate-debug-intrinsics-from-ir/68939

Differential Revision: https://reviews.llvm.org/D152537

show more ...

[LoopUnrollRuntime] Allow indirect transition to deopt non-latch exit blocks

Relax condition on runtime trip count unrolling loops with 1 non-latch exit
that leads to a deop block.

There are cases

[LoopUnrollRuntime] Allow indirect transition to deopt non-latch exit blocks

Relax condition on runtime trip count unrolling loops with 1 non-latch exit
that leads to a deop block.

There are cases when the deopt blocks are common exits for different loops.
LoopSimplify pass splits such edges to the common deopting blocks to make
sure that all exit nodes of the loop only have predecessors that are inside
of the loop (See simplifyOneLoop()). This breaks the current condition for
unrolling. This patch allows the split transitive blocks that still lead to
the deopting blocks.

Differential Revision: https://reviews.llvm.org/D152639

show more ...

[Transforms,InstCombine] std::optional::value => operator*/operator->

value() has undesired exception checking semantics and calls
__throw_bad_optional_access in libc++. Moreover, the API is unavail

[Transforms,InstCombine] std::optional::value => operator*/operator->

value() has undesired exception checking semantics and calls
__throw_bad_optional_access in libc++. Moreover, the API is unavailable without
_LIBCPP_NO_EXCEPTIONS on older Mach-O platforms (see
_LIBCPP_AVAILABILITY_BAD_OPTIONAL_ACCESS).

show more ...

[NFC] Cleanup: Replace Function::getBasicBlockList().splice() with Function::splice()

This is part of a series of patches that aim at making Function::getBasicBlockList() private.

Differential Revi

[NFC] Cleanup: Replace Function::getBasicBlockList().splice() with Function::splice()

This is part of a series of patches that aim at making Function::getBasicBlockList() private.

Differential Revision: https://reviews.llvm.org/D139984

show more ...

Transforms/Utils: llvm::Optional => std::optional

[Transforms] Fix comment typos (NFC)

[llvm][NFC] Refactor code to use ProfDataUtils

In this patch we replace common code patterns with the use of utility
functions for dealing with profiling metadata. There should be no change
in funct

[llvm][NFC] Refactor code to use ProfDataUtils

In this patch we replace common code patterns with the use of utility
functions for dealing with profiling metadata. There should be no change
in functionality, as the existing checks should be preserved in all
cases.

Reviewed By: bogner, davidxl

Differential Revision: https://reviews.llvm.org/D128860

show more ...

Revert "[llvm][NFC] Refactor code to use ProfDataUtils"

This reverts commit 300c9a78819b4608b96bb26f9320bea6b8a0c4d0.

We will reland once these issues are ironed out.

[llvm][NFC] Refactor code to use ProfDataUtils

In this patch we replace common code patterns with the use of utility
functions for dealing with profiling metadata. There should be no change
in funct

[llvm][NFC] Refactor code to use ProfDataUtils

In this patch we replace common code patterns with the use of utility
functions for dealing with profiling metadata. There should be no change
in functionality, as the existing checks should be preserved in all
cases.

Reviewed By: bogner, davidxl

Differential Revision: https://reviews.llvm.org/D128860

show more ...