[MCP] Do not try forward non-existent sub-register of a copy

In this example:
```
$d14 = COPY killed $d18
$s0 = MI $s28
```

$s28 is a sub-register of $d14. However, $d18 does not have
sub-registers

[MCP] Do not try forward non-existent sub-register of a copy

In this example:
```
$d14 = COPY killed $d18
$s0 = MI $s28
```

$s28 is a sub-register of $d14. However, $d18 does not have
sub-registers and thus cannot be forwarded. Previously, this resulted
in $noreg being substituted in place of the use of $s28, which later
led to an assertion failure.

Fixes https://github.com/llvm/llvm-project/issues/60908, a regression
that was introduced in D141747.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D146930

show more ...

[MachineCopyPropagation] Pass DestSourcePair to isBackwardPropagatableCopy. NFC

Instead of calling isCopyInstr again, just pass the DestSourcePair
from the isCopyInstr call from the caller.

[MachineCopyPropagation] Eliminate spillage copies that might be caused by eviction chain

Remove spill-reload like copy chains. For example
```
r0 = COPY r1
r1 = COPY r2
r2 = COPY r3
r3 = COPY r4
<d

[MachineCopyPropagation] Eliminate spillage copies that might be caused by eviction chain

Remove spill-reload like copy chains. For example
```
r0 = COPY r1
r1 = COPY r2
r2 = COPY r3
r3 = COPY r4
<def-use r4>
r4 = COPY r3
r3 = COPY r2
r2 = COPY r1
r1 = COPY r0
```
will be folded into
```
r0 = COPY r1
r1 = COPY r4
<def-use r4>
r4 = COPY r1
r1 = COPY r0
```

Reviewed By: qcolombet

Differential Revision: https://reviews.llvm.org/D122118

show more ...

Resolve a FIXME in MachineCopyPropagation by allowig propagation to subregister uses.

Reviewed By: barannikov88

Differential Revision: https://reviews.llvm.org/D141747

[Target] llvm::Optional => std::optional

The updated functions are mostly internal with a few exceptions (virtual functions in
TargetInstrInfo.h, TargetRegisterInfo.h).
To minimize changes to LLVMCo

[Target] llvm::Optional => std::optional

The updated functions are mostly internal with a few exceptions (virtual functions in
TargetInstrInfo.h, TargetRegisterInfo.h).
To minimize changes to LLVMCodeGen, GlobalISel files are skipped.

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[CodeGen] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of

[CodeGen] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[MachineCopyPropagation][RISCV] Fix D125335 accidentally change control flow.

D125335 makes regsOverlap skip following control flow, which is not entended
in the original code.

Differential Revisio

[MachineCopyPropagation][RISCV] Fix D125335 accidentally change control flow.

D125335 makes regsOverlap skip following control flow, which is not entended
in the original code.

Differential Revision: https://reviews.llvm.org/D128039

show more ...

Give option to use isCopyInstr to determine which MI is
treated as Copy instruction in MCP.

This is then used in AArch64 to remove copy instructions after taildup
ran in machine block placement

Dif

Give option to use isCopyInstr to determine which MI is
treated as Copy instruction in MCP.

This is then used in AArch64 to remove copy instructions after taildup
ran in machine block placement

Differential Revision: https://reviews.llvm.org/D125335

show more ...

[MachineCopyPropagation] More robust isForwardableRegClassCopy

Change the implementation of isForwardableRegClassCopy so that it
does not rely on getMinimalPhysRegClass. Instead, iterate over all
cl

[MachineCopyPropagation] More robust isForwardableRegClassCopy

Change the implementation of isForwardableRegClassCopy so that it
does not rely on getMinimalPhysRegClass. Instead, iterate over all
classes looking for any that satisfy a required property.

NFCI on current upstream targets, but this copes better with
downstream AMDGPU changes where some new smaller classes have been
introduced, which was breaking regclass equality tests in the old
code like:
if (UseDstRC != CrossCopyRC && CopyDstRC == CrossCopyRC)

Differential Revision: https://reviews.llvm.org/D121903

show more ...

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-in

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681

show more ...

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https:/

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169

show more ...

Cleanup codegen includes

after: 1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169

[CodeGen] Use range-based for loops (NFC)

[CodeGen] Use make_early_inc_range (NFC)

[MachineCopyPropagation] Handle propagation of undef copies

When propagating undefined copies the undef flag must also be
propagated.

Reviewed By: foad

Differential Revision: https://reviews.llvm.

[MachineCopyPropagation] Handle propagation of undef copies

When propagating undefined copies the undef flag must also be
propagated.

Reviewed By: foad

Differential Revision: https://reviews.llvm.org/D111219

show more ...

[MachineCopyPropagation] Check CrossCopyRegClass for cross-class copys

On some AMDGPU subtargets, copying to and from AGPR registers using another
AGPR register is not possible. A intermediate VGPR

[MachineCopyPropagation] Check CrossCopyRegClass for cross-class copys

On some AMDGPU subtargets, copying to and from AGPR registers using another
AGPR register is not possible. A intermediate VGPR register is needed for AGPR
to AGPR copy. This is an issue when machine copy propagation forwards a
COPY $agpr, replacing a COPY $vgpr which results in $agpr = COPY $agpr. It is
removing a cross class copy that may have been optimized by previous passes and
potentially creating an unoptimized cross class copy later on.

To avoid this issue, check CrossCopyRegClass if a different register class will
be needed for the copy. If so then avoid forwarding the copy when the
destination does not match the desired register class and if the original copy
already matches the desired register class.

Issue seen while attempting to optimize another AGPR to AGPR issue:

Live-ins: $agpr0
$vgpr0 = COPY $agpr0
$agpr1 = V_ACCVGPR_WRITE_B32 $vgpr0
$agpr2 = COPY $vgpr0
$agpr3 = COPY $vgpr0
$agpr4 = COPY $vgpr0

After machine-cp:

$vgpr0 = COPY $agpr0
$agpr1 = V_ACCVGPR_WRITE_B32 $vgpr0
$agpr2 = COPY $agpr0
$agpr3 = COPY $agpr0
$agpr4 = COPY $agpr0

Machine-cp propagated COPY $agpr0 to replace $vgpr0 creating 3 AGPR to AGPR
copys. Later this creates a cross-register copy from AGPR->VGPR->AGPR for each
copy when the prior VGPR->AGPR copy was already optimal.

Reviewed By: lkail, rampitec

Differential Revision: https://reviews.llvm.org/D108011

show more ...

[MachineCopyPropagation] Fix differences in code gen when compiling with -g

Fixes bugs [[ https://bugs.llvm.org/show_bug.cgi?id=50580 | 50580 ]] and [[ https://bugs.llvm.org/show_bug.cgi?id=49446 |

[MachineCopyPropagation] Fix differences in code gen when compiling with -g

Fixes bugs [[ https://bugs.llvm.org/show_bug.cgi?id=50580 | 50580 ]] and [[ https://bugs.llvm.org/show_bug.cgi?id=49446 | 49446 ]]

When compiling with -g "DBG_VALUE <reg>" instructions are added in the MIR, if such a instruction is inserted between instructions that use <reg> then MachineCopyPropagation invalidates <reg> , this causes some copies to not be propagated and causes differences in code generation (ex bugs 50580 and 49446 ). DBG_VALUE instructions should be ignored since they don't actually modify the register.

Reviewed By: lkail

Differential Revision: https://reviews.llvm.org/D104394

show more ...

Reapply "[DebugInfo] Fix updateDbgUsersToReg to support DBG_VALUE_LIST"

Previous crashes caused by this patch were the result of machine
subregisters being incorrectly handled in updateDbgUsersToReg

Reapply "[DebugInfo] Fix updateDbgUsersToReg to support DBG_VALUE_LIST"

Previous crashes caused by this patch were the result of machine
subregisters being incorrectly handled in updateDbgUsersToReg; this has
been fixed by using RegUnits to determine overlapping registers, instead
of using the register values directly.

Differential Revision: https://reviews.llvm.org/D101523

This reverts commit 7ca26c5fa2df253878cab22e1e2f0d6f1b481218.

show more ...

Revert "[DebugInfo] Fix updateDbgUsersToReg to support DBG_VALUE_LIST"

This reverts commit 0791f968fee259e5c34523167bd58179b8b081c2.

Causing crashes: https://crbug.com/1206764

[DebugInfo] Fix updateDbgUsersToReg to support DBG_VALUE_LIST

This patch modifies updateDbgUsersToReg to properly handle
DBG_VALUE_LIST instructions, by replacing the hard-coded operand indices
(i.e

[DebugInfo] Fix updateDbgUsersToReg to support DBG_VALUE_LIST

This patch modifies updateDbgUsersToReg to properly handle
DBG_VALUE_LIST instructions, by replacing the hard-coded operand indices
(i.e. getOperand(0)) with the more general getDebugOperandsForReg(), and
updating the register for all matching operands.

Differential Revision: https://reviews.llvm.org/D101523

show more ...

[NFC][Regalloc] Use MCRegister in MachineCopyPropagation

Differential Revision: https://reviews.llvm.org/D89250

[MachineCopyPropagation] In isNopCopy, check the destination registers match in addition to the source registers.

Previously if the source match we asserted that the destination
matched. But GPR <->

[MachineCopyPropagation] In isNopCopy, check the destination registers match in addition to the source registers.

Previously if the source match we asserted that the destination
matched. But GPR <-> mask register copies on X86 can violate this
since we use the same K-registers for multiple sizes.

Fixes this ISPC issue https://github.com/ispc/ispc/issues/1851

Differential Revision: https://reviews.llvm.org/D86507

show more ...

[MachineCopyPropagation] BackwardPropagatableCopy: add check for hasOverlappingMultipleDef

In MachineCopyPropagation::BackwardPropagatableCopy(),
a check is added for multiple destination registers.

[MachineCopyPropagation] BackwardPropagatableCopy: add check for hasOverlappingMultipleDef

In MachineCopyPropagation::BackwardPropagatableCopy(),
a check is added for multiple destination registers.

The copy propagation is avoided if the copied destination register
is the same register as another destination on the same instruction.

A new test is added. This used to fail on ARM like this:
error: unpredictable instruction, RdHi and RdLo must be different
umull r9, r9, lr, r0

Reviewed By: lkail

Differential Revision: https://reviews.llvm.org/D82638

show more ...

[NFCI][MachineCopyPropagation] invalidateRegister(): use SmallSet<8> instead of DenseSet.

This decreases the time consumed by the pass [during RawSpeed unity build]
by 25% (0.0586 s -> 0.04388 s).

[NFCI][MachineCopyPropagation] invalidateRegister(): use SmallSet<8> instead of DenseSet.

This decreases the time consumed by the pass [during RawSpeed unity build]
by 25% (0.0586 s -> 0.04388 s).

While that isn't really impressive overall, that wasn't the goal here.
The memory results here are noticeable.
The baseline results are:
```
total runtime: 55.65s.
calls to allocation functions: 19754254 (354960/s)
temporary memory allocations: 4951609 (88974/s)
peak heap memory consumption: 239.13MB
peak RSS (including heaptrack overhead): 463.79MB
total memory leaked: 198.01MB
```
While with this patch the results are:
```
total runtime: 55.37s.
calls to allocation functions: 19068237 (344403/s) # -3.47 %
temporary memory allocations: 4261772 (76974/s) # -13.93 % (!!!)
peak heap memory consumption: 239.13MB
peak RSS (including heaptrack overhead): 463.73MB
total memory leaked: 198.01MB
```

So we get rid of *a lot* of temporary allocations.

Using `SmallSet<8>` makes sense to me because at least here
for x86 BdVer2, the size of that set is *never* more than 3,
over all of llvm test-suite + RawSpeed.

The story might be different on other targets,
not sure if it will ever justify whole DenseSet,
but if it does SmallDenseSet might be a compromise.

show more ...

[MCP] Add stats for backward copy propagation. NFC.