[CodeGen][NewPM] Port machine-branch-prob to new pass manager (#96389)

Like IR version `print<branch-prob>`, there is also a
`print<machine-branch-prob>`.

[MachineSink] Use SmallDenseMap (NFC) (#95676)

The use of SmallDenseMap saves 0.39% of heap allocations during the
compilation of a large preprocessed file, namely X86ISelLowering.cpp,
for the X86

[MachineSink] Use SmallDenseMap (NFC) (#95676)

The use of SmallDenseMap saves 0.39% of heap allocations during the
compilation of a large preprocessed file, namely X86ISelLowering.cpp,
for the X86 target.

show more ...

[CodeGen][NewPM] Split `MachinePostDominators` into a concrete analysis result (#95113)

`MachinePostDominators` version of #94571.

[CodeGen][NewPM] Split `MachineDominatorTree` into a concrete analysis result (#94571)

Prepare for new pass manager version of `MachineDominatorTreeAnalysis`.
We may need a machine dominator tree v

[CodeGen][NewPM] Split `MachineDominatorTree` into a concrete analysis result (#94571)

Prepare for new pass manager version of `MachineDominatorTreeAnalysis`.
We may need a machine dominator tree version of `DomTreeUpdater` to
handle `SplitCriticalEdge` in some CodeGen passes.

show more ...

[CodeGen] Make the parameter TRI required in some functions. (#85968)

Fixes #82659

There are some functions, such as `findRegisterDefOperandIdx` and `findRegisterDefOperand`, that have too many

[CodeGen] Make the parameter TRI required in some functions. (#85968)

Fixes #82659

There are some functions, such as `findRegisterDefOperandIdx` and `findRegisterDefOperand`, that have too many default parameters. As a result, we have encountered some issues due to the lack of TRI parameters, as shown in issue #82411.

Following @RKSimon 's suggestion, this patch refactors 9 functions, including `{reads, kills, defines, modifies}Register`, `registerDefIsDead`, and `findRegister{UseOperandIdx, UseOperand, DefOperandIdx, DefOperand}`, adjusting the order of the TRI parameter and making it required. In addition, all the places that call these functions have also been updated correctly to ensure no additional impact.

After this, the caller of these functions should explicitly know whether to pass the `TargetRegisterInfo` or just a `nullptr`.

show more ...

[CodeGen] Simplify updateLiveIn in MachineSink (#79831)

When a whole register is added a basic block's liveins, use
LaneBitmask::getAll for the live lanes instead of trying to calculate an
accurat

[CodeGen] Simplify updateLiveIn in MachineSink (#79831)

When a whole register is added a basic block's liveins, use
LaneBitmask::getAll for the live lanes instead of trying to calculate an
accurate mask of the lanes that comprise the register.

This simplifies the code and matches other places where a whole register
is marked as livein.

This also avoids problems when regunits that are synthesized by TableGen
to represent ad hoc aliasing have a lane mask of 0.

Fixes #78942

show more ...

[MachineSink] Clear kill flags of sunk addressing mode registers (#75072)

When doing sink-and-fold, the MachineSink clears the "killed" flags of
the operands of the sunk (and deleted) instruction.

[MachineSink] Clear kill flags of sunk addressing mode registers (#75072)

When doing sink-and-fold, the MachineSink clears the "killed" flags of
the operands of the sunk (and deleted) instruction. However, this is not
always sufficient. In some cases we can create the new load/store
instruction with operands other than the ones present in the deleted
instruction. One such example is folding a zero word extend into a
memory load on AArch64. The zero-extend is represented by a pair of
instructions - `MOV` (i.e. `ORRwrs`) followed by a `SUBREG_TO_REG`. The
`SUBREG_TO_REG` is deleted (it is the sunk instruction), but the new
load instruction mentions operands "killed" in the `MOV`, which is no
longer correct.

To fix this, clear the "killed" flags of the registers participating in
the addressing mode.

show more ...

[MachineSink] Some more preserving of debug location when rematerialising an instruction to replace a COPY (#73155)

Somewhat similar to ef9bcace834e63f25bbbc5e8e2b615f89d85fb2f
([MachineSink][AArch

[MachineSink] Some more preserving of debug location when rematerialising an instruction to replace a COPY (#73155)

Somewhat similar to ef9bcace834e63f25bbbc5e8e2b615f89d85fb2f
([MachineSink][AArch64] Preserve debug location when rematerialising
an instruction to replace a COPY (#72685))
reuse the debug location of the COPY, iff the rematerialised instruction
did not have a location.

Fixes a regression in `DebugInfo/AArch64/constant-dbgloc.ll` after
enabling sink-and-fold.

show more ...

[MachineSink][AArch64] Preserve debug location when rematerialising an instruction to replace a COPY (#72685)

Fixes a regression in `tools/lldb-dap/optimized/TestDAP_optimized.py`
caused by enablin

[MachineSink][AArch64] Preserve debug location when rematerialising an instruction to replace a COPY (#72685)

Fixes a regression in `tools/lldb-dap/optimized/TestDAP_optimized.py`
caused by enabling "sink-and-fold" in MachineSink.

show more ...

[MachineSink] Drop debug info for instructions deleted by sink-and-fold (#71443)

After performing sink-and-fold over a COPY, the original instruction is
replaced with one that produces its output i

[MachineSink] Drop debug info for instructions deleted by sink-and-fold (#71443)

After performing sink-and-fold over a COPY, the original instruction is
replaced with one that produces its output in the destination of the
copy. Its value is still available (in a hard register), so if there are
debug instructions which refer to the (now deleted) virtual register
they could be updated to refer to the hard register, in principle.
However, it's not clear how to do that, moreover in some cases the debug
instructions may need to be replicated proportionally to the number of
the COPY instructions replaced and in some extreme cases we can end up
with quadratic increase in the number of debug instructions, e.g:

int f(int);

void g(int x) {
int y = x + 1;

int t0 = y;
f(t0);

int t1 = y;
f(t1);
}

show more ...

[MachineSink] Reduce the number of unnecessary invalidations of StoreInstrCache (NFC) (#68676)

Don't invalidate the cache when erasing instructions which cannot ever
appear in the cache.

[MachineSink] Use LLVM ADTs (NFC) (#68677)

Replace a few uses of `std::map` with `llvm::DenseMap`.

[MachineSink] Fix crash due to use-after-free in a MachineInstr* cache.

After the SinkAndFold optimization was enabled, we saw some crashes with
GISel due to SinkAndFold erasing an MI while a refere

[MachineSink] Fix crash due to use-after-free in a MachineInstr* cache.

After the SinkAndFold optimization was enabled, we saw some crashes with
GISel due to SinkAndFold erasing an MI while a reference was being held in a
cache.

show more ...

AMDGPU: Fix temporal divergence introduced by machine-sink (#67456)

Temporal divergence that was present in input or introduced in IR
transforms, like code-sinking or LICM, is handled in SIFixSGPRCo

AMDGPU: Fix temporal divergence introduced by machine-sink (#67456)

Temporal divergence that was present in input or introduced in IR
transforms, like code-sinking or LICM, is handled in SIFixSGPRCopies
by changing sgpr source instr to vgpr instr.
After 5b657f5, that moved LICM after AMDGPUCodeGenPrepare,
machine-sinking can introduce temporal divergence by sinking
instructions outside of the cycle.
Add isSafeToSink callback in TargetInstrInfo.

show more ...

Revert "MachineSink: Fix sinking VGPR def out of a divergent loop"

This reverts commit 3f8ef57bede94445b1a1042c987cc914a886e7ff.

[AArch64] Fix an incorrect handling of debug values in MachineSink (#68107)

[AArch64] Fix a compiler crash in MachineSink (#67705)

There were a couple of issues with maintaining register def/uses held
in `MachineRegisterInfo`:

* when an operand is changed from one regis

[AArch64] Fix a compiler crash in MachineSink (#67705)

There were a couple of issues with maintaining register def/uses held
in `MachineRegisterInfo`:

* when an operand is changed from one register to another, the
corresponding instruction must already be inserted into the function,
or MRI won't be updated

* when traversing the set of all uses of a register, that set must not
change

show more ...

[MachineSink][AArch64] Sink instruction copies when they can replace copy into hard register or folded into addressing mode

This patch adds a new code transformation to the `MachineSink` pass,
that

[MachineSink][AArch64] Sink instruction copies when they can replace copy into hard register or folded into addressing mode

This patch adds a new code transformation to the `MachineSink` pass,
that tries to sink copies of an instruction, when the copies can be folded
into the addressing modes of load/store instructions, or
replace another instruction (currently, copies into a hard register).

The criteria for performing the transformation is that:
* the register pressure at the sink destination block must not
exceed the register pressure limits
* the latency and throughput of the load/store or the copy must not deteriorate
* the original instruction must be deleted

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D152828

show more ...

[CodeGen] Set regunitmasks for leaf regs to all instead of none

This simplifies every use of MCRegUnitMaskIterator.

Differential Revision: https://reviews.llvm.org/D157864

Remove a reference to rdar://problem/8030636

The surrounding comment has more than enough context to describe the problem.

MachineSink: Fix strict weak ordering in GetAllSortedSuccessors

CodeGen/X86/pseudo_cmov_lower2.ll fails using libc++ debug mode
(D150264) without this change.

Reviewed By: MaskRay, aeubanks

Differ

MachineSink: Fix strict weak ordering in GetAllSortedSuccessors

CodeGen/X86/pseudo_cmov_lower2.ll fails using libc++ debug mode
(D150264) without this change.

Reviewed By: MaskRay, aeubanks

Differential Revision: https://reviews.llvm.org/D155811

show more ...

MachineSink: Fix sinking VGPR def out of a divergent loop

This fixes sinking a VGPR def out of a loop past the reconvergence
point at the SI_END_CF. There was a prior fix which introduced
blockProlo

MachineSink: Fix sinking VGPR def out of a divergent loop

This fixes sinking a VGPR def out of a loop past the reconvergence
point at the SI_END_CF. There was a prior fix which introduced
blockPrologueInterferes (D121277) to fix the same basic problem for
the post RA sink. This also had the special case isIgnorableUse case
which was incorrect, because in some contexts the exec use is not
ignorable.

I'm thinking about a new way to represent this which will avoid
needing hasIgnorableUse and isBasicBlockPrologue, which would function
more like the exception handling.

Fixes: SWDEV-407790

https://reviews.llvm.org/D155343

show more ...

MachineSink: Remove unnecessary empty block check

MachineSink: Move helper function and use more const

[MC] Add MCRegisterInfo::regunits for iteration over register units

Reviewed By: foad

Differential Revision: https://reviews.llvm.org/D152098