[CodeGen] Avoid repeated hash lookups (NFC) (#123500)

[Statepoint] Treat undef operands less specially (#119682)

This reverts commit f7443905af1e06eaacda1e437fff8d54dc89c487.

This is to avoid an assertion if an undef operand appears in a
stackmap. Thi

[Statepoint] Treat undef operands less specially (#119682)

This reverts commit f7443905af1e06eaacda1e437fff8d54dc89c487.

This is to avoid an assertion if an undef operand appears in a
stackmap. This is important to avoid hitting verifier errors
when register allocation starts adding undefs in error scenarios.

Rather than trying to treat undef operands as special, leave them
alone and avoid producing an invalid spill. It would a bit more
precise to produce a spill of an undef register here, but that's not
exposed through the storeRegToStackSlot API.

https://reviews.llvm.org/D122605

This was an alternative to https://reviews.llvm.org/D122582

show more ...

[CodeGen] Make the parameter TRI required in some functions. (#85968)

Fixes #82659

There are some functions, such as `findRegisterDefOperandIdx` and `findRegisterDefOperand`, that have too many

[CodeGen] Make the parameter TRI required in some functions. (#85968)

Fixes #82659

There are some functions, such as `findRegisterDefOperandIdx` and `findRegisterDefOperand`, that have too many default parameters. As a result, we have encountered some issues due to the lack of TRI parameters, as shown in issue #82411.

Following @RKSimon 's suggestion, this patch refactors 9 functions, including `{reads, kills, defines, modifies}Register`, `registerDefIsDead`, and `findRegister{UseOperandIdx, UseOperand, DefOperandIdx, DefOperand}`, adjusting the order of the TRI parameter and making it required. In addition, all the places that call these functions have also been updated correctly to ensure no additional impact.

After this, the caller of these functions should explicitly know whether to pass the `TargetRegisterInfo` or just a `nullptr`.

show more ...

[AMDGPU] RA inserted scalar instructions can be at the BB top (#72140)

We adjust the insertion point at the BB top for spills/copies during RA
to ensure they are placed after the exec restore instr

[AMDGPU] RA inserted scalar instructions can be at the BB top (#72140)

We adjust the insertion point at the BB top for spills/copies during RA
to ensure they are placed after the exec restore instructions required
for the divergent control flow execution. This is, however, required
only for the vector operations. The insertions for scalar registers can
still go to the BB top.

show more ...

[Statepoint] Use correct RegisterClass for spilling

Copy propagation might have changed the register class of the register

Differential Revision: https://reviews.llvm.org/D155792

Use *{Map,Set}::contains (NFC)

Differential Revision: https://reviews.llvm.org/D146104

[CodeGen] Additional Register argument to storeRegToStackSlot/loadRegFromStackSlot

With D134950, targets get notified when a virtual register is created and/or
cloned. Targets can do the needful wit

[CodeGen] Additional Register argument to storeRegToStackSlot/loadRegFromStackSlot

With D134950, targets get notified when a virtual register is created and/or
cloned. Targets can do the needful with the delegate callback. AMDGPU propagates
the virtual register flags maintained in the target file itself. They are useful
to identify a certain type of machine operands while inserting spill stores and
reloads. Since RegAllocFast spills the physical register itself, there is no way
its virtual register can be mapped back to retrieve the flags. It can be solved
by passing the virtual register as an additional argument. This argument has no
use when the spill interfaces are called during the greedy allocator or even the
PrologEpilogInserter and can pass a null register in such cases.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D138656

show more ...

[Statepoints] FixupStatepoint: Clear isKill flag if COPY is not deleted.

When spilling CSRs, FixupStatepoint pass does simple copy propagation,
trying to find COPY instruction which defines register

[Statepoints] FixupStatepoint: Clear isKill flag if COPY is not deleted.

When spilling CSRs, FixupStatepoint pass does simple copy propagation,
trying to find COPY instruction which defines register being spilled
and spill COPY source instead. I.e., if we have CSR $x and found
$x = COPY $y
we will spill $y instead.
But we may be unable to delete COPY instruction for some reason.
Then, spill will be inserted after it, adding another use of $y.
If COPY instruction was last use of $y (killed it), after insertion of
the spill it is not, so `isKill` flag must be cleared. We failed to do
so and this patch fixes this issue.

Reviewed By: skatkov

Differential Revision: https://reviews.llvm.org/D127308

show more ...

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-in

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681

show more ...

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https:/

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169

show more ...

Cleanup codegen includes

after: 1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169

[llvm][clang][NFC] updates inline licence info

Some files still contained the old University of Illinois Open Source
Licence header. This patch replaces that with the Apache 2 with LLVM
Exception li

[llvm][clang][NFC] updates inline licence info

Some files still contained the old University of Illinois Open Source
Licence header. This patch replaces that with the Apache 2 with LLVM
Exception licence.

Differential Revision: https://reviews.llvm.org/D107528

show more ...

[Statepoint] Handle 'undef' operand tied to def

FixupStatepoints pass does not take into account the undef use
it skips may have a tied def. So when defs are handled pass
considers that tied-use sho

[Statepoint] Handle 'undef' operand tied to def

FixupStatepoints pass does not take into account the undef use
it skips may have a tied def. So when defs are handled pass
considers that tied-use should be spilled and triggers an assert.

FixupStatepoints should skip undef def as well.

Reviewers: reames, dantrushin
Reviewed By: dantrushin
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D95858

show more ...

[Statepoint] Handle `undef` operands in statepoint.

Currently when spilling statepoint register operands in FixupStatepoints
we do not pay attention that it might be `undef`. We just generate a
spil

[Statepoint] Handle `undef` operands in statepoint.

Currently when spilling statepoint register operands in FixupStatepoints
we do not pay attention that it might be `undef`. We just generate a
spill, which may lead to verifier error because we have a use without def.

To handle it, let FixupStateponts ignore `undef` register operands
completely and change them to some constant value when generating
stack map. Use same value as used by ISel for this purpose (0xFEFEFEFE).

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D94703

show more ...

[llvm] Use *::empty (NFC)

[CodeGen, Transforms] Use llvm::sort (NFC)

[Statepoints] Change statepoint machine instr format to better suit VReg lowering.

Current Statepoint MI format is this:

STATEPOINT
<id>, <num patch bytes >, <num call arguments>, <call targe

[Statepoints] Change statepoint machine instr format to better suit VReg lowering.

Current Statepoint MI format is this:

STATEPOINT
<id>, <num patch bytes >, <num call arguments>, <call target>,
[call arguments...],
<StackMaps::ConstantOp>, <calling convention>,
<StackMaps::ConstantOp>, <statepoint flags>,
<StackMaps::ConstantOp>, <num deopt args>, [deopt args...],
<gc base/derived pairs...> <gc allocas...>

Note that GC pointers are listed in pairs <base,derived>.
This causes base pointers to appear many times (at least twice) in
instruction, which is bad for us when VReg lowering is ON.
The problem is that machine operand tiedness is 1-1 relation, so
it might look like this:

%vr2 = STATEPOINT ... %vr1, %vr1(tied-def0)

Since only one instance of %vr1 is tied, that may lead to incorrect
codegen (see PR46917 for more details), so we have to always spill
base pointers. This mostly defeats new VReg lowering scheme.

This patch changes statepoint instruction format so that every
gc pointer appears only once in operand list. That way they all can
be tied. Additional set of operands is added to preserve base-derived
relation required to build stackmap.
New statepoint has following format:

STATEPOINT
<id>, <num patch bytes>, <num call arguments>, <call target>,
[call arguments...],
<StackMaps::ConstantOp>, <calling convention>,
<StackMaps::ConstantOp>, <statepoint flags>,
<StackMaps::ConstantOp>, <num deopt args>, [deopt args...],
<StackMaps::ConstantOp>, <num gc pointers>, [gc pointers...],
<StackMaps::ConstantOp>, <num gc allocas>, [gc allocas...]
<StackMaps::ConstantOp>, <num entries in gc map>, [base/derived indices...]

Changes are:
- every gc pointer is listed only once in a flat length-prefixed list;
- alloca list is prefixed with its length too;
- following alloca list is length-prefixed list of base-derived
indices of pointers from gc pointer list. Note that indices are
logical (number of pointer), not absolute (index of machine operand).

Differential Revision: https://reviews.llvm.org/D87154

show more ...

[NFC] Silence variables unused in release builds

[Statepoints] FixupStatepoint: properly set isKill on spilled register.

When spilling statepoint meta arg register it is incorrect to blindly
mark it as killed - it may be used in non-meta args (e.g

[Statepoints] FixupStatepoint: properly set isKill on spilled register.

When spilling statepoint meta arg register it is incorrect to blindly
mark it as killed - it may be used in non-meta args (e.g., as call
parameter).

show more ...

[Statepoints] Spill GC Ptr regs in FixupStatepoints.

Extend FixupStatepointCallerSaved pass with ability to spill
statepoint GC pointer arguments (optionally allowing them on CSRs).
Special handling

[Statepoints] Spill GC Ptr regs in FixupStatepoints.

Extend FixupStatepointCallerSaved pass with ability to spill
statepoint GC pointer arguments (optionally allowing them on CSRs).
Special handling is required for invoke statepoints, because at MI
level single landing pad may be shared by multiple statepoints, so
we must ensure we spill landing pad's live-ins into the same stack
slots.

Full statepoint refactoring change set is available at D81603.

Reviewed By: skatkov

Differential Revision: https://reviews.llvm.org/D81647

show more ...

[Alignment][NFC] Migrate MachineFrameInfo::CreateSpillStackObject to Align

iThis patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermai

[Alignment][NFC] Migrate MachineFrameInfo::CreateSpillStackObject to Align

iThis patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790

Differential Revision: https://reviews.llvm.org/D82934

show more ...

[Statepoint] Mark FixupStatepointCallerSaved as preserving the CFG

I'm hoping this will restore some compile time lost by D75936 and D75937.

Differential Revision: https://reviews.llvm.org/D79813

Give helpers internal linkage. NFC.

[Statepoint] Add getters to StatepointOpers.

To simplify future work on statepoint representation, hide
direct access to statepoint field indices and provide getters
for them. Add getters for couple

[Statepoint] Add getters to StatepointOpers.

To simplify future work on statepoint representation, hide
direct access to statepoint field indices and provide getters
for them. Add getters for couple more statepoint fields.

This also fixes two bugs in MachineVerifier for statepoint:
First, the `break` statement was falling out of `if` statement
scope, thus disabling following checks.
Second, it was incorrectly accessing some fields like CallingConv -
StatepointOpers gives index to their value directly, not to
preceeding field type encoding.

Reviewed By: skatkov
Differential Revision: https://reviews.llvm.org/D78119

show more ...

Re-land [Codegen/Statepoint] Allow usage of registers for non gc deopt values.

The change introduces the usage of physical registers for non-gc deopt values.
This require runtime support to know how

Re-land [Codegen/Statepoint] Allow usage of registers for non gc deopt values.

The change introduces the usage of physical registers for non-gc deopt values.
This require runtime support to know how to take a value from register.
By default usage is off and can be switched on by option.

The change also introduces additional fix-up patch which forces the spilling
of caller saved registers (clobbered after the call) and re-writes statepoint
to use spill slots instead of caller saved registers.

Reviewers: reames, danstrushin
Reviewed By: dantrushin
Subscribers: mgorny, hiraditya, mgrang, llvm-commits
Differential Revision: https://reviews.llvm.org/D77797

show more ...