[aarch64][win] Update Called Globals info when updating Call Site info (#122762)

Fixes the "use after poison" issue introduced by #121516 (see
<https://github.com/llvm/llvm-project/pull/121516#issue

[aarch64][win] Update Called Globals info when updating Call Site info (#122762)

Fixes the "use after poison" issue introduced by #121516 (see
<https://github.com/llvm/llvm-project/pull/121516#issuecomment-2585912395>).

The root cause of this issue is that #121516 introduced "Called Global"
information for call instructions modeling how "Call Site" info is
stored in the machine function, HOWEVER it didn't copy the
copy/move/erase operations for call site information.

The fix is to rename and update the existing copy/move/erase functions
so they also take care of Called Global info.

show more ...

Fix MSVC "not all control paths return a value" warning. NFC.

[AArch64][PAC] Support BLRA* instructions in SLS Hardening pass (#98062)

Make SLS Hardening pass handle BLRA* instructions the same way it
handles BLR. The thunk names have the form

__llvm_s

[AArch64][PAC] Support BLRA* instructions in SLS Hardening pass (#98062)

Make SLS Hardening pass handle BLRA* instructions the same way it
handles BLR. The thunk names have the form

__llvm_slsblr_thunk_xN for BLR thunks
__llvm_slsblr_thunk_(aaz|abz)_xN for BLRAAZ and BLRABZ thunks
__llvm_slsblr_thunk_(aa|ab)_xN_xM for BLRAA and BLRAB thunks

Now there are about 1800 possible thunk names, so do not rely on linear
thunk function's name lookup and parse the name instead.

This patch reapplies llvm/llvm-project#97605.

show more ...

Revert "[AArch64][PAC] Support BLRA* instructions in SLS Hardening pass" (#97887)

This reverts commit 88b26293a24bdd85fce2b2f7191cc0a5bc0cecfe due to
failures of

CodeGen/AArch64/speculation-

Revert "[AArch64][PAC] Support BLRA* instructions in SLS Hardening pass" (#97887)

This reverts commit 88b26293a24bdd85fce2b2f7191cc0a5bc0cecfe due to
failures of

CodeGen/AArch64/speculation-hardening-sls-blra.mir

show more ...

[AArch64][PAC] Support BLRA* instructions in SLS Hardening pass (#97605)

Make SLS Hardening pass handle BLRA* instructions the same way it
handles BLR. The thunk names have the form

__llvm_s

[AArch64][PAC] Support BLRA* instructions in SLS Hardening pass (#97605)

Make SLS Hardening pass handle BLRA* instructions the same way it
handles BLR. The thunk names have the form

__llvm_slsblr_thunk_xN for BLR thunks
__llvm_slsblr_thunk_(aaz|abz)_xN for BLRAAZ and BLRABZ thunks
__llvm_slsblr_thunk_(aa|ab)_xN_xM for BLRAA and BLRAB thunks

Now there are about 1800 possible thunk names, so do not rely on linear
thunk function's name lookup and parse the name instead.

show more ...

[AArch64] When hardening against SLS, only create called thunks (#97472)

In preparation for implementing hardening of BLRA* instructions,
restrict thunk function generation to only the thunks bein

[AArch64] When hardening against SLS, only create called thunks (#97472)

In preparation for implementing hardening of BLRA* instructions,
restrict thunk function generation to only the thunks being actually
called from any function. As described in the existing comments,
emitting all possible thunks for BLRAA and BLRAB instructions would mean
adding about 1800 functions in total, most of which are likely not to be
called.

This commit merges AArch64SLSHardening class into SLSBLRThunkInserter,
so thunks can be created as needed while rewriting a machine function.
The usages of TII, TRI and ST fields of AArch64SLSHardening class are
replaced with requesting them in-place, as ThunkInserter assumes
multiple "entry points" in contrast to the only runOnMachineFunction
method of AArch64SLSHardening.

The runOnMachineFunction method essentially replaces pre-existing
insertThunks implementation as there is no more need to insert all
possible thunks unconditionally. Instead, thunks are created on first
use from inside of insertThunks method.

show more ...

[CodeGen] Refactor and document ThunkInserter (#97468)

In preparation for supporting BLRA* instructions in SLS Hardening on
AArch64, refactor ThunkInserter class.

The main intention of this comm

[CodeGen] Refactor and document ThunkInserter (#97468)

In preparation for supporting BLRA* instructions in SLS Hardening on
AArch64, refactor ThunkInserter class.

The main intention of this commit is to document the way to merge the
BLR-rewriting logic of the AArch64SLSHardening pass into the
SLSBLRThunkInserter class. This makes it possible to only call
createThunkFunction for the thunks that are actually referenced.
Ultimately, it will prevent SLSBLRThunkInserter from unconditionally
generating about 1800 thunk functions corresponding to every possible
combination of operands passed to BLRAA or BLRAB instructions.

This particular commit does not affect the generated machine code and
consists of the following changes:
* document the existing behavior of ThunkInserter class
* introduce ThunkInserterPass template class to get rid of mostly
identical boilerplate code in ARM, AArch64 and X86 implementations
* move the InsertedThunks parameter from `mayUseThunk` to `insertThunks`
method

show more ...

[AArch64] Move SLS later in pass pipeline (#84210)

Currently, the SLS hardening pass is run before the machine outliner,
which means that the outliner creates new functions and calls which do
not

[AArch64] Move SLS later in pass pipeline (#84210)

Currently, the SLS hardening pass is run before the machine outliner,
which means that the outliner creates new functions and calls which do
not have the SLS hardening applied.

The fix for this is to move the SLS passes to after the outliner, as has
recently been done for the return address signing pass.

This also avoids a bug where the SLS outliner emits code with
instructions after a return, which the outliner doesn't correctly
handle.

show more ...

[NFC] Use fold expressions to replace discarded initializer_lists (#83693)

[llvm] Use StringRef::{starts,ends}_with (NFC) (#74956)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::

[llvm] Use StringRef::{starts,ends}_with (NFC) (#74956)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.

I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.

show more ...

[Target] Remove unnecessary includes (NFC)

[llvm] Stop including llvm/ADT/BitVector.h (NFC)

Identified with clangd.

Revert "[AArch64] Move SLS later in pass pipeline"

The (MF.size() == 0) assertis is triggering when building at -O0.
Reverting this while I work out what is going wrong.

This reverts commit 7e8eccd

Revert "[AArch64] Move SLS later in pass pipeline"

The (MF.size() == 0) assertis is triggering when building at -O0.
Reverting this while I work out what is going wrong.

This reverts commit 7e8eccd990d37d2771ca5ad7a84f54c3cfc4a5e1.

show more ...

[AArch64] Move SLS later in pass pipeline

Currently, the SLS hardening pass is run before the machine outliner,
which means that the outliner creates new functions and calls which do
not have the SL

[AArch64] Move SLS later in pass pipeline

Currently, the SLS hardening pass is run before the machine outliner,
which means that the outliner creates new functions and calls which do
not have the SLS hardening applied.

The fix for this is to move the SLS passes to after the outliner, as has
recently been done for the return address signing pass.

This also avoids a bug where the SLS outliner emits code with
instructions after a return, which the outliner doesn't correctly
handle.

Reviewed By: kristof.beyls

Differential Revision: https://reviews.llvm.org/D158511

show more ...

[llvm][AArch64] Fix an interaction of SLS and BTI after a returns twice call

This fixes the combination of two things:
* Placing a BTI after calls to a returns twice function like setjmp.
This all

[llvm][AArch64] Fix an interaction of SLS and BTI after a returns twice call

This fixes the combination of two things:
* Placing a BTI after calls to a returns twice function like setjmp.
This allows the setjmp to return with a br instead of a ret.
* Straight line speculation mitigations that replace BLR with a BL
to a thunk that does the mitigation, and then goes to the original
target.

Originally I marked AArch64call_bti as requiring that SLS mitigation
be disabled. This caused a crash when you tried to codegen with both.
Since CALL_BTI tried to match with AArch64call_bti but could not.

This change does 2 things:
* Follow the pattern set by AArch64call and add 2 patterns for
AArch64call_bti. One with no IP (interprocedural) registers,
and one with. For SLS mitigation on and off respectively.
* Modify the sls hardening pass to iterate through bundled
instructions, as the AArch64 KCFI pass does.

Since there is a 1:1 replacement of the BLR with a BL,
the bundle remains intact. This is checked with an MIR test.

The ir -> asm testing is updated to add runs with the sls
mitigation enabled.

Reviewed By: kristof.beyls, pzheng

Differential Revision: https://reviews.llvm.org/D143915

show more ...

[ARM] Don't emit Arm speculation hardening thunks under Thumb and vice-versa

Given a patch like D129506, using instructions not valid for the current
target feature set becomes an error. This means

[ARM] Don't emit Arm speculation hardening thunks under Thumb and vice-versa

Given a patch like D129506, using instructions not valid for the current
target feature set becomes an error. This means that emitting Arm
instructions in a Thumb target (or vice versa) becomes an error. When
running in Thumb mode only thumb thunks will be needed, and in Arm mode
only arm thunks are needed. This patch limits the emitted thunks to just
the ones valid for the current architecture.

Differential Revision: https://reviews.llvm.org/D129693

show more ...

[NFC][CodeGen] Rename some functions in MachineInstr.h and remove duplicated comments

[ARM][AArch64] SLSHardening: make non-comdat thunks possible

Linker scripts might not handle COMDAT sections. SLSHardeing adds
new section for each __llvm_slsblr_thunk_xN. This new option allows
the

[ARM][AArch64] SLSHardening: make non-comdat thunks possible

Linker scripts might not handle COMDAT sections. SLSHardeing adds
new section for each __llvm_slsblr_thunk_xN. This new option allows
the generation of the thunks into the normal text section to handle these
exceptional cases.
,comdat or ,noncomdat can be added to harden-sls to control the codegen.
-mharden-sls=[all|retbr|blr],nocomdat.

Reviewed By: kristof.beyls

Differential Revision: https://reviews.llvm.org/D100546

show more ...

[NFC] Clean up uses of MachineModuleInfoWrapperPass

[AArch64] Avoid incompatibility between SLSBLR mitigation and BTI codegen.

A "BTI c" instruction only allows jumping/calling to using a BLR* instruction.
However, the SLSBLR mitigation changes a BLR

[AArch64] Avoid incompatibility between SLSBLR mitigation and BTI codegen.

A "BTI c" instruction only allows jumping/calling to using a BLR* instruction.
However, the SLSBLR mitigation changes a BLR to a BR to implement the
function call. Therefore, a "BTI c" check that passed before could
trigger after the BLR->BL change done by the SLSBLR mitigation.
However, if the register used in BR is X16 or X17, this trigger will not
fire (see ArmARM for further details).

Therefore, this patch simply changes the function stubs for the SLSBLR
mitigation from
__llvm_slsblr_thunk_x<N>:
br x<N>
SpeculationBarrier
to
__llvm_slsblr_thunk_x<N>:
mov x16, x<N>
br x16
SpeculationBarrier

Differential Revision: https://reviews.llvm.org/D81405

show more ...

[IndirectThunks] Make generated MF structure as expected by all instruction selectors.

This also enables running the AArch64 SLSHardening pass with GlobalISel,
so add a test for that.

Differential

[IndirectThunks] Make generated MF structure as expected by all instruction selectors.

This also enables running the AArch64 SLSHardening pass with GlobalISel,
so add a test for that.

Differential Revision: https://reviews.llvm.org/D81403

show more ...

[AArch64] SLSHardening: compute correct thunk name for X29.

The enum values for AArch64 registers are not all consecutive.
Therefore, the computation
"__llvm_slsblr_thunk_x" + utostr(Reg - AArch64

[AArch64] SLSHardening: compute correct thunk name for X29.

The enum values for AArch64 registers are not all consecutive.
Therefore, the computation
"__llvm_slsblr_thunk_x" + utostr(Reg - AArch64::X0)
is not always correct. utostr(Reg - AArch64::X0) will not generate the
expected string for the registers that do not have consecutive values in
the enum.
This happened to work for most registers, but does not for AArch64::FP
(i.e. register X29).
This can get triggered when the X29 is not used as a frame pointer.

Differential Revision: https://reviews.llvm.org/D81997

show more ...

[NFC] Silence compiler warning [-Wmissing-braces].

llvm/lib/Target/AArch64/AArch64SLSHardening.cpp:146:5: warning: suggest braces around initialization of subobject [-Wmissing-braces]
"__llvm_sl

[NFC] Silence compiler warning [-Wmissing-braces].

llvm/lib/Target/AArch64/AArch64SLSHardening.cpp:146:5: warning: suggest braces around initialization of subobject [-Wmissing-braces]
"__llvm_slsblr_thunk_x0", "__llvm_slsblr_thunk_x1",
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{
llvm/lib/Target/AArch64/AArch64SLSHardening.cpp:168:5: warning: suggest braces around initialization of subobject [-Wmissing-braces]
AArch64::X0, AArch64::X1, AArch64::X2, AArch64::X3, AArch64::X4,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{

show more ...

[AArch64] Extend AArch64SLSHardeningPass to harden BLR instructions.

To make sure that no barrier gets placed on the architectural execution
path, each
BLR x<N>
instruction gets transformed to a

[AArch64] Extend AArch64SLSHardeningPass to harden BLR instructions.

To make sure that no barrier gets placed on the architectural execution
path, each
BLR x<N>
instruction gets transformed to a
BL __llvm_slsblr_thunk_x<N>
instruction, with __llvm_slsblr_thunk_x<N> a thunk that contains
__llvm_slsblr_thunk_x<N>:
BR x<N>
<speculation barrier>

Therefore, the BLR instruction gets split into 2; one BL and one BR.
This transformation results in not inserting a speculation barrier on
the architectural execution path.

The mitigation is off by default and can be enabled by the
harden-sls-blr subtarget feature.

As a linker is allowed to clobber X16 and X17 on function calls, the
above code transformation would not be correct in case a linker does so
when N=16 or N=17. Therefore, when the mitigation is enabled, generation
of BLR x16 or BLR x17 is avoided.

As BLRA* indirect calls are not produced by LLVM currently, this does
not aim to implement support for those.

Differential Revision: https://reviews.llvm.org/D81402

show more ...

[AArch64] Introduce AArch64SLSHardeningPass, implementing hardening of RET and BR instructions.

Some processors may speculatively execute the instructions immediately
following RET (returns) and BR

[AArch64] Introduce AArch64SLSHardeningPass, implementing hardening of RET and BR instructions.

Some processors may speculatively execute the instructions immediately
following RET (returns) and BR (indirect jumps), even though
control flow should change unconditionally at these instructions.
To avoid a potential miss-speculatively executed gadget after these
instructions leaking secrets through side channels, this pass places a
speculation barrier immediately after every RET and BR instruction.

Since these barriers are never on the correct, architectural execution
path, performance overhead of this is expected to be low.

On targets that implement that Armv8.0-SB Speculation Barrier extension,
a single SB instruction is emitted that acts as a speculation barrier.
On other targets, a DSB SYS followed by a ISB is emitted to act as a
speculation barrier.

These speculation barriers are implemented as pseudo instructions to
avoid later passes to analyze them and potentially remove them.

Even though currently LLVM does not produce BRAA/BRAB/BRAAZ/BRABZ
instructions, these are also mitigated by the pass and tested through a
MIR test.

The mitigation is off by default and can be enabled by the
harden-sls-retbr subtarget feature.

Differential Revision: https://reviews.llvm.org/D81400

show more ...