[SelectDagISEL] refactor HandlePHINodesInSuccessorBlocks NFC.

While working on this code to support outputs from callbr along indirect
branches, I kept making these changes again and again. Precommi

[SelectDagISEL] refactor HandlePHINodesInSuccessorBlocks NFC.

While working on this code to support outputs from callbr along indirect
branches, I kept making these changes again and again. Precommit these.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D137445

show more ...

[SelectionDAGBuilder] use bitcast instead of AnyExtOrTrunc if copy parts from an int vector to a float vector to fix issue #58615

The getCopyFromPartsVector doesn't work correctly when PartEVT and V

[SelectionDAGBuilder] use bitcast instead of AnyExtOrTrunc if copy parts from an int vector to a float vector to fix issue #58615

The getCopyFromPartsVector doesn't work correctly when PartEVT and ValueVT have both different element type and different size.

This patch
1) removes the part of a comment that contains the incorrect assumption that element type are the same
2) use bitcast when copy parts of int vector to a float vector after the subvector extraction

Reviewed By: Peter, efriedma

Differential Revision: https://reviews.llvm.org/D136726

show more ...

[SPARC] Make calls to function with big return values work

Implement CanLowerReturn and associated CallingConv changes for SPARC/SPARC64.

In particular, for SPARC64 there's new `RetCC_Sparc64_*` fu

[SPARC] Make calls to function with big return values work

Implement CanLowerReturn and associated CallingConv changes for SPARC/SPARC64.

In particular, for SPARC64 there's new `RetCC_Sparc64_*` functions that handles the return case of the calling convention.
It uses the same analysis as `CC_Sparc64_*` family of funtions, but fails if the return value doesn't fit into the return registers.

This makes calls to functions with big return values converted to an sret function as expected, instead of crashing LLVM.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D132465

show more ...

[CodeGen] Using ZExt for extractelement indices.

In https://github.com/llvm/llvm-project/issues/57452, we found that IRTranslator is translating `i1 true` into `i32 -1`.
This is because IRTranslator

[CodeGen] Using ZExt for extractelement indices.

In https://github.com/llvm/llvm-project/issues/57452, we found that IRTranslator is translating `i1 true` into `i32 -1`.
This is because IRTranslator uses SExt for indices.

In this fix, we change the expected behavior of extractelement's index, moving from SExt to ZExt.
This change includes both documentation, SelectionDAG and IRTranslator.
We also included a test for AMDGPU, updated tests for AArch64, Mips, PowerPC, RISCV, VE, WebAssembly and X86

This patch fixes issue #57452.

Differential Revision: https://reviews.llvm.org/D132978

show more ...

[InlineAsm][bugfix] Correct function addressing in inline asm
In Linux PIC model, there are 4 cases about value/label addressing:
Case 1: Function call or Label jmp inside the module.
Case 2: Data ac

[InlineAsm][bugfix] Correct function addressing in inline asm
In Linux PIC model, there are 4 cases about value/label addressing:
Case 1: Function call or Label jmp inside the module.
Case 2: Data access (such as global variable, static variable) inside the module.
Case 3: Function call or Label jmp outside the module.
Case 4: Data access (such as global variable) outside the module.

Due to current llvm inline asm architecture designed to not "recognize" the asm
code, there are quite troubles for us to treat mem addressing differently for
same value/adress used in different instuctions.
For example, in pic model, call a func may in plt way or direclty pc-related,
but lea/mov a function adress may use got.

This patch fix/refine the case 1 and case 2 in inline asm.
Due to currently inline asm didn't support jmp the outsider lable, this patch
mainly focus on fix the function call addressing bugs in inline asm.

Reviewed By: Pengfei, RKSimon

Differential Revision: https://reviews.llvm.org/D133914

show more ...

[DAG] Extract helper for (neg x) [nfc]

This is a frequently reoccurring pattern, let's factor it out.

Differential Revision: https://reviews.llvm.org/D135301

[DAG] Introduce getSplat utility for common dispatch pattern [nfc]

We have a very common pattern of dispatching between BUILD_VECTOR and SPLAT_VECTOR creation repeated in many cases in code. Common

[DAG] Introduce getSplat utility for common dispatch pattern [nfc]

We have a very common pattern of dispatching between BUILD_VECTOR and SPLAT_VECTOR creation repeated in many cases in code. Common the pattern into a utility function.

show more ...

[DAG] Use getSplatBuildVector in a couple more places [nfc]

[RISCV][SelectionDAGBuilder] Fix crash when copying a v1f32 vector between basic blocks.

On a rv64 without f32 or vector support, this will be passed across
the basic block as an i64. We need use i3

[RISCV][SelectionDAGBuilder] Fix crash when copying a v1f32 vector between basic blocks.

On a rv64 without f32 or vector support, this will be passed across
the basic block as an i64. We need use i32 as an intermediate type
with bitcast and anyext/trunc.

Fixes PR58025

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D134758

show more ...

[VP][RISCV] Add vp.fmuladd.

Expanded in SelectionDAGBuilder similar to llvm.fmuladd.

Reviewed By: frasercrmck, simoll

Differential Revision: https://reviews.llvm.org/D134474

[PowerPC] XCOFF exception section support on the direct assembler path

This feature implements support for making entries in the exception section
on XCOFF on the direct assembly path using the ".ex

[PowerPC] XCOFF exception section support on the direct assembler path

This feature implements support for making entries in the exception section
on XCOFF on the direct assembly path using the ".except" pseudo-op. It also
provides functionality to lower entries (comprised of language and reason
codes) into the exception section through the use of annotation metadata
attached to llvm.ppc.trap/trapd/tw/tdw intrinsics. Integrated assembler
support will be provided in another review. https://reviews.llvm.org/D133030
needs to merge first for LIT tests

Reviewed By: shchenz, RKSimon

Differential Revision: https://reviews.llvm.org/D132146

show more ...

[RISCV] Disallow scale for scatter/gather

RISCV doesn't actually support a scaled form of indexed load and store. We previously handled this by forming the scaled SDNode, and then doing custom legal

[RISCV] Disallow scale for scatter/gather

RISCV doesn't actually support a scaled form of indexed load and store. We previously handled this by forming the scaled SDNode, and then doing custom legalization during lowering. This patch instead adds a callback via TLI to prevent formation entirely.

This has two effects:
* First, the GEP gets expanded (and used). Instead of the shift being created with an SDLoc of the memory operation, it has the SDLoc of the GEP instruction. This avoids the scheduler perturbing IR order when there's no reason to.
* Second, we fix what appears to be a bug in index calculation with RV32. The rules for GEPs require index calculation be done in particular bitwidth, and it appears the custom legalization code got this wrong for the case where index type exceeds pointer width. (Or at least, I trust the generic GEP lowering to be correct a lot more.)

The DAGCombiner change to handle VPScatter/VPGather is technically separate, but is required to prevent a regression on those intrinsics.

Differential Revision: https://reviews.llvm.org/D134382

show more ...

[SelectionDAGBuilder] Simplify how VTs is created for constrained intrinsics. NFC

All constrained intrinsics return a single value. We can directly
convert it to an EVT instead of going through Comp

[SelectionDAGBuilder] Simplify how VTs is created for constrained intrinsics. NFC

All constrained intrinsics return a single value. We can directly
convert it to an EVT instead of going through ComputeValueTypes.

show more ...

[SDAG] Split handling of VPLoad/VPGather and VPStore/VPScatter [nfc]

The merged routines are not-idiomatic, and the code sharing that results is prettty minimal. The confusion factor is not justifi

[SDAG] Split handling of VPLoad/VPGather and VPStore/VPScatter [nfc]

The merged routines are not-idiomatic, and the code sharing that results is prettty minimal. The confusion factor is not justified.

show more ...

SelectionDAG: Add AssumptionCache analysis dependency

Fixes compile time regression after
bb70b5d40652207c0bd3d385def10ef3ef1d45b4

Analysis: Add AssumptionCache argument to isDereferenceableAndAlignedPointer

This does not try to pass it through from the end users.

[SDAG] Add `getCALLSEQ_END` overload taking `uint64_t`s

All in-tree targets pass pointer-sized ConstantSDNodes to the
method. This overload reduced amount of boilerplate code a bit. This
also makes

[SDAG] Add `getCALLSEQ_END` overload taking `uint64_t`s

All in-tree targets pass pointer-sized ConstantSDNodes to the
method. This overload reduced amount of boilerplate code a bit. This
also makes getCALLSEQ_END consistent with getCALLSEQ_START, which
already takes uint64_ts.

show more ...

DAG: Sink some getter code closer to uses

CodeGen: Set MODereferenceable from isDereferenceableAndAlignedPointer

Previously this was assuming piontsToConstantMemory implies
dereferenceable.

[SelectionDAG] Propagate PCSections through SDNodes

Add a new entry to SDNodeExtraInfo to propagate PCSections through
SelectionDAG.

Reviewed By: vitalybuka

Differential Revision: https://reviews.

[SelectionDAG] Propagate PCSections through SDNodes

Add a new entry to SDNodeExtraInfo to propagate PCSections through
SelectionDAG.

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D130882

show more ...

KCFI sanitizer

The KCFI sanitizer, enabled with `-fsanitize=kcfi`, implements a
forward-edge control flow integrity scheme for indirect calls. It
uses a !kcfi_type metadata node to attach a type ide

KCFI sanitizer

The KCFI sanitizer, enabled with `-fsanitize=kcfi`, implements a
forward-edge control flow integrity scheme for indirect calls. It
uses a !kcfi_type metadata node to attach a type identifier for each
function and injects verification code before indirect calls.

Unlike the current CFI schemes implemented in LLVM, KCFI does not
require LTO, does not alter function references to point to a jump
table, and never breaks function address equality. KCFI is intended
to be used in low-level code, such as operating system kernels,
where the existing schemes can cause undue complications because
of the aforementioned properties. However, unlike the existing
schemes, KCFI is limited to validating only function pointers and is
not compatible with executable-only memory.

KCFI does not provide runtime support, but always traps when a
type mismatch is encountered. Users of the scheme are expected
to handle the trap. With `-fsanitize=kcfi`, Clang emits a `kcfi`
operand bundle to indirect calls, and LLVM lowers this to a
known architecture-specific sequence of instructions for each
callsite to make runtime patching easier for users who require this
functionality.

A KCFI type identifier is a 32-bit constant produced by taking the
lower half of xxHash64 from a C++ mangled typename. If a program
contains indirect calls to assembly functions, they must be
manually annotated with the expected type identifiers to prevent
errors. To make this easier, Clang generates a weak SHN_ABS
`__kcfi_typeid_<function>` symbol for each address-taken function
declaration, which can be used to annotate functions in assembly
as long as at least one C translation unit linked into the program
takes the function address. For example on AArch64, we might have
the following code:

```
.c:
int f(void);
int (*p)(void) = f;
p();

.s:
.4byte __kcfi_typeid_f
.global f
f:
...
```

Note that X86 uses a different preamble format for compatibility
with Linux kernel tooling. See the comments in
`X86AsmPrinter::emitKCFITypeId` for details.

As users of KCFI may need to locate trap locations for binary
validation and error handling, LLVM can additionally emit the
locations of traps to a `.kcfi_traps` section.

Similarly to other sanitizers, KCFI checking can be disabled for a
function with a `no_sanitize("kcfi")` function attribute.

Relands 67504c95494ff05be2a613129110c9bcf17f6c13 with a fix for
32-bit builds.

Reviewed By: nickdesaulniers, kees, joaomoreira, MaskRay

Differential Revision: https://reviews.llvm.org/D119296

show more ...

Revert "KCFI sanitizer"

This reverts commit 67504c95494ff05be2a613129110c9bcf17f6c13 as using
PointerEmbeddedInt to store 32 bits breaks 32-bit arm builds.

KCFI sanitizer

The KCFI sanitizer, enabled with `-fsanitize=kcfi`, implements a
forward-edge control flow integrity scheme for indirect calls. It
uses a !kcfi_type metadata node to attach a type ide

KCFI sanitizer

The KCFI sanitizer, enabled with `-fsanitize=kcfi`, implements a
forward-edge control flow integrity scheme for indirect calls. It
uses a !kcfi_type metadata node to attach a type identifier for each
function and injects verification code before indirect calls.

Unlike the current CFI schemes implemented in LLVM, KCFI does not
require LTO, does not alter function references to point to a jump
table, and never breaks function address equality. KCFI is intended
to be used in low-level code, such as operating system kernels,
where the existing schemes can cause undue complications because
of the aforementioned properties. However, unlike the existing
schemes, KCFI is limited to validating only function pointers and is
not compatible with executable-only memory.

KCFI does not provide runtime support, but always traps when a
type mismatch is encountered. Users of the scheme are expected
to handle the trap. With `-fsanitize=kcfi`, Clang emits a `kcfi`
operand bundle to indirect calls, and LLVM lowers this to a
known architecture-specific sequence of instructions for each
callsite to make runtime patching easier for users who require this
functionality.

A KCFI type identifier is a 32-bit constant produced by taking the
lower half of xxHash64 from a C++ mangled typename. If a program
contains indirect calls to assembly functions, they must be
manually annotated with the expected type identifiers to prevent
errors. To make this easier, Clang generates a weak SHN_ABS
`__kcfi_typeid_<function>` symbol for each address-taken function
declaration, which can be used to annotate functions in assembly
as long as at least one C translation unit linked into the program
takes the function address. For example on AArch64, we might have
the following code:

```
.c:
int f(void);
int (*p)(void) = f;
p();

.s:
.4byte __kcfi_typeid_f
.global f
f:
...
```

Note that X86 uses a different preamble format for compatibility
with Linux kernel tooling. See the comments in
`X86AsmPrinter::emitKCFITypeId` for details.

As users of KCFI may need to locate trap locations for binary
validation and error handling, LLVM can additionally emit the
locations of traps to a `.kcfi_traps` section.

Similarly to other sanitizers, KCFI checking can be disabled for a
function with a `no_sanitize("kcfi")` function attribute.

Reviewed By: nickdesaulniers, kees, joaomoreira, MaskRay

Differential Revision: https://reviews.llvm.org/D119296

show more ...

[ADT] Deprecate is_splat and replace all uses with all_equal

See the discussion thread for more details:
https://discourse.llvm.org/t/adt-is-splat-and-empty-ranges/64692

Reviewed By: dblaikie

Diff

[ADT] Deprecate is_splat and replace all uses with all_equal

See the discussion thread for more details:
https://discourse.llvm.org/t/adt-is-splat-and-empty-ranges/64692

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D132335

show more ...

Remove redundant initialization of Optional (NFC)