[GISel] Explicitly disable BF16 tablegen patterns. (#124113)

We currently have an issue where bf16 patters can be used to match fp16
types, as GISel does not know about the difference between the t

[GISel] Explicitly disable BF16 tablegen patterns. (#124113)

We currently have an issue where bf16 patters can be used to match fp16
types, as GISel does not know about the difference between the two. This
patch explicitly disables them to make sure that they are never used.

The opposite can also happen too, where fp16 patterns are used for
operators that should be bf16. So this also changes any operations with
bf16 types to now cause a fallback to SDAG.

The pass setup for GISel has been slightly adjusted to make sure that a
verify pass does not get added between AMD-SDAG and SIFixSGPRCopiesPass,
which otherwise can cause verifier issues when falling back.

show more ...

[NFC][DebugInfo] Use iterator-flavour getFirstNonPHI at many call-sites (#123737)

As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNo

[NFC][DebugInfo] Use iterator-flavour getFirstNonPHI at many call-sites (#123737)

As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNonPHI and similar feed into
instruction insertion positions. Call-sites where that's necessary were
updated a year ago; but to ensure some type safety however, we'd like to
have all calls to getFirstNonPHI use the iterator-returning version.

This patch changes a bunch of call-sites calling getFirstNonPHI to use
getFirstNonPHIIt, which returns an iterator. All these call sites are
where it's obviously safe to fetch the iterator then dereference it. A
follow-up patch will contain less-obviously-safe changes.

We'll eventually deprecate and remove the instruction-pointer
getFirstNonPHI, but not before adding concise documentation of what
considerations are needed (very few).

---------

Co-authored-by: Stephen Tozer <Melamoto@gmail.com>

show more ...

[GISel] Add generic implementation for @llvm.expect.with.probability when optimizations are disabled (#117835)

Handle @llvm.expect.with.probability in GlobalISel in the same way
@llvm.expect is han

[GISel] Add generic implementation for @llvm.expect.with.probability when optimizations are disabled (#117835)

Handle @llvm.expect.with.probability in GlobalISel in the same way
@llvm.expect is handled, passing the value through as-is. This can be
encountered if the intrinsic is used without optimizations, which would
otherwise transform it out.

Fixes #115411 for GlobalISel

show more ...

[llvm] Remove redundant control flow statements (NFC) (#115831)

Identified with readability-redundant-control-flow.

[GlobalISel] Import llvm.stepvector (#115721)

[AArch64][GlobalISel] Do not create LIFETIME instructions in functions. (#115669)

For the same reason that we do not translate lifetime markers in a -O0,
we should not translate them for optnone fu

[AArch64][GlobalISel] Do not create LIFETIME instructions in functions. (#115669)

For the same reason that we do not translate lifetime markers in a -O0,
we should not translate them for optnone functions too.

show more ...

[GlobalISel] Remove unused includes (NFC) (#115429)

Identified with misc-include-cleaner.

[GlobalISel][AArch64] Legalize G_EXTRACT_VECTOR_ELT for SVE (#115161)

AArch64InstrGISel.td defines:
def : GINodeEquiv<G_EXTRACT_VECTOR_ELT, vector_extract>;

There are many patterns for SVE. Let'

[GlobalISel][AArch64] Legalize G_EXTRACT_VECTOR_ELT for SVE (#115161)

AArch64InstrGISel.td defines:
def : GINodeEquiv<G_EXTRACT_VECTOR_ELT, vector_extract>;

There are many patterns for SVE. Let's exploit that fact.

show more ...

[GlobalISel] Import samesign flag (#114267)

Credits: https://github.com/llvm/llvm-project/pull/111419

Fixes icmp-flags.mir

First attempt: https://github.com/llvm/llvm-project/pull/113090

Re

[GlobalISel] Import samesign flag (#114267)

Credits: https://github.com/llvm/llvm-project/pull/111419

Fixes icmp-flags.mir

First attempt: https://github.com/llvm/llvm-project/pull/113090

Revert: https://github.com/llvm/llvm-project/pull/114256

show more ...

Revert "[GlobalISel] Import samesign flag" (#114256)

Reverts llvm/llvm-project#113090

[GlobalISel] Import samesign flag (#113090)

Credits: https://github.com/llvm/llvm-project/pull/111419

[IR] Add `llvm.sincos` intrinsic (#109825)

This adds the `llvm.sincos` intrinsic, legalization, and lowering.

The `llvm.sincos` intrinsic takes a floating-point value and returns
both the sine a

[IR] Add `llvm.sincos` intrinsic (#109825)

This adds the `llvm.sincos` intrinsic, legalization, and lowering.

The `llvm.sincos` intrinsic takes a floating-point value and returns
both the sine and cosine (as a struct).

```
declare { float, float } @llvm.sincos.f32(float %Val)
declare { double, double } @llvm.sincos.f64(double %Val)
declare { x86_fp80, x86_fp80 } @llvm.sincos.f80(x86_fp80 %Val)
declare { fp128, fp128 } @llvm.sincos.f128(fp128 %Val)
declare { ppc_fp128, ppc_fp128 } @llvm.sincos.ppcf128(ppc_fp128 %Val)
declare { <4 x float>, <4 x float> } @llvm.sincos.v4f32(<4 x float> %Val)
```

The lowering is built on top of the existing FSINCOS ISD node, with
additional type legalization to allow for f16, f128, and vector values.

show more ...

[PowerPC][ISelLowering] Support -mstack-protector-guard=tls (#110928)

Add support for using a thread-local variable with a specified offset
for holding the stack guard canary value. This supports b

[PowerPC][ISelLowering] Support -mstack-protector-guard=tls (#110928)

Add support for using a thread-local variable with a specified offset
for holding the stack guard canary value. This supports both 32- and 64-
bit PowerPC targets.

This mirrors changes from #108942 but targeting PowerPC instead of
RISCV. Because both of these PRs modify the same driver functions, this
series is stack on top of the RISC-V one.

---------

Signed-off-by: Keith Packard <keithp@keithp.com>

show more ...

[CodeGen] Generalize trap emission after SP check fail (#109744)

Generalize and improve some target-specific code that emits traps after
stack protector failure in SelectionDAG & GlobalIsel.

[LLVM] Add HasFakeUses to MachineFunction (#110097)

Following the addition of the llvm.fake.use intrinsic and corresponding
MIR instruction, two further changes are planned: to add an
-fextend-lif

[LLVM] Add HasFakeUses to MachineFunction (#110097)

Following the addition of the llvm.fake.use intrinsic and corresponding
MIR instruction, two further changes are planned: to add an
-fextend-lifetimes flag to Clang that emits these intrinsics, and to
have -Og enable this flag by default. Currently, some logic for handling
fake uses is gated by the optdebug attribute, which is intended to be
switched on by -fextend-lifetimes (and by extension -Og later on).
However, the decision was made that a general optdebug attribute should
be incompatible with other opt_ attributes (e.g. optsize, optnone),
since they all express different intents for how to optimize the
program. We would still like to allow -fextend-lifetimes with optsize
however (i.e. -Os -fextend-lifetimes should be legal), since it may be a
useful configuration and there is no technical reason to not allow it.

This patch resolves this by tracking MachineFunctions that have fake
uses, allowing us to run passes that interact with them and skip passes
that clash with them.

show more ...

[GlobalISel] Import extract/insert subvector (#110287)

Test: AArch64/GlobalISel/irtranslator-subvector.ll

Reference:

https://llvm.org/docs/LangRef.html#llvm-vector-extract-intrinsic
https://

[GlobalISel] Import extract/insert subvector (#110287)

Test: AArch64/GlobalISel/irtranslator-subvector.ll

Reference:

https://llvm.org/docs/LangRef.html#llvm-vector-extract-intrinsic
https://llvm.org/docs/LangRef.html#llvm-vector-insert-intrinsic

show more ...

[SPIRV] Add atan2 function lowering (p2) (#110037)

This change is part of this proposal:
https://discourse.llvm.org/t/rfc-all-the-math-intrinsics/78294

- Add generic opcode for atan2
- Add SPIR

[SPIRV] Add atan2 function lowering (p2) (#110037)

This change is part of this proposal:
https://discourse.llvm.org/t/rfc-all-the-math-intrinsics/78294

- Add generic opcode for atan2
- Add SPIRV lowering for atan2

Part 2 for Implement the atan2 HLSL Function #70096.

show more ...

[LLVM] Use {} instead of std::nullopt to initialize empty ArrayRef (#109133)

It is almost always simpler to use {} instead of std::nullopt to
initialize an empty ArrayRef. This patch changes all oc

[LLVM] Use {} instead of std::nullopt to initialize empty ArrayRef (#109133)

It is almost always simpler to use {} instead of std::nullopt to
initialize an empty ArrayRef. This patch changes all occurrences I could
find in LLVM itself. In future the ArrayRef(std::nullopt_t) constructor
could be deprecated or removed.

show more ...

[CodeGen] Use Register in SwitchLoweringUtils. NFC (#109092)

Use an empty Register() instead of -1U.

[GlobalISel][AArch64] Add G_FPTOSI_SAT/G_FPTOUI_SAT (#96297)

This is an implementation of the saturating fp to int conversions for
GlobalISel. On AArch64 the converstion instrctions work this way,

[GlobalISel][AArch64] Add G_FPTOSI_SAT/G_FPTOUI_SAT (#96297)

This is an implementation of the saturating fp to int conversions for
GlobalISel. On AArch64 the converstion instrctions work this way,
producing saturating results. LegalizerHelper::lowerFPTOINT_SAT is
ported from SDAG.

AArch64 has a lot of existing tests for fptosi_sat, covering a wide
range of types. I have tried to make most of them work all at once, but
a few fall back due to other missing features such as f128 handling for
min/max.

show more ...

[IRTranslator][RISCV] Support scalable vector zeroinitializer. (#108666)

[IRTranslator] Simplify fixed vector ConstantAggregateZero handling. NFC (#108667)

We don't need to loop through the elements, they're all the same zero.
We can get the first element and create a s

[IRTranslator] Simplify fixed vector ConstantAggregateZero handling. NFC (#108667)

We don't need to loop through the elements, they're all the same zero.
We can get the first element and create a splat build_vector.

show more ...

[GISel] Use Function::getFunctionType() instead of getType() in some remarks. (#107651)

getType() on a Function is always 'ptr'. We should use getFunctionType()
so we get the function signature.

Add usub_cond and usub_sat operations to atomicrmw (#105568)

These both perform conditional subtraction, returning the minuend and
zero respectively, if the difference is negative.

[GISel][RISCV] Correctly handle scalable vector shuffles of pointer vectors in IRTranslator. (#106580)