[X86] Remove x86-experimental-unordered-atomic-isel option and associated code

This option enables an experimental lowering for unordered atomics I worked
on a few years back. It never reached prod

[X86] Remove x86-experimental-unordered-atomic-isel option and associated code

This option enables an experimental lowering for unordered atomics I worked
on a few years back. It never reached production quality, and hasn't been
worked on in years. So let's rip it out.

This wasn't a crazy idea, but I hit some stumbling block which prevented me
from pushing it across the finish line. From the look of 027aa27, that
change description is probably a good summary. I don't remember the
details any longer.

show more ...

[llvm][TypeSize] Fix addition/subtraction in TypeSize. (#72979)

It seems TypeSize is currently broken in the sense that:

TypeSize::Fixed(4) + TypeSize::Scalable(4) => TypeSize::Fixed(8)

with

[llvm][TypeSize] Fix addition/subtraction in TypeSize. (#72979)

It seems TypeSize is currently broken in the sense that:

TypeSize::Fixed(4) + TypeSize::Scalable(4) => TypeSize::Fixed(8)

without failing its assert that explicitly tests for this case:

assert(LHS.Scalable == RHS.Scalable && ...);

The reason this fails is that `Scalable` is a static method of class
TypeSize,
and LHS and RHS are both objects of class TypeSize. So this is
evaluating
if the pointer to the function Scalable == the pointer to the function
Scalable,
which is always true because LHS and RHS have the same class.

This patch fixes the issue by renaming `TypeSize::Scalable` ->
`TypeSize::getScalable`, as well as `TypeSize::Fixed` to
`TypeSize::getFixed`,
so that it no longer clashes with the variable in
FixedOrScalableQuantity.

The new methods now also better match the coding standard, which
specifies that:
* Variable names should be nouns (as they represent state)
* Function names should be verb phrases (as they represent actions)

show more ...

[DebugInfo][RemoveDIs] Interpret DPValue objects in SelectionDAG (#72253)

DPValues are the non-intrinsic replacements for dbg.values, and when an
IR function is converted by SelectionDAG we need to

[DebugInfo][RemoveDIs] Interpret DPValue objects in SelectionDAG (#72253)

DPValues are the non-intrinsic replacements for dbg.values, and when an
IR function is converted by SelectionDAG we need to convert the variable
location information in the same way. Happily all the information is in
the same format, it's just stored in a slightly different object,
therefore this patch refactors a few things to store the set of
{Variable,Expr,DILocation,Location} instead of just a pointer to a
DbgValueInst.

This also adds a hook in llc that's much like the one I've added to opt
in PR #71937, allowing tests to optionally ask for the use RemoveDIs
mode if support for it is built into the compiler.

I've added that flag to a variety of SelectionDAG debug-info tests to
ensure that we get some coverage on the RemoveDIs / debug-info-iterator
buildbot.

show more ...

[CodeGen][DebugInfo] Add missing debug info for jump table BB (#71021)

visitJumpTable is called on FinishBasicBlock. At that time, getCurSDLoc
will always return SDLoc without DebugLoc since CurInst

[CodeGen][DebugInfo] Add missing debug info for jump table BB (#71021)

visitJumpTable is called on FinishBasicBlock. At that time, getCurSDLoc
will always return SDLoc without DebugLoc since CurInst was set to
nullptr after visiting each instruction.
This patch passes SDLoc to buildJumpTable when visiting SwitchInst so
that visitJumpTable can use it later.

show more ...

[SelectionDAG] Handling Oversized Alloca Types under 32 bit Mode to Avoid Code Generator Crash (#71472)

Situations may arise leading to negative `NumElements` argument of an
`alloca` instruction. I

[SelectionDAG] Handling Oversized Alloca Types under 32 bit Mode to Avoid Code Generator Crash (#71472)

Situations may arise leading to negative `NumElements` argument of an
`alloca` instruction. In this case the `NumElements` is treated as a
large unsigned value. Such large arrays may cause the size constant to
overflow during code generation under 32 bit mode, leading to a crash.
This PR limits the constant's bit width to the width of the pointer on
the target. With this fix,
```
alloca i32, i32 -1
```
and
```
alloca [4294967295 x i32], i32 1
```
generates the exact same PowerPC assembly code under 32 bit mode.

show more ...

[AMDGPU] ISel for @llvm.amdgcn.cs.chain intrinsic (#68186)

The @llvm.amdgcn.cs.chain intrinsic is essentially a call. The call
parameters are bundled up into 2 intrinsic arguments, one for those th

[AMDGPU] ISel for @llvm.amdgcn.cs.chain intrinsic (#68186)

The @llvm.amdgcn.cs.chain intrinsic is essentially a call. The call
parameters are bundled up into 2 intrinsic arguments, one for those that
should go in the SGPRs (the 3rd intrinsic argument), and one for those
that should go in the VGPRs (the 4th intrinsic argument). Both will
often be some kind of aggregate.

Both instruction selection frameworks have some internal representation
for intrinsics (G_INTRINSIC[_WITH_SIDE_EFFECTS] for GlobalISel,
ISD::INTRINSIC_[VOID|WITH_CHAIN] for DAGISel), but we can't use those
because aggregates are dissolved very early on during ISel and we'd lose
the inreg information. Therefore, this patch shortcircuits both the
IRTranslator and SelectionDAGBuilder to lower this intrinsic as a call
from the very start. It tries to use the existing infrastructure as much
as possible, by calling into the code for lowering tail calls.

This has already gone through a few rounds of review in Phab:

Differential Revision: https://reviews.llvm.org/D153761

show more ...

[SelectionDAG] Add initial support for nneg flag on ISD::ZERO_EXTEND. (#70872)

This adds the nneg flag to SDNodeFlags and the node printing code.
SelectionDAGBuilder will add this flag to the node

[SelectionDAG] Add initial support for nneg flag on ISD::ZERO_EXTEND. (#70872)

This adds the nneg flag to SDNodeFlags and the node printing code.
SelectionDAGBuilder will add this flag to the node if the target doesn't
prefer sign extend.

A future RISC-V patch can remove the sign extend preference from
SelectionDAGBuilder.

I've also added the flag to the DAG combine that converts
ISD::SIGN_EXTEND to ISD::ZERO_EXTEND.

show more ...

[RISCV] Add experimental support for making i32 a legal type on RV64 in SelectionDAG. (#70357)

This will select i32 operations directly to W instructions without
custom nodes. Hopefully this can al

[RISCV] Add experimental support for making i32 a legal type on RV64 in SelectionDAG. (#70357)

This will select i32 operations directly to W instructions without
custom nodes. Hopefully this can allow us to be less dependent on
hasAllNBitUsers to recover i32 operations in RISCVISelDAGToDAG.cpp.

This support is enabled with a command line option that is off by
default.

Generated code is still not optimal.

I've duplicated many test cases for this, but its not complete. Enabling this runs all existing lit tests without crashing.

show more ...

[AArch64] Add intrinsic to count trailing zero elements

This patch introduces an experimental intrinsic for counting the
trailing zero elements in a vector. The intrinsic has generic expansion
in Se

[AArch64] Add intrinsic to count trailing zero elements

This patch introduces an experimental intrinsic for counting the
trailing zero elements in a vector. The intrinsic has generic expansion
in SelectionDAGBuilder, and for AArch64 there is a pattern which matches
to brkb & cntp instructions where SVE is enabled.

The intrinsic has a second operand, is_zero_poison, similar to the
existing cttz intrinsic.

These changes have been split out from D158291.

show more ...

[SDAG] Prefer forming sign_extend for zext nneg per target preference (#70725)

Builds on #67982 which recently introduced the nneg flag on a zext
instruction. Note that this change is the first poi

[SDAG] Prefer forming sign_extend for zext nneg per target preference (#70725)

Builds on #67982 which recently introduced the nneg flag on a zext
instruction. Note that this change is the first point where the flag is
being used for an optimization, and thus may expose latent miscompiles.
We've recently taught both CVP and InstCombine to infer the flag when
forming zext, but nothing else is using the flag just yet.

show more ...

Reland "[InstrProf][compiler-rt] Enable MC/DC Support in LLVM Source-based Code Coverage (1/3)"

Part 1 of 3. This includes the LLVM back-end processing and profile
reading/writing components. compil

Reland "[InstrProf][compiler-rt] Enable MC/DC Support in LLVM Source-based Code Coverage (1/3)"

Part 1 of 3. This includes the LLVM back-end processing and profile
reading/writing components. compiler-rt changes are included.

Differential Revision: https://reviews.llvm.org/D138846

show more ...

[IR] Require that ptrmask mask matches pointer index size (#69343)

Currently, we specify that the ptrmask intrinsic allows the mask to have
any size, which will be zero-extended or truncated to the

[IR] Require that ptrmask mask matches pointer index size (#69343)

Currently, we specify that the ptrmask intrinsic allows the mask to have
any size, which will be zero-extended or truncated to the pointer size.

However, what semantics of the specified GEP expansion actually imply is
that the mask is only meaningful up to the pointer type *index* size --
any higher bits of the pointer will always be preserved. In other words,
the mask gets 1-extended from the index size to the pointer size. This
is also the behavior we want for CHERI architectures.

This PR makes two changes:
* It spells out the interaction with the pointer type index size more
explicitly.
* It requires that the mask matches the pointer type index size. The
intention here is to make handling of this intrinsic more robust, to
avoid accidental mix-ups of pointer size and index size in code
generating this intrinsic. If a zero-extend or truncate of the mask is
desired, it should just be done explicitly in IR. This also cuts down on
the amount of testing we have to do, and things transforms needs to
check for.

As far as I can tell, we don't actually support pointers with different
index type size at the SDAG level, so I'm just asserting the sizes match
there for now. Out-of-tree targets using different index sizes may need
to adjust that code.

show more ...

[SVE ACLE] Allow default zero initialisation for svcount_t. (#69321)

This matches the behaviour of the other SVE ACLE types.

[StatepointLowering] Take return attributes of `gc.result` into account (#68439)

The current lowering of statepoints does not take into account return
attributes present on the `gc.result` leading

[StatepointLowering] Take return attributes of `gc.result` into account (#68439)

The current lowering of statepoints does not take into account return
attributes present on the `gc.result` leading to different code being
generated than if one were to not use statepoints. These return
attributes can affect the ABI which is why it is important that they are
applied in the lowering.

show more ...

[NVPTX] Fix code generation for `trap-unreachable`. (#67478)

https://reviews.llvm.org/D152789 added an `exit` op before each
`unreachable`. This means we never get to the `trap` instruction.

Thi

[NVPTX] Fix code generation for `trap-unreachable`. (#67478)

https://reviews.llvm.org/D152789 added an `exit` op before each
`unreachable`. This means we never get to the `trap` instruction.

This change limits the insertion of `exit` instructions to the cases
where `unreachable` is not lowered to `trap`. Trap itself is changed to
be emitted as `trap; exit;` to convey to `ptxas` that it exits the CFG.

show more ...

Revert "[InstrProf][compiler-rt] Enable MC/DC Support in LLVM Source-based Code Coverage (1/3)"

This seems to cause Clang to crash, see comments on the code review. Reverting
until the problem can b

Revert "[InstrProf][compiler-rt] Enable MC/DC Support in LLVM Source-based Code Coverage (1/3)"

This seems to cause Clang to crash, see comments on the code review. Reverting
until the problem can be investigated.

> Part 1 of 3. This includes the LLVM back-end processing and profile
> reading/writing components. compiler-rt changes are included.
>
> Differential Revision: https://reviews.llvm.org/D138846

This reverts commit a50486fd736ab2fe03fcacaf8b98876db77217a7.

show more ...

[InstrProf][compiler-rt] Enable MC/DC Support in LLVM Source-based Code Coverage (1/3)

Part 1 of 3. This includes the LLVM back-end processing and profile
reading/writing components. compiler-rt cha

[InstrProf][compiler-rt] Enable MC/DC Support in LLVM Source-based Code Coverage (1/3)

Part 1 of 3. This includes the LLVM back-end processing and profile
reading/writing components. compiler-rt changes are included.

Differential Revision: https://reviews.llvm.org/D138846

show more ...

[SelectionDAG][RISCV][PowerPC][X86] Use TargetConstant for immediates for ISD::PREFETCH. (#66601)

The intrinsic uses ImmArg so TargetConstant would be consistent
with how other intrinsics are handl

[SelectionDAG][RISCV][PowerPC][X86] Use TargetConstant for immediates for ISD::PREFETCH. (#66601)

The intrinsic uses ImmArg so TargetConstant would be consistent
with how other intrinsics are handled.

This hides the constants from type legalization so we can remove
the promotion support.

isel patterns are updated accordingly.

show more ...

[NFC][CodeGen] Change CodeGenOpt::Level/CodeGenFileType into enum classes (#66295)

This will make it easy for callers to see issues with and fix up calls
to createTargetMachine after a future chang

[NFC][CodeGen] Change CodeGenOpt::Level/CodeGenFileType into enum classes (#66295)

This will make it easy for callers to see issues with and fix up calls
to createTargetMachine after a future change to the params of
TargetMachine.

This matches other nearby enums.

For downstream users, this should be a fairly straightforward
replacement,
e.g. s/CodeGenOpt::Aggressive/CodeGenOptLevel::Aggressive
or s/CGFT_/CodeGenFileType::

show more ...

reland [InlineAsm] wrap ConstraintCode in enum class NFC (#66264)

reland [InlineAsm] wrap ConstraintCode in enum class NFC (#66003)

This reverts commit ee643b706be2b6bef9980b25cc9cc988dab94bb5.

Fi

reland [InlineAsm] wrap ConstraintCode in enum class NFC (#66264)

reland [InlineAsm] wrap ConstraintCode in enum class NFC (#66003)

This reverts commit ee643b706be2b6bef9980b25cc9cc988dab94bb5.

Fix up build failures in targets I missed in #66003

Kept as 3 commits for reviewers to see better what's changed. Will
squash when
merging.

- reland [InlineAsm] wrap ConstraintCode in enum class NFC (#66003)
- fix all the targets I missed in #66003
- fix off by one found by llvm/test/CodeGen/SystemZ/inline-asm-addr.ll

show more ...

Revert "[InlineAsm] wrap ConstraintCode in enum class NFC (#66003)"

This reverts commit 2ca4d136124d151216aac77a0403dcb5c5835bcd.

Also revert the followup, "[InlineAsm] fix botched merge conflict r

Revert "[InlineAsm] wrap ConstraintCode in enum class NFC (#66003)"

This reverts commit 2ca4d136124d151216aac77a0403dcb5c5835bcd.

Also revert the followup, "[InlineAsm] fix botched merge conflict resolution"

This reverts commit 8b9bf3a9f715ee5dce96eb1194441850c3663da1.

There were SystemZ and Mips build errors, too many to fix forward.

show more ...

[InlineAsm] wrap ConstraintCode in enum class NFC (#66003)

Similar to
commit 2fad6e69851e ("[InlineAsm] wrap Kind in enum class NFC")

Fix the TODOs added in
commit 93bd428742f9 ("[InlineAsm] refact

[InlineAsm] wrap ConstraintCode in enum class NFC (#66003)

Similar to
commit 2fad6e69851e ("[InlineAsm] wrap Kind in enum class NFC")

Fix the TODOs added in
commit 93bd428742f9 ("[InlineAsm] refactor InlineAsm class NFC
(#65649)")

show more ...

[InlineAsm] refactor InlineAsm class NFC (#65649)

I would like to steal one of these bits to denote whether a kind may be
spilled by the register allocator or not, but I'm afraid to touch of any
thi

[InlineAsm] refactor InlineAsm class NFC (#65649)

I would like to steal one of these bits to denote whether a kind may be
spilled by the register allocator or not, but I'm afraid to touch of any
this code using bitwise operands.

Make flags a first class type using bitfields, rather than launder data
around via `unsigned`.

show more ...

IR: Add llvm.exp10 intrinsic

We currently have log, log2, log10, exp and exp2 intrinsics. Add exp10
to fix this asymmetry. AMDGPU already has most of the code for f32
exp10 expansion implemented alo

IR: Add llvm.exp10 intrinsic

We currently have log, log2, log10, exp and exp2 intrinsics. Add exp10
to fix this asymmetry. AMDGPU already has most of the code for f32
exp10 expansion implemented alongside exp, so the current
implementation is duplicating nearly identical effort between the
compiler and library which is inconvenient.

https://reviews.llvm.org/D157871

show more ...

SelectionDAG: Swap operands of atomic_store

Irritatingly, atomic_store had operands in the opposite order from
regular store. This made it difficult to share patterns between
regular and atomic stor

SelectionDAG: Swap operands of atomic_store

Irritatingly, atomic_store had operands in the opposite order from
regular store. This made it difficult to share patterns between
regular and atomic stores.

There was a previous incomplete attempt to move atomic_store into the
regular StoreSDNode which would be better.

I think it was a mistake for all atomicrmw to swap the operand order,
so maybe it's better to take this one step further.

https://reviews.llvm.org/D123143

show more ...