AtomicExpand: Preserve metadata when bitcasting fp atomicrmw xchg (#115240)

[CodeGen] Remove unused includes (NFC) (#115996)

Identified with misc-include-cleaner.

AMDGPU: Custom expand flat cmpxchg which may access private (#109410)

64-bit flat cmpxchg instructions do not work correctly for scratch
addresses, and need to be expanded as non-atomic.

Allow cust

AMDGPU: Custom expand flat cmpxchg which may access private (#109410)

64-bit flat cmpxchg instructions do not work correctly for scratch
addresses, and need to be expanded as non-atomic.

Allow custom expansion of cmpxchg in AtomicExpand, as is
already the case for atomicrmw.

show more ...

AtomicExpand: Copy metadata from atomicrmw to cmpxchg (#109409)

When expanding an atomicrmw with a cmpxchg, preserve any metadata
attached to it. This will avoid unwanted double expansions
in a futu

AtomicExpand: Copy metadata from atomicrmw to cmpxchg (#109409)

When expanding an atomicrmw with a cmpxchg, preserve any metadata
attached to it. This will avoid unwanted double expansions
in a future commit.

The initial load should also probably receive the same metadata
(which for some reason is not emitted as an atomic).

show more ...

AtomicExpand: Really allow incremental legalization (#108613)

Fix up 100d9b89947bb1d42af20010bb594fa4c02542fc. The iterator
fixes ended up defeating the point, since newly inserted blocks
were not

AtomicExpand: Really allow incremental legalization (#108613)

Fix up 100d9b89947bb1d42af20010bb594fa4c02542fc. The iterator
fixes ended up defeating the point, since newly inserted blocks
were not visited. This never erases the current block, so we can
simply not preincrement the block iterator.

The AArch64 FP atomic tests now expand the cmpxchg in the second
round of legalization.

show more ...

Add usub_cond and usub_sat operations to atomicrmw (#105568)

These both perform conditional subtraction, returning the minuend and
zero respectively, if the difference is negative.

Reapply "AtomicExpand: Allow incrementally legalizing atomicrmw" (#107307)

This reverts commit 63da545ccdd41d9eb2392a8d0e848a65eb24f5fa.

Use reverse iteration in the instruction loop to avoid sanit

Reapply "AtomicExpand: Allow incrementally legalizing atomicrmw" (#107307)

This reverts commit 63da545ccdd41d9eb2392a8d0e848a65eb24f5fa.

Use reverse iteration in the instruction loop to avoid sanitizer
errors. This also has the side effect of avoiding the AArch64
codegen quality regressions.

Closes #107309

show more ...

Revert "Reland "AtomicExpand: Allow incrementally legalizing atomicrmw"" (#107307)

Reverts llvm/llvm-project#106793

`Next == E` is not enough:
https://lab.llvm.org/buildbot/#/builders/169/builds

Revert "Reland "AtomicExpand: Allow incrementally legalizing atomicrmw"" (#107307)

Reverts llvm/llvm-project#106793

`Next == E` is not enough:
https://lab.llvm.org/buildbot/#/builders/169/builds/2834

`Next` is deleted by `processAtomicInstr`

show more ...

Reland "Revert "AtomicExpand: Allow incrementally legalizing atomicrmw"" (#106793)

Reverts llvm/llvm-project#106792

The first commit of PR is pure revert, the rest is a possible fix.

Revert "AtomicExpand: Allow incrementally legalizing atomicrmw" (#106792)

Reverts llvm/llvm-project#103371

There is `heap-use-after-free`, commented on
206b5aff44a95754f6dd7a5696efa024e983ac59

Revert "AtomicExpand: Allow incrementally legalizing atomicrmw" (#106792)

Reverts llvm/llvm-project#103371

There is `heap-use-after-free`, commented on
206b5aff44a95754f6dd7a5696efa024e983ac59

Maybe `if (Next == E || BB != Next->getParent()) {` is enough,
but not sure, what was the intent there,

show more ...

AtomicExpand: Allow incrementally legalizing atomicrmw (#103371)

If a lowering changed control flow, resume the legalization
loop at the first newly inserted block.

This will allow incrementally le

AtomicExpand: Allow incrementally legalizing atomicrmw (#103371)

If a lowering changed control flow, resume the legalization
loop at the first newly inserted block.

This will allow incrementally legalizing atomicrmw and cmpxchg.

The AArch64 test might be a bugfix. Previously it would lower
the vector FP case as a cmpxchg loop, but cmpxchgs get lowered
but previously weren't. Maybe it shouldn't be reporting cmpxchg
for the expand type in the first place though.

show more ...

AtomicExpand: Add assert that atomicrmw is an xchg

It turns out it's trivial to hit this path with any rmw
operation.

AtomicExpand: Refactor atomic instruction handling (#102914)

Move the processing of an instruction into a helper function. Also
avoid redundant checking for all types of atomic instructions.
Includi

AtomicExpand: Refactor atomic instruction handling (#102914)

Move the processing of an instruction into a helper function. Also
avoid redundant checking for all types of atomic instructions.
Including the assert, it was effectively performing the same check
3 times.

show more ...

[llvm] Make InstSimplifyFolder constructor explicit (NFC) (#101654)

Reapply "[LLVM][LTO] Factor out RTLib calls and allow them to be dropped (#98512)"

This reverts commit 740161a9b98c9920dedf1852b5f1c94d0a683af5.

I moved the `ISD` dependencies into the CodeGen port

Reapply "[LLVM][LTO] Factor out RTLib calls and allow them to be dropped (#98512)"

This reverts commit 740161a9b98c9920dedf1852b5f1c94d0a683af5.

I moved the `ISD` dependencies into the CodeGen portion of the handling,
it's a little awkward but it's the easiest solution I can think of for
now.

show more ...

Revert "[LLVM][LTO] Factor out RTLib calls and allow them to be dropped (#98512)"

This reverts commit c05126bdfc3b02daa37d11056fa43db1a6cdef69.
(llvmorg-19-init-17714-gc05126bdfc3b)
See #99610

[LLVM][LTO] Factor out RTLib calls and allow them to be dropped (#98512)

Summary:
The LTO pass and LLD linker have logic in them that forces extraction
and prevent internalization of needed runtime

[LLVM][LTO] Factor out RTLib calls and allow them to be dropped (#98512)

Summary:
The LTO pass and LLD linker have logic in them that forces extraction
and prevent internalization of needed runtime calls. However, these
currently take all RTLibcalls into account, even if the target does not
support them. The target opts-out of a libcall if it sets its name to
nullptr. This patch pulls this logic out into a class in the header so
that LTO / lld can use it to determine if a symbol actually needs to be
kept.

This is important for targets like AMDGPU that want to be able to use
`lld` to perform the final link step, but does not want the overhead of
uncalled functions. (This adds like a second to the link time trivially)

show more ...

[IR] Add getDataLayout() helpers to Function and GlobalValue (#96919)

Similar to https://github.com/llvm/llvm-project/pull/96902, this adds
`getDataLayout()` helpers to Function and GlobalValue, re

[IR] Add getDataLayout() helpers to Function and GlobalValue (#96919)

Similar to https://github.com/llvm/llvm-project/pull/96902, this adds
`getDataLayout()` helpers to Function and GlobalValue, replacing the
current `getParent()->getDataLayout()` pattern.

show more ...

[IR] Add getDataLayout() helpers to BasicBlock and Instruction (#96902)

This is a helper to avoid writing `getModule()->getDataLayout()`. I
regularly try to use this method only to remember it does

[IR] Add getDataLayout() helpers to BasicBlock and Instruction (#96902)

This is a helper to avoid writing `getModule()->getDataLayout()`. I
regularly try to use this method only to remember it doesn't exist...

`getModule()->getDataLayout()` is also a common (the most common?)
reason why code has to include the Module.h header.

show more ...

Revert "[IR][NFC] Update IRBuilder to use InsertPosition (#96497)"

Reverts the above commit, as it updates a common header function and
did not update all callsites:

https://lab.llvm.org/buildbot

Revert "[IR][NFC] Update IRBuilder to use InsertPosition (#96497)"

Reverts the above commit, as it updates a common header function and
did not update all callsites:

https://lab.llvm.org/buildbot/#/builders/29/builds/382

This reverts commit 6481dc57612671ebe77fe9c34214fba94e1b3b27.

show more ...

[IR][NFC] Update IRBuilder to use InsertPosition (#96497)

Uses the new InsertPosition class (added in #94226) to simplify some of
the IRBuilder interface, and removes the need to pass a BasicBlock

[IR][NFC] Update IRBuilder to use InsertPosition (#96497)

Uses the new InsertPosition class (added in #94226) to simplify some of
the IRBuilder interface, and removes the need to pass a BasicBlock
alongside a BasicBlock::iterator, using the fact that we can now get the
parent basic block from the iterator even if it points to the sentinel.
This patch removes the BasicBlock argument from each constructor or call
to setInsertPoint.

This has no functional effect, but later on as we look to remove the
`Instruction *InsertBefore` argument from instruction-creation
(discussed
[here](https://discourse.llvm.org/t/psa-instruction-constructors-changing-to-iterator-only-insertion/77845)),
this will simplify the process by allowing us to deprecate the
InsertPosition constructor directly and catch all the cases where we use
instructions rather than iterators.

show more ...

AtomicExpand: Fix creating invalid ptrmask for fat pointers (#94955)

The ptrmask intrinsic requires the integer mask to be the index size,
not the pointer size.

AtomicExpand: Preserve metadata when expanding partword RMW (#89769)

This will be important for AMDGPU in a future patch.

AMDGPU: Do not bitcast atomicrmw in IR (#90045)

This is the first step to eliminating shouldCastAtomicRMWIInIR. This and
the other atomic expand casting hooks should be removed. This adds
duplicat

AMDGPU: Do not bitcast atomicrmw in IR (#90045)

This is the first step to eliminating shouldCastAtomicRMWIInIR. This and
the other atomic expand casting hooks should be removed. This adds
duplicate legalization machinery and interfaces. This is already what
codegen is supposed to do, and already does for the promotion case.

In the case of atomicrmw xchg, there seems to be some benefit to having
the bitcasts moved outside of the cmpxchg loop on targets with separate
int and FP registers, which we should be able to deal with by directly
checking for the legality of the underlying operation.

The casting path was also losing metadata when it recreated the
instruction.

show more ...

AtomicExpand: Fix dropping a syncscope when bitcasting atomicrmw