[DAGISel] Fix MMRA Handling in copyExtraInfo (#124730)

#78569 did not implement this correctly and an edge case breaks it by
triggering `Assertion `!Leafs.empty()' failed.`

Fixes SWDEV-507698

[SDAG] Use BatchAAResults for querying alias analysis (AA) results (#123934)

Once we get to SelectionDAG the IR should not be changing anymore, so we
can use BatchAAResults rather than AAResults to

[SDAG] Use BatchAAResults for querying alias analysis (AA) results (#123934)

Once we get to SelectionDAG the IR should not be changing anymore, so we
can use BatchAAResults rather than AAResults to cache AA queries.

This should be a NFC change for targets that enable AA during codegen
(such as AArch64), but also give a nice compile-time improvement in some
cases. See:
https://github.com/llvm/llvm-project/pull/123787#issuecomment-2606797041

Note: This follows Nikita's suggestion on #123787.

show more ...

[SDAG] Fix CSE for ADDRSPACECAST nodes (#122912)

Correct CSE in SelectionDAG can make DAG combining more effective and
reduces the size of the DAG and thus should improve compile time.

[DAGCombiner] Turn `(neg (max x, (neg x)))` into `(min x, (neg x))` (#120666)

This pattern was originally spotted in 429.mcf by @topperc.

We already have a DAGCombiner pattern to turn `(neg (abs

[DAGCombiner] Turn `(neg (max x, (neg x)))` into `(min x, (neg x))` (#120666)

This pattern was originally spotted in 429.mcf by @topperc.

We already have a DAGCombiner pattern to turn `(neg (abs x))` into `(min
x, (neg x))`. But in some cases `(neg (max x, (neg x)))` is formed by an
expanded `abs` followed by a `neg` that is generated only after the
`abs` expansion. This patch adds a separate pattern to match cases like
this, as well as its inverse pattern: `(neg (min X, (neg X))) --> (max
X, (neg X))`.

This pattern is applicable to both signed and unsigned min/max.

show more ...

[SelectionDAG] Virtualize isTargetStrictFPOpcode / isTargetMemoryOpcode (#119969)

With this change, targets are no longer required to put memory / strict-fp opcodes after special
`ISD::FIRST_TARGET_

[SelectionDAG] Virtualize isTargetStrictFPOpcode / isTargetMemoryOpcode (#119969)

With this change, targets are no longer required to put memory / strict-fp opcodes after special
`ISD::FIRST_TARGET_MEMORY_OPCODE`/`ISD::FIRST_TARGET_STRICTFP_OPCODE` markers.
This will also allow autogenerating `isTargetMemoryOpcode`/`isTargetStrictFPOpcode (#119709).

Pull Request: https://github.com/llvm/llvm-project/pull/119969

show more ...

[SelectionDAG] Use SmallVectorImpl& to avoid repeating SmallVector size. NFC

[SelectionDAG] Split SDNode::use_iterator into user_iterator and use_iterator. (#120531)

SDNode::use_iterator now returns an SDUse& when dereferenced.
SDNode::user_iterator returns SDNode*. SDNode:

[SelectionDAG] Split SDNode::use_iterator into user_iterator and use_iterator. (#120531)

SDNode::use_iterator now returns an SDUse& when dereferenced.
SDNode::user_iterator returns SDNode*. SDNode::use_begin/use_end/uses
work on use_iterator. SDNode::user_begin/user_end/users work on
user_iterator.

We can now write range based for loops using SDUse& and SDNode::uses().
I've converted many of these in this patch. I didn't update loops that
have additional variables updated in their for statement.

Some loops use SDNode::use_iterator::getOperandNo() which also prevents
using range based for loops. I plan to move this into SDUse in a follow
up patch.

show more ...

[SelectionDAG] Rename SDNode::uses() to users(). (#120499)

This function is most often used in range based loops or algorithms
where the iterator is implicitly dereferenced. The dereference returns

[SelectionDAG] Rename SDNode::uses() to users(). (#120499)

This function is most often used in range based loops or algorithms
where the iterator is implicitly dereferenced. The dereference returns
an SDNode * of the user rather than SDUse * so users() is a better name.

I've long beeen annoyed that we can't write a range based loop over
SDUse when we need getOperandNo. I plan to rename use_iterator to
user_iterator and add a use_iterator that returns SDUse& on dereference.
This will make it more like IR.

show more ...

[SDAG] Allow folding stack slots into sincos/frexp in more cases (#118117)

This adds a new helper `canFoldStoreIntoLibCallOutputPointers()` to
check that it is safe to fold a store into a node that

[SDAG] Allow folding stack slots into sincos/frexp in more cases (#118117)

This adds a new helper `canFoldStoreIntoLibCallOutputPointers()` to
check that it is safe to fold a store into a node that will expand to a
library call that takes output pointers. This requires checking for two
(independent) properties:

1. The store is not within a CALLSEQ_START..CALLSEQ_END pair
* If it is, the expansion would lead to nested call sequences (which is
invalid)
2. The node does not appear as a predecessor to the store
* If it does, attempting to merge the store into the call would result
in a cycle in the DAG

These two properties are checked as part of the same traversal in
`canFoldStoreIntoLibCallOutputPointers()`

show more ...

[CGData][GlobalIsel][Legalizer][DAG][MC][AsmParser][X86][AMX] Use `std::move` to avoid copy (#118068)

[SelectionDAG] Stop storing EVTs in a function scoped static std::set. (#118715)

EVTs potentially contain a Type * that points into memory owned by an
LLVMContext. Storing them in a function scoped

[SelectionDAG] Stop storing EVTs in a function scoped static std::set. (#118715)

EVTs potentially contain a Type * that points into memory owned by an
LLVMContext. Storing them in a function scoped static means they may
outlive the LLVMContext they point to.

This std::set is used to unique single element VT lists containing a
single extended EVT. Single element VT list with a simple EVT are
uniqued by a separate cache indexed by the MVT::SimpleValueType enum. VT
lists with more than one element are uniqued by a FoldingSet owned by
the SelectionDAG object.

This patch moves the single element cache into SelectionDAG so that it
will be destroyed when SelectionDAG is destroyed.

Fixes #88233

show more ...

[SDAG] Don't allow implicit trunc in getConstant() (#117558)

Assert that the passed value is a valid unsigned integer value for the
specified type.

For signed values getSignedConstant() / getSig

[SDAG] Don't allow implicit trunc in getConstant() (#117558)

Assert that the passed value is a valid unsigned integer value for the
specified type.

For signed values getSignedConstant() / getSignedTargetConstant() should
be used instead.

show more ...

[SelectionDAG] Require last operand of (STRICT_)FP_ROUND to be a TargetConstant. (#117639)

Fix all the places I could find that did't do this. We were already
mostly correct for FP_ROUND after
9a9

[SelectionDAG] Require last operand of (STRICT_)FP_ROUND to be a TargetConstant. (#117639)

Fix all the places I could find that did't do this. We were already
mostly correct for FP_ROUND after
9a976f36615dbe15e76c12b22f711b2e597a8e51, but not STRICT_FP_ROUND.

show more ...

[SDAG] Limit sincos/frexp stack slot folding to stores chained to entry (#115906)

When the chain is not the entry node there is a risk the stores are
within a (CALLSEQ_START, CALLSEQ_END), which wh

[SDAG] Limit sincos/frexp stack slot folding to stores chained to entry (#115906)

When the chain is not the entry node there is a risk the stores are
within a (CALLSEQ_START, CALLSEQ_END), which when the node is expanded
will lead to nested call sequences.

It should be possible to check for this and allow more cases, but for
now, let's limit this to cases where it's definitely safe.

Fixes #115323

show more ...

[SelectionDAG] Add support for extending masked loads in computeKnownBits (#115450)

We already support computing known bits for extending loads, but not for
masked loads. For now I've only added su

[SelectionDAG] Add support for extending masked loads in computeKnownBits (#115450)

We already support computing known bits for extending loads, but not for
masked loads. For now I've only added support for zero-extends because
that's the only thing currently tested. Even when the passthru value is
poison we still know the top X bits are zero.

show more ...

[SDAG] Merge multiple-result libcall expansion into DAG.expandMultipleResultFPLibCall() (#114792)

This merges the logic for expanding both FFREXP and FSINCOS into one
method `DAG.expandMultipleResu

[SDAG] Merge multiple-result libcall expansion into DAG.expandMultipleResultFPLibCall() (#114792)

This merges the logic for expanding both FFREXP and FSINCOS into one
method `DAG.expandMultipleResultFPLibCall()`. This reduces duplication
and also allows FFREXP to benefit from the stack slot elimination
implemented for FSINCOS. This method will also be used in future to
implement more multiple-result intrinsics (such as modf and sincospi).

show more ...

[SelectionDAG] Remove unneeded assert from SelectionDAG::getSignedConstant. NFC (#114336)

This assert is also present inside the APInt constructor after #114539.

[SDAG] Intersect poison-generating flags after CSE (#114650)

This patch intersects poison-generating flags after CSE to fix assertion
failure reported in
https://github.com/llvm/llvm-project/pull/

[SDAG] Intersect poison-generating flags after CSE (#114650)

This patch intersects poison-generating flags after CSE to fix assertion
failure reported in
https://github.com/llvm/llvm-project/pull/112354#issuecomment-2452369552.

Co-authored-by: Antonio Frighetto <me@antoniofrighetto.com>

show more ...

[SDAG] Support expanding `FSINCOS` to vector library calls (#114039)

This shares most of its code with the scalar sincos expansion. It allows
expanding vector FSINCOS nodes to a library call from t

[SDAG] Support expanding `FSINCOS` to vector library calls (#114039)

This shares most of its code with the scalar sincos expansion. It allows
expanding vector FSINCOS nodes to a library call from the specified
`-vector-library`. The upside of this is it will mean the vectorizer
only needs to handle the sincos intrinsic, which has no memory effects,
and this can handle lowering the intrinsic to a call that takes output
pointers.

show more ...

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
htt

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
https://github.com/llvm/llvm-project/pull/113808#discussion_r1819923625.

show more ...

DAG: Fix legalization of vector addrspacecasts (#113964)

[IR] Add `llvm.sincos` intrinsic (#109825)

This adds the `llvm.sincos` intrinsic, legalization, and lowering.

The `llvm.sincos` intrinsic takes a floating-point value and returns
both the sine a

[IR] Add `llvm.sincos` intrinsic (#109825)

This adds the `llvm.sincos` intrinsic, legalization, and lowering.

The `llvm.sincos` intrinsic takes a floating-point value and returns
both the sine and cosine (as a struct).

```
declare { float, float } @llvm.sincos.f32(float %Val)
declare { double, double } @llvm.sincos.f64(double %Val)
declare { x86_fp80, x86_fp80 } @llvm.sincos.f80(x86_fp80 %Val)
declare { fp128, fp128 } @llvm.sincos.f128(fp128 %Val)
declare { ppc_fp128, ppc_fp128 } @llvm.sincos.ppcf128(ppc_fp128 %Val)
declare { <4 x float>, <4 x float> } @llvm.sincos.v4f32(<4 x float> %Val)
```

The lowering is built on top of the existing FSINCOS ISD node, with
additional type legalization to allow for f16, f128, and vector values.

show more ...

Check hasOptSize() in shouldOptimizeForSize() (#112626)

Ensure !NDEBUG with LLVM_ENABLE_ABI_BREAKING_CHECKS does not segfault (#113588)

In SelectionDAG, `TargetTransformInfo::hasBranchDivergence()` can be
called when both `NDEBUG` and `LLVM_ENABLE_ABI_B

Ensure !NDEBUG with LLVM_ENABLE_ABI_BREAKING_CHECKS does not segfault (#113588)

In SelectionDAG, `TargetTransformInfo::hasBranchDivergence()` can be
called when both `NDEBUG` and `LLVM_ENABLE_ABI_BREAKING_CHECKS` are
enabled. In that case, the class member `TTI` is still initialized to
`nullptr`, causing a segfault.

Fix this by ensuring that all the calls to `hasBranchDivergence` and
`VerifyDAGDivergence` only occur when `NDEBUG` is disabled, and
`LLVM_ENABLE_ABI_BREAKING_CHECKS` is enabled.

show more ...

[DAG] isConstantIntBuildVectorOrConstantInt - peek through bitcasts (#112710) (REAPPLIED)

Alter both isConstantIntBuildVectorOrConstantInt + isConstantFPBuildVectorOrConstantFP to return a bool inst

[DAG] isConstantIntBuildVectorOrConstantInt - peek through bitcasts (#112710) (REAPPLIED)

Alter both isConstantIntBuildVectorOrConstantInt + isConstantFPBuildVectorOrConstantFP to return a bool instead of the underlying SDNode, and adjust usage to account for this.

Update isConstantIntBuildVectorOrConstantInt to peek though bitcasts when attempting to find a constant, in particular this improves canonicalization of constants to the RHS on commutable instructions.

X86 is the beneficiary here as it often bitcasts rematerializable 0/-1 vector constants as vXi32 and bitcasts to the requested type

Minor cleanup that helps with #107423

Reapplied after regression fix ba1255def64a9c3c68d97ace051eec76f546eeb0

show more ...