[SelectionDAG] late init DeferredNodes (#123461)

No need to init `SmallVector` (even stack allocated) if we can exit
literally on next line.

Co-authored-by: klensy <nightouser@gmail.com>

[SelectionDAG] Virtualize isTargetStrictFPOpcode / isTargetMemoryOpcode (#119969)

With this change, targets are no longer required to put memory / strict-fp opcodes after special
`ISD::FIRST_TARGET_

[SelectionDAG] Virtualize isTargetStrictFPOpcode / isTargetMemoryOpcode (#119969)

With this change, targets are no longer required to put memory / strict-fp opcodes after special
`ISD::FIRST_TARGET_MEMORY_OPCODE`/`ISD::FIRST_TARGET_STRICTFP_OPCODE` markers.
This will also allow autogenerating `isTargetMemoryOpcode`/`isTargetStrictFPOpcode (#119709).

Pull Request: https://github.com/llvm/llvm-project/pull/119969

show more ...

[SelectionDAG] Move SDNode::use_iterator::getOperandNo to SDUse. (#120536)

This allows us to write more range based for loops because we no
longer need the iterator. It also matches IR's Use class.

[SelectionDAG] Split SDNode::use_iterator into user_iterator and use_iterator. (#120531)

SDNode::use_iterator now returns an SDUse& when dereferenced.
SDNode::user_iterator returns SDNode*. SDNode:

[SelectionDAG] Split SDNode::use_iterator into user_iterator and use_iterator. (#120531)

SDNode::use_iterator now returns an SDUse& when dereferenced.
SDNode::user_iterator returns SDNode*. SDNode::use_begin/use_end/uses
work on use_iterator. SDNode::user_begin/user_end/users work on
user_iterator.

We can now write range based for loops using SDUse& and SDNode::uses().
I've converted many of these in this patch. I didn't update loops that
have additional variables updated in their for statement.

Some loops use SDNode::use_iterator::getOperandNo() which also prevents
using range based for loops. I plan to move this into SDUse in a follow
up patch.

show more ...

[SelectionDAG] Add SDNode::user_begin() and use it in some places (#120509)

Most of these are just places that want the first user and aren't
iterating over the whole list.

While there I changed

[SelectionDAG] Add SDNode::user_begin() and use it in some places (#120509)

Most of these are just places that want the first user and aren't
iterating over the whole list.

While there I changed some use_size() == 1 to hasOneUse() which
is more efficient.

This is part of an effort to rename use_iterator to user_iterator
and provide a use_iterator that dereferences to SDUse&. This patch
helps reduce the diff on later patches.

show more ...

[SelectionDAG] Rename SDNode::uses() to users(). (#120499)

This function is most often used in range based loops or algorithms
where the iterator is implicitly dereferenced. The dereference returns

[SelectionDAG] Rename SDNode::uses() to users(). (#120499)

This function is most often used in range based loops or algorithms
where the iterator is implicitly dereferenced. The dereference returns
an SDNode * of the user rather than SDUse * so users() is a better name.

I've long beeen annoyed that we can't write a range based loop over
SDUse when we need getOperandNo. I plan to rename use_iterator to
user_iterator and add a use_iterator that returns SDUse& on dereference.
This will make it more like IR.

show more ...

[SelectionDAG] Stop storing EVTs in a function scoped static std::set. (#118715)

EVTs potentially contain a Type * that points into memory owned by an
LLVMContext. Storing them in a function scoped

[SelectionDAG] Stop storing EVTs in a function scoped static std::set. (#118715)

EVTs potentially contain a Type * that points into memory owned by an
LLVMContext. Storing them in a function scoped static means they may
outlive the LLVMContext they point to.

This std::set is used to unique single element VT lists containing a
single extended EVT. Single element VT list with a simple EVT are
uniqued by a separate cache indexed by the MVT::SimpleValueType enum. VT
lists with more than one element are uniqued by a FoldingSet owned by
the SelectionDAG object.

This patch moves the single element cache into SelectionDAG so that it
will be destroyed when SelectionDAG is destroyed.

Fixes #88233

show more ...

[SelectionDAG] Simplify classof of MemSDNode and MemIntrinsicSDNode (NFC) (#115720)

`SDNodeBits.IsMemIntrinsic` is set if and only if the node is an
instance of `MemIntrinsicSDNode`. Thus, to check

[SelectionDAG] Simplify classof of MemSDNode and MemIntrinsicSDNode (NFC) (#115720)

`SDNodeBits.IsMemIntrinsic` is set if and only if the node is an
instance of `MemIntrinsicSDNode`. Thus, to check if a node is an
instance of `MemIntrinsicSDNode` we only need to check this bit.

show more ...

Reapply "[SelectionDAG] Add preliminary plumbing for `samesign` flag"

Original commit: 19c8475871faee5ebb06281034872a85a2552675

Multiple 2-stage sanitizer buildbots were reporting failures, the iss

Reapply "[SelectionDAG] Add preliminary plumbing for `samesign` flag"

Original commit: 19c8475871faee5ebb06281034872a85a2552675

Multiple 2-stage sanitizer buildbots were reporting failures, the issue
has been addressed separately in 29246a92aee87e86cbc2bb64ee520d7385644f34.

show more ...

Revert "[SelectionDAG] Add preliminary plumbing for `samesign` flag" (#114647)

Crashes on ARM builds
https://lab.llvm.org/buildbot/#/builders/85/builds/2548
```
DAGCombiner.cpp:16157: SDValue (an

Revert "[SelectionDAG] Add preliminary plumbing for `samesign` flag" (#114647)

Crashes on ARM builds
https://lab.llvm.org/buildbot/#/builders/85/builds/2548
```
DAGCombiner.cpp:16157: SDValue (anonymous
namespace)::DAGCombiner::visitFREEZE(SDNode *):
Assertion `DAG.isGuaranteedNotToBeUndefOrPoison(R,
false) && "Can't create node that may be
undef/poison!"' failed.
```

Issue #114648

This reverts commit 19c8475871faee5ebb06281034872a85a2552675.

show more ...

[SelectionDAG] Add preliminary plumbing for `samesign` flag

Extend recently-added poison-generating IR flag to codegen as well.

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
htt

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
https://github.com/llvm/llvm-project/pull/113808#discussion_r1819923625.

show more ...

[SDAG][NFC] Convert `SDNodeFlags` into an enumeration (#114167)

This patch converts `SDNodeFlags` into an enumeration as we did for
`FastMathFlags`. It simplifies the implementation and improves
c

[SDAG][NFC] Convert `SDNodeFlags` into an enumeration (#114167)

This patch converts `SDNodeFlags` into an enumeration as we did for
`FastMathFlags`. It simplifies the implementation and improves
compile-time. This patch is NFC since it doesn't break SDNodeFlags API.

show more ...

[AArch64] Improve index selection for histograms (#111150)

Removes unnecessary extends on the indices passed into histogram instructions. It also removes the instruction when the mask is zero.

[LLVM][SelectionDAG] Ensure Constant[FP]SDnode only store references to scalar Constant{Int,FP}. (#111005)

This fixes a failure path when the use-constant-##-for-###-splat IR
options are enabled.

Add usub_cond and usub_sat operations to atomicrmw (#105568)

These both perform conditional subtraction, returning the minuend and
zero respectively, if the difference is negative.

[DAG][PatternMatch] Add support for matchers with flags; NFC

Add support for matching with `SDNodeFlags` i.e `add` with `nuw`.

This patch adds helpers for `or disjoint` or `zext nneg` with the same

[DAG][PatternMatch] Add support for matchers with flags; NFC

Add support for matching with `SDNodeFlags` i.e `add` with `nuw`.

This patch adds helpers for `or disjoint` or `zext nneg` with the same
names as we have in IR/PatternMatch api.

Closes #103060

show more ...

[DAG] Move isNullConstantOrUndef helper to SelectionDAGNodes.h to allow other future uses. NFC.

[CodeGen][SDAG] Remove CombinedNodes SmallPtrSet (#94609)

This "small" set grows quite large and it's more performant to store
whether a node has been combined before in the node itself.

As this

[CodeGen][SDAG] Remove CombinedNodes SmallPtrSet (#94609)

This "small" set grows quite large and it's more performant to store
whether a node has been combined before in the node itself.

As this information is only relevant for nodes that are currently not in
the worklist, add a second state to the CombinerWorklistIndex (-2) to
indicate that a node is currently not in a worklist, but was combined
before.

This brings a substantial performance improvement.

show more ...

[CodeGen][SDAG] Remove Combiner WorklistMap (#92900)

DenseMap for pointer lookup is expensive, and this is only used for
deduplication and index lookup. Instead, store the worklist index in the
no

[CodeGen][SDAG] Remove Combiner WorklistMap (#92900)

DenseMap for pointer lookup is expensive, and this is only used for
deduplication and index lookup. Instead, store the worklist index in the
node itself.

This brings a substantial performance improvement.

show more ...

[AArch64] Add an all-in-one histogram intrinsic

Based on discussion from
https://discourse.llvm.org/t/rfc-vectorization-support-for-histogram-count-operations/74788

Current interface is:

llvm

[AArch64] Add an all-in-one histogram intrinsic

Based on discussion from
https://discourse.llvm.org/t/rfc-vectorization-support-for-histogram-count-operations/74788

Current interface is:

llvm.experimental.histogram(<vecty> ptrs, <intty> inc_amount, <vecty> mask)

The integer type used by 'inc_amount' needs to match the type of the buckets in memory.

The intrinsic covers the following operations:
* Gather load
* histogram on the elements of 'ptrs'
* multiply the histogram results by 'inc_amount'
* add the result of the multiply to the values loaded by the gather
* scatter store the results of the add

Supports lowering to histcnt instructions for AArch64 targets, and scalarization for all others at present.

show more ...

[DAG] Early exit for flags in canCreateUndefOrPoison [nfc] (#89834)

This matches the style used in the Analysis version of this routine, and
makes it less likely we'll miss a poison generating flag

[DAG] Early exit for flags in canCreateUndefOrPoison [nfc] (#89834)

This matches the style used in the Analysis version of this routine, and
makes it less likely we'll miss a poison generating flag in future
changes. Unlike IR, the check for poison generating flags doesn't need
to switch over opcode since all nodes have the SDFlags storage.

show more ...

[SelectionDAG] Move GlobalAddressSDNode and AddrSpaceCastSDNode constructors into header. NFC

These constructors are no more complicated than any of the other
*SDNode constructors that are already i

[SelectionDAG] Move GlobalAddressSDNode and AddrSpaceCastSDNode constructors into header. NFC

These constructors are no more complicated than any of the other
*SDNode constructors that are already in the header.

show more ...

[SelectionDAG] Pass SDVTList instead of VTs to *SDNode constructors. NFC (#89880)

All of these constructors were creating a SDVTList using an EVT* created
by SDNode::getValueTypeList. This EVT need

[SelectionDAG] Pass SDVTList instead of VTs to *SDNode constructors. NFC (#89880)

All of these constructors were creating a SDVTList using an EVT* created
by SDNode::getValueTypeList. This EVT needs to live at least as long as
the SDNode that uses it. To do this, SDNode::getValueTypeList contains
several function scoped static variables that hold the memory for the
EVT. So the EVT lives until global destructors run.

This is problematic since an EVT contains a Type* that points to memory
allocated by an LLVMContext. If multiple LLVMContexts are used that
don't have overlapping lifetimes, we can end up with stale or or
incorrect pointers cached in the EVTs owned by SDNode::getValueTypeList.

I want to try to make the EVTs be owned by SelectionDAG instead. This is
already done for SDVTLists with more than 1 VT. The single value case is
a very old optimizaton that should be re-evaluated. In order to do this,
I need the SDVTLists to be created by SelectionDAG rather than by the
SDNode itself.

This patch doesn't change how the allocation is done yet. It just moves
the code around.

This patch does reduce the number of calls to getVTList since we now
share with the call needed for the SDNode FoldingSet.

Part of fixing #88233.

show more ...

[SystemZ] Don't lower ATOMIC_LOAD/STORE to LOAD/STORE (#75879)

- Instead of lowering float/double ISD::ATOMIC_LOAD / ISD::ATOMIC_STORE
nodes to regular LOAD/STORE nodes, make them legal and select

[SystemZ] Don't lower ATOMIC_LOAD/STORE to LOAD/STORE (#75879)

- Instead of lowering float/double ISD::ATOMIC_LOAD / ISD::ATOMIC_STORE
nodes to regular LOAD/STORE nodes, make them legal and select those nodes
properly instead. This avoids exposing them to the DAGCombiner.

- AtomicExpand pass no longer casts float/double atomic load/stores to integer
(FP128 is still casted).

show more ...