[X86][avx512] Fix bug in masked compress store.

Differential Revision: https://reviews.llvm.org/D23984

llvm-svn: 282381

[CodeGen] Split out the notions of MI invariance and MI dereferenceability.

Summary:
An IR load can be invariant, dereferenceable, neither, or both. But
currently, MI's notion of invariance is IR-i

[CodeGen] Split out the notions of MI invariance and MI dereferenceability.

Summary:
An IR load can be invariant, dereferenceable, neither, or both. But
currently, MI's notion of invariance is IR-invariant &&
IR-dereferenceable.

This patch splits up the notions of invariance and dereferenceability at
the MI level. It's NFC, so adds some probably-unnecessary
"is-dereferenceable" checks, which we can remove later if desired.

Reviewers: chandlerc, tstellarAMD

Subscribers: jholewinski, arsenm, nemanjai, llvm-commits

Differential Revision: https://reviews.llvm.org/D23371

llvm-svn: 281151

show more ...

Remove dead code in the SelectionDAG headers (NFC)

I tested this with "ninja check-llvm-codegen" on a Release build with
all architectures enabled, and again with a Debug build on x86.

Found with l

Remove dead code in the SelectionDAG headers (NFC)

I tested this with "ninja check-llvm-codegen" on a Release build with
all architectures enabled, and again with a Debug build on x86.

Found with llvm-cov.

Differential Revision: https://reviews.llvm.org/D24433

llvm-svn: 281120

show more ...

Attempt to fix static_asserts in SelectionDAGNodes.h that MSVC chokes on.

These were added in r281051, which, I am embarrassed to admit, has an
incomplete commit message that I forgot to update befo

Attempt to fix static_asserts in SelectionDAGNodes.h that MSVC chokes on.

These were added in r281051, which, I am embarrassed to admit, has an
incomplete commit message that I forgot to update before pushing. You
can ignore element (2) in that list.

llvm-svn: 281054

show more ...

[SelectionDAG] Fix two issues with SDNode::getRawSubclassData().

1) On some platforms, sizeof(SDNodeBits) == 1, so we were only copying
one byte out of the bitfield when we wanted to copy two, and w

[SelectionDAG] Fix two issues with SDNode::getRawSubclassData().

1) On some platforms, sizeof(SDNodeBits) == 1, so we were only copying
one byte out of the bitfield when we wanted to copy two, and we were
leaving half of the return value of getRawSubclassData() undefined.

2) Something something bitfields, not sure exactly what the issue or fix
is, yet. (TODO)

Summary:
Previously we were assuming that SDNodeBits covered all of SDNode's
anonymous subclass data bitfield union. But that's not right; it might
have size 1, in which it clearly doesn't.

This patch adds a field that does cover the whole union and adds
static_asserts to ensure it stays correct.

Reviewers: ahatanak, chandlerc

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D24223

llvm-svn: 281051

show more ...

s/static inline/static/ for headers I have changed in r279475. NFC.

llvm-svn: 280257

ADT: Split ilist_node_traits into alloc and callback, NFC

Many lists want to override only allocation semantics, or callbacks for
iplist. Split these up to prevent code duplication.
- Specialize il

ADT: Split ilist_node_traits into alloc and callback, NFC

Many lists want to override only allocation semantics, or callbacks for
iplist. Split these up to prevent code duplication.
- Specialize ilist_alloc_traits to change the implementations of
deleteNode() and createNode().
- One common desire is to do nothing deleteNode() and disable
createNode(). Specialize ilist_alloc_traits to inherit from
ilist_noalloc_traits for that behaviour.
- Specialize ilist_callback_traits to use the addNodeToList(),
removeNodeFromList(), and transferNodesFromList() callbacks.

As a drive-by, add some coverage to the callback-related unit tests.

llvm-svn: 280128

show more ...

Fix the static_assert added in r279536.

The assertion doesn't always hold true as sizeof(SDNodeBits) isn't equal
to sizeof(uint16_t) for some targets. For example, sizeof(SDNodeBits)
evaluates to 1,

Fix the static_assert added in r279536.

The assertion doesn't always hold true as sizeof(SDNodeBits) isn't equal
to sizeof(uint16_t) for some targets. For example, sizeof(SDNodeBits)
evaluates to 1, not 2, for ARM's APCS targets.

llvm-svn: 279797

show more ...

[SelectionDAG] Use a union of bitfield structs for SDNode::SubclassData.

Summary:
This greatly simplifies our handling of SDNode::SubclassData.

NFC, hopefully. :)

See discussion in D23035 for dis

[SelectionDAG] Use a union of bitfield structs for SDNode::SubclassData.

Summary:
This greatly simplifies our handling of SDNode::SubclassData.

NFC, hopefully. :)

See discussion in D23035 for discussion about the design API of these
bitfields.

Reviewers: chandlerc

Subscribers: llvm-commits, rnk

Differential Revision: https://reviews.llvm.org/D23036

llvm-svn: 279537

show more ...

[GraphTraits] Replace all NodeType usage with NodeRef

This should finish the GraphTraits migration.

Differential Revision: http://reviews.llvm.org/D23730

llvm-svn: 279475

[GraphWriter] Change GraphWriter to use NodeRef in GraphTraits

Summary:
This is part of the "NodeType* -> NodeRef" migration. Notice that since
GraphWriter prints object address as identity, I added

[GraphWriter] Change GraphWriter to use NodeRef in GraphTraits

Summary:
This is part of the "NodeType* -> NodeRef" migration. Notice that since
GraphWriter prints object address as identity, I added a static_assert on
NodeRef to be a pointer type.

Reviewers: dblaikie

Subscribers: llvm-commits, MatzeB

Differential Revision: https://reviews.llvm.org/D23580

llvm-svn: 278966

show more ...

[DAG] make isConstantSplatVector() available to the rest of lowering

llvm-svn: 275025

fix documentation comments; NFC

llvm-svn: 275021

[CodeGen] Make ShuffleVectorSDNode::commuteMask take a MutableArrayRef instead of SmallVectorImpl. NFC.

llvm-svn: 274095

Pass DebugLoc and SDLoc by const ref.

This used to be free, copying and moving DebugLocs became expensive
after the metadata rewrite. Passing by reference eliminates a ton of
track/untrack operation

Pass DebugLoc and SDLoc by const ref.

This used to be free, copying and moving DebugLocs became expensive
after the metadata rewrite. Passing by reference eliminates a ton of
track/untrack operations. No functionality change intended.

llvm-svn: 272512

show more ...

[x86] use BMI 'andn' for logic + compare ops

With BMI, we can use 'andn' to save an instruction when the result is only used in a compare.
This is related to one of the potential sequences to check

[x86] use BMI 'andn' for logic + compare ops

With BMI, we can use 'andn' to save an instruction when the result is only used in a compare.
This is related to one of the potential sequences to check 'isfinite' in:
https://llvm.org/bugs/show_bug.cgi?id=27164

Differential Revision: http://reviews.llvm.org/D18910

llvm-svn: 265875

show more ...

NFC: make AtomicOrdering an enum class

Summary:
In the context of http://wg21.link/lwg2445 C++ uses the concept of
'stronger' ordering but doesn't define it properly. This should be fixed
in C++17 b

NFC: make AtomicOrdering an enum class

Summary:
In the context of http://wg21.link/lwg2445 C++ uses the concept of
'stronger' ordering but doesn't define it properly. This should be fixed
in C++17 barring a small question that's still open.

The code currently plays fast and loose with the AtomicOrdering
enum. Using an enum class is one step towards tightening things. I later
also want to tighten related enums, such as clang's
AtomicOrderingKind (which should be shared with LLVM as a 'C++ ABI'
enum).

This change touches a few lines of code which can be improved later, I'd
like to keep it as NFC for now as it's already quite complex. I have
related changes for clang.

As a follow-up I'll add:
bool operator<(AtomicOrdering, AtomicOrdering) = delete;
bool operator>(AtomicOrdering, AtomicOrdering) = delete;
bool operator<=(AtomicOrdering, AtomicOrdering) = delete;
bool operator>=(AtomicOrdering, AtomicOrdering) = delete;
This is separate so that clang and LLVM changes don't need to be in sync.

Reviewers: jyknight, reames

Subscribers: jyknight, llvm-commits

Differential Revision: http://reviews.llvm.org/D18775

llvm-svn: 265602

show more ...

SelectionDAG: Remove arbitrary and bug-prone complexity from SDLoc

The implementation of SDLoc has an extra layer of indirection here for
no particular reason, and was leading to problems where we w

SelectionDAG: Remove arbitrary and bug-prone complexity from SDLoc

The implementation of SDLoc has an extra layer of indirection here for
no particular reason, and was leading to problems where we were
dereferencing pointers to SDNodes that had already been deleted so
that we could get at the DebugLoc for a new SDNode. This is one of the
errors that came up often in PR26808.

Instead, we can just track the DebugLoc and IROrder directly. This
makes the code both easier to understand and more correct. It's also
basically NFC other than fixing a large number of places where we were
reading the memory of deleted SDNodes.

llvm-svn: 264470

show more ...

Prevent construction of cycle in DAG store merge

When merging stores in DAGCombiner, add check to ensure that no
dependenices exist that would cause the construction of a cycle in our
DAG. This may

Prevent construction of cycle in DAG store merge

When merging stores in DAGCombiner, add check to ensure that no
dependenices exist that would cause the construction of a cycle in our
DAG. This may happen if one store has a data dependence on another
instruction (e.g. a load) which itself has a (chain) dependence on
another store being merged. These stores cannot be merged safely and
doing so results in a cycle that is discovered in LegalizeDAG.

This test is only done in cases where Antialias analysis is used (UseAA)
as non-AA store merge candidates will be merged logically after all
loads which have been checked to not alias.

Reviewers: ahatanak, spatel, niravd, arsenm, hfinkel, tstellarAMD, jyknight

Subscribers: llvm-commits, tberghammer, danalbert, srhines

Differential Revision: http://reviews.llvm.org/D18336

llvm-svn: 264461

show more ...

SelectionDAG: Appease the bots that don't like my union

Should fix the breakage in r262902.

llvm-svn: 262908

Re-apply "SelectionDAG: Store SDNode operands in an ArrayRecycler"

This re-applies r262886 with a fix for 32 bit platforms that have 8 byte
pointer alignment, effectively reverting r262892.

Origina

Re-apply "SelectionDAG: Store SDNode operands in an ArrayRecycler"

This re-applies r262886 with a fix for 32 bit platforms that have 8 byte
pointer alignment, effectively reverting r262892.

Original Message:

Currently some SDNode operands are malloc'd, some are stored inline in
subclasses of SDNode, and some are thrown into a BumpPtrAllocator.
This scheme is complex, inconsistent, and makes refactoring SDNodes
fairly difficult.

Instead, we can allocate all of the operands using an ArrayRecycler
that wraps a BumpPtrAllocator. This keeps the cache locality when
iterating operands, improves locality when iterating SDNodes without
looking at operands, and vastly simplifies the ownership semantics.

It also means we stop overallocating SDNodes by 2-3x and will make it
simpler to fix the rampant undefined behaviour we have in how we
mutate SDNodes from one kind to another (See llvm.org/pr26808).

This is NFC other than the changes in memory behaviour, and I ran some
LNT tests to make sure this didn't hurt compile time. Not many tests
changed: there were a couple of 1-2% regressions reported, but there
were more improvements (of up to 4%) than regressions.

llvm-svn: 262902

show more ...

Revert "SelectionDAG: Store SDNode operands in an ArrayRecycler"

Looks like the largest SDNode is different between 32 and 64 bit now,
so this is breaking 32 bit bots. Reverting while I figure out a

Revert "SelectionDAG: Store SDNode operands in an ArrayRecycler"

Looks like the largest SDNode is different between 32 and 64 bit now,
so this is breaking 32 bit bots. Reverting while I figure out a fix.

This reverts r262886.

llvm-svn: 262892

show more ...

SelectionDAG: Store SDNode operands in an ArrayRecycler

Currently some SDNode operands are malloc'd, some are stored inline in
subclasses of SDNode, and some are thrown into a BumpPtrAllocator.
This

SelectionDAG: Store SDNode operands in an ArrayRecycler

Currently some SDNode operands are malloc'd, some are stored inline in
subclasses of SDNode, and some are thrown into a BumpPtrAllocator.
This scheme is complex, inconsistent, and makes refactoring SDNodes
fairly difficult.

Instead, we can allocate all of the operands using an ArrayRecycler
that wraps a BumpPtrAllocator. This keeps the cache locality when
iterating operands, improves locality when iterating SDNodes without
looking at operands, and vastly simplifies the ownership semantics.

It also means we stop overallocating SDNodes by 2-3x and will make it
simpler to fix the rampant undefined behaviour we have in how we
mutate SDNodes from one kind to another (See llvm.org/pr26808).

This is NFC other than the changes in memory behaviour, and I ran some
LNT tests to make sure this didn't hurt compile time. Not many tests
changed: there were a couple of 1-2% regressions reported, but there
were more improvements (of up to 4%) than regressions.

llvm-svn: 262886

show more ...

SelectionDAG: Remove some unused AtomicSDNode constructors. NFC

llvm-svn: 262849

Detecte vector reduction operations just before instruction selection.

(This is the second attemp to commit this patch, after fixing pr26652 & pr26653).

This patch detects vector reductions before

Detecte vector reduction operations just before instruction selection.

(This is the second attemp to commit this patch, after fixing pr26652 & pr26653).

This patch detects vector reductions before instruction selection. Vector
reductions are vectorized reduction operations, and for such operations we have
freedom to reorganize the elements of the result as long as the reduction of them
stay unchanged. This will enable some reduction pattern recognition during
instruction combine such as SAD/dot-product on X86. A flag is added to
SDNodeFlags to mark those vector reduction nodes to be checked during instruction
combine.

To detect those vector reductions, we search def-use chains starting from the
given instruction, and check if all uses fall into two categories:

1. Reduction with another vector.
2. Reduction on all elements.

in which 2 is detected by recognizing the pattern that the loop vectorizer
generates to reduce all elements in the vector outside of the loop, which
includes several ShuffleVector and one ExtractElement instructions.


Differential revision: http://reviews.llvm.org/D15250

llvm-svn: 261804

show more ...