[IR] Allow scalable vectors in structs to support intrinsics returning multiple values.

RISC-V would like to use a struct of scalable vectors to return multiple
values from intrinsics. This woud als

[IR] Allow scalable vectors in structs to support intrinsics returning multiple values.

RISC-V would like to use a struct of scalable vectors to return multiple
values from intrinsics. This woud also be needed for target independent
intrinsics like llvm.sadd.overflow.

This patch removes the existing restriction for this. I've modified
StructType::isSized to consider a struct containing scalable vectors
as unsized so the verifier won't allow loads/stores/allocas of these
structs.

Reviewed By: sdesmalen

Differential Revision: https://reviews.llvm.org/D94142

show more ...

[CodeGen] Remove unused function hasInlineAsmMemConstraint (NFC)

The last use of the function was removed on Sep 13, 2010 in commit
1094c80281e3cdd9e9a9d7ee716da6386b33359b.

[CSSPGO] IR intrinsic for pseudo-probe block instrumentation

This change introduces a new IR intrinsic named `llvm.pseudoprobe` for pseudo-probe block instrumentation. Please refer to https://review

[CSSPGO] IR intrinsic for pseudo-probe block instrumentation

This change introduces a new IR intrinsic named `llvm.pseudoprobe` for pseudo-probe block instrumentation. Please refer to https://reviews.llvm.org/D86193 for the whole story.

A pseudo probe is used to collect the execution count of the block where the probe is instrumented. This requires a pseudo probe to be persisting. The LLVM PGO instrumentation also instruments in similar places by placing a counter in the form of atomic read/write operations or runtime helper calls. While these operations are very persisting or optimization-resilient, in theory we can borrow the atomic read/write implementation from PGO counters and cut it off at the end of compilation with all the atomics converted into binary data. This was our initial design and we’ve seen promising sample correlation quality with it. However, the atomics approach has a couple issues:

1. IR Optimizations are blocked unexpectedly. Those atomic instructions are not going to be physically present in the binary code, but since they are on the IR till very end of compilation, they can still prevent certain IR optimizations and result in lower code quality.
2. The counter atomics may not be fully cleaned up from the code stream eventually.
3. Extra work is needed for re-targeting.

We choose to implement pseudo probes based on a special LLVM intrinsic, which is expected to have most of the semantics that comes with an atomic operation but does not block desired optimizations as much as possible. More specifically the semantics associated with the new intrinsic enforces a pseudo probe to be virtually executed exactly the same number of times before and after an IR optimization. The intrinsic also comes with certain flags that are carefully chosen so that the places they are probing are not going to be messed up by the optimizer while most of the IR optimizations still work. The core flags given to the special intrinsic is `IntrInaccessibleMemOnly`, which means the intrinsic accesses memory and does have a side effect so that it is not removable, but is does not access memory locations that are accessible by any original instructions. This way the intrinsic does not alias with any original instruction and thus it does not block optimizations as much as an atomic operation does. We also assign a function GUID and a block index to an intrinsic so that they are uniquely identified and not merged in order to achieve good correlation quality.

Let's now look at an example. Given the following LLVM IR:

```
define internal void @foo2(i32 %x, void (i32)* %f) !dbg !4 {
bb0:
%cmp = icmp eq i32 %x, 0
br i1 %cmp, label %bb1, label %bb2
bb1:
br label %bb3
bb2:
br label %bb3
bb3:
ret void
}
```

The instrumented IR will look like below. Note that each `llvm.pseudoprobe` intrinsic call represents a pseudo probe at a block, of which the first parameter is the GUID of the probe’s owner function and the second parameter is the probe’s ID.

```
define internal void @foo2(i32 %x, void (i32)* %f) !dbg !4 {
bb0:
%cmp = icmp eq i32 %x, 0
call void @llvm.pseudoprobe(i64 837061429793323041, i64 1)
br i1 %cmp, label %bb1, label %bb2
bb1:
call void @llvm.pseudoprobe(i64 837061429793323041, i64 2)
br label %bb3
bb2:
call void @llvm.pseudoprobe(i64 837061429793323041, i64 3)
br label %bb3
bb3:
call void @llvm.pseudoprobe(i64 837061429793323041, i64 4)
ret void
}

```

Reviewed By: wmi

Differential Revision: https://reviews.llvm.org/D86490

show more ...

[CodeGen] Enable tail call position check for speculatable functions

In the function "Analysis.cpp:isInTailCallPosition", it only checks whether
a call is in a tail call position if the call has sid

[CodeGen] Enable tail call position check for speculatable functions

In the function "Analysis.cpp:isInTailCallPosition", it only checks whether
a call is in a tail call position if the call has side effects, access memory
or it is not safe to speculative execute. Therefore, a speculatable function
will not go through tail call position check and improperly tail called when
it is not in a tail-call position. This patch enables tail call position check
for speculatable functions.

Differential Revision: https://reviews.llvm.org/D80661

show more ...

TargetLowering.h - remove unnecessary TargetMachine.h include. NFC

Replace with forward declaration and move dependency down to source files that actually need it.

Both TargetLowering.h and TargetM

TargetLowering.h - remove unnecessary TargetMachine.h include. NFC

Replace with forward declaration and move dependency down to source files that actually need it.

Both TargetLowering.h and TargetMachine.h are 2 of the most expensive headers (top 10) in the ClangBuildAnalyzer report when building llc.

show more ...

[CallSite removal][CodeGen] Replace ImmutableCallSite with CallBase in isInTailCallPosition.

[CallSite removal][CodeGen] Use CallBase instead of CallSite in getNoopInput in Analysis.cpp. NFC

Remove CompositeType class.

The existence of the class is more confusing than helpful, I think; the
commonality is mostly just "GEP is legal", which can be queried using
APIs on GetElementPtrInst.

Remove CompositeType class.

The existence of the class is more confusing than helpful, I think; the
commonality is mostly just "GEP is legal", which can be queried using
APIs on GetElementPtrInst.

Differential Revision: https://reviews.llvm.org/D75660

show more ...

[NFC] Fix trivial typos in comments

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D72143

Patch by Kazuaki Ishizaki.

[Analysis] Attribute deref/deref_or_null should not prevent tail call optimization

Fix missing memcpy, memmove and memset tail calls

Summary:
If a wrapper around one of the mem* stdlib functions bitcasts the returned
pointer value before returning it (e.g. to a wchar_t*), LLVM doe

Fix missing memcpy, memmove and memset tail calls

Summary:
If a wrapper around one of the mem* stdlib functions bitcasts the returned
pointer value before returning it (e.g. to a wchar_t*), LLVM does not emit a
tail call.

Add a check for this scenario so that we emit a tail call.

Reviewers: wmi, mkuper, ramred01, dmgreen

Reviewed By: wmi, dmgreen

Subscribers: hiraditya, sanwou01, javed.absar, lebedev.ri, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D59078

show more ...

[SVE][IR] Scalable Vector size queries and IR instruction support

* Adds a TypeSize struct to represent the known minimum size of a type
along with a flag to indicate that the runtime size is a in

[SVE][IR] Scalable Vector size queries and IR instruction support

* Adds a TypeSize struct to represent the known minimum size of a type
along with a flag to indicate that the runtime size is a integer multiple
of that size
* Converts existing size query functions from Type.h and DataLayout.h to
return a TypeSize result
* Adds convenience methods (including a transparent conversion operator
to uint64_t) so that most existing code 'just works' as if the return
values were still scalars.
* Uses the new size queries along with ElementCount to ensure that all
supported instructions used with scalable vectors can be constructed
in IR.

Reviewers: hfinkel, lattner, rkruppe, greened, rovka, rengolin, sdesmalen

Reviewed By: rovka, sdesmalen

Differential Revision: https://reviews.llvm.org/D53137

llvm-svn: 374042

show more ...

[X86] Add new calling convention that guarantees tail call optimization

When the target option GuaranteedTailCallOpt is specified, calls with
the fastcc calling convention will be transformed into t

[X86] Add new calling convention that guarantees tail call optimization

When the target option GuaranteedTailCallOpt is specified, calls with
the fastcc calling convention will be transformed into tail calls if
they are in tail position. This diff adds a new calling convention,
tailcc, currently supported only on X86, which behaves the same way as
fastcc, except that the GuaranteedTailCallOpt flag does not need to
enabled in order to enable tail call optimization.

Patch by Dwight Guth <dwight.guth@runtimeverification.com>!

Reviewed By: lebedev.ri, paquette, rnk

Differential Revision: https://reviews.llvm.org/D67855

llvm-svn: 373976

show more ...

IR. Change strip* family of functions to not look through aliases.

I noticed another instance of the issue where references to aliases were
being replaced with aliasees, this time in InstCombine. In

IR. Change strip* family of functions to not look through aliases.

I noticed another instance of the issue where references to aliases were
being replaced with aliasees, this time in InstCombine. In the instance that
I saw it turned out to be only a QoI issue (a symbol ended up being missing
from the symbol table due to the last reference to the alias being removed,
preventing HWASAN from symbolizing a global reference), but it could easily
have manifested as incorrect behaviour.

Since this is the third such issue encountered (previously: D65118, D65314)
it seems to be time to address this common error/QoI issue once and for all
and make the strip* family of functions not look through aliases.

Includes a test for the specific issue that I saw, but no doubt there are
other similar bugs fixed here.

As with D65118 this has been tested to make sure that the optimization isn't
load bearing. I built Clang, Chromium for Linux, Android and Windows as well
as the test-suite and there were no size regressions.

Differential Revision: https://reviews.llvm.org/D66606

llvm-svn: 369697

show more ...

[CodeGen/Analysis] Intrinsic llvm.assume should not block tail call optimization

In function Analysis.cpp:isInTailCallPosition, instructions between call and ret are checked to see if they block tai

[CodeGen/Analysis] Intrinsic llvm.assume should not block tail call optimization

In function Analysis.cpp:isInTailCallPosition, instructions between call and ret are checked to see if they block tail call optimization. If an instruction is an intrinsic call, only llvm.lifetime_end is allowed and other intrinsic functions block tail call. When compiling tcmalloc, we found llvm.assume between a hot function call and ret, it blocks the optimization. But llvm.assume doesn't generate instructions, it should not block tail call.

Differential Revision: https://reviews.llvm.org/D66096

llvm-svn: 369125

show more ...

CodeGen: Don't follow aliases when extracting type info.

This fixes a crash in the case where the type info object is an alias
pointing to a non-zero offset within a global or is otherwise unanalyza

CodeGen: Don't follow aliases when extracting type info.

This fixes a crash in the case where the type info object is an alias
pointing to a non-zero offset within a global or is otherwise unanalyzable
by the stripPointerCasts() function. Looking through the alias is not the
right thing to do anyway for similar reasons as D65118.

Differential Revision: https://reviews.llvm.org/D65314

llvm-svn: 367696

show more ...

DAG: allow DAG pointer size different from memory representation.

In preparation for supporting ILP32 on AArch64, this modifies the SelectionDAG
builder code so that pointers are allowed to have a l

DAG: allow DAG pointer size different from memory representation.

In preparation for supporting ILP32 on AArch64, this modifies the SelectionDAG
builder code so that pointers are allowed to have a larger type when "live" in
the DAG compared to memory.

Pointers get zero-extended whenever they are loaded, and truncated prior to
stores. In addition, a few not quite so obvious locations need updating:

* A GEP that has not been marked inbounds needs to enforce the IR-documented
2s-complement wrapping at the memory pointer size. Inbounds GEPs are
undefined if they overflow the address space, so no additional operations
are needed.
* Signed comparisons would give incorrect results if performed on the
zero-extended values.

This shouldn't affect CodeGen for now, but will become active when the AArch64
ILP32 support is committed.

llvm-svn: 359676

show more ...

GlobalISel: Move computeValueLLTs

Call lowering should use this directly instead of going through the
EVT version, but more work is needed to deal with this (mostly the
passing of the IR type pointe

GlobalISel: Move computeValueLLTs

Call lowering should use this directly instead of going through the
EVT version, but more work is needed to deal with this (mostly the
passing of the IR type pointer instead of the relevant properties in
ArgInfo).

llvm-svn: 358111

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

[CodeGen] Ignore return sext/zext attributes of unused results for tail calls

If the caller's return type does not have a zeroext attribute but the
callee does a tail call zeroext, we won't consider

[CodeGen] Ignore return sext/zext attributes of unused results for tail calls

If the caller's return type does not have a zeroext attribute but the
callee does a tail call zeroext, we won't consider the tail call during
CodeGenPrepare because the attributes don't match.

However, if the result of the tail call has no uses, it makes sense to
drop the sext/zext attributes.

Differential Revision: https://reviews.llvm.org/D56486

llvm-svn: 350753

show more ...

[CodeGen] skip lifetime end marker in isInTailCallPosition

A lifetime end intrinsic between a tail call and the return should not
prevent the call from being tail call optimized.

Differential Revis

[CodeGen] skip lifetime end marker in isInTailCallPosition

A lifetime end intrinsic between a tail call and the return should not
prevent the call from being tail call optimized.

Differential Revision: https://reviews.llvm.org/D53519

llvm-svn: 345163

show more ...

[TI removal] Make variables declared as `TerminatorInst` and initialized
by `getTerminator()` calls instead be declared as `Instruction`.

This is the biggest remaining chunk of the usage of `getTerm

[TI removal] Make variables declared as `TerminatorInst` and initialized
by `getTerminator()` calls instead be declared as `Instruction`.

This is the biggest remaining chunk of the usage of `getTerminator()`
that insists on the narrow type and so is an easy batch of updates.
Several files saw more extensive updates where this would cascade to
requiring API updates within the file to use `Instruction` instead of
`TerminatorInst`. All of these were trivial in nature (pervasively using
`Instruction` instead just worked).

llvm-svn: 344502

show more ...

[CodeGen] Enable tail calls for functions with NonNull attributes.

Adding NonNull as attributes to returned pointers has the unfortunate side
effect of disabling tail calls. This patch ignores the N

[CodeGen] Enable tail calls for functions with NonNull attributes.

Adding NonNull as attributes to returned pointers has the unfortunate side
effect of disabling tail calls. This patch ignores the NonNull attribute when
we decide whether to tail merge, in the same way that we ignore the NoAlias
attribute, as it has no affect on the call sequence.

Differential Revision: https://reviews.llvm.org/D52238

llvm-svn: 343091

show more ...

[WebAssembly] Add isEHScopeReturn instruction property

Summary:
So far, `isReturn` property is used to mean both a return instruction
from a functon and the end of an EH scope, a scope that starts w

[WebAssembly] Add isEHScopeReturn instruction property

Summary:
So far, `isReturn` property is used to mean both a return instruction
from a functon and the end of an EH scope, a scope that starts with a EH
scope entry BB and ends with a catchret or a cleanupret instruction.
Because WinEH uses funclets, all EH-scope-ending instructions are also
real return instruction from a function. But for wasm, they only serve
as the end marker of an EH scope but not a return instruction that
exits a function. This mismatch caused incorrect prolog and epilog
generation in wasm EH scopes. This patch fixes this.

This patch is in the same vein with rL333045, which splits
`MachineBasicBlock::isEHFuncletEntry` into `isEHFuncletEntry` and
`isEHScopeEntry`.

Reviewers: dschuff

Subscribers: sbc100, jgravelle-google, sunfish, llvm-commits

Differential Revision: https://reviews.llvm.org/D50653

llvm-svn: 340325

show more ...

Change ambiguous uses of term 'funclet' to 'EH scopes'. NFC.

Summary:
`getEHScopeMembership()` function is used not only for funclet-based
EHs; they apply to all EH schemes that use the scoped IR
(c

Change ambiguous uses of term 'funclet' to 'EH scopes'. NFC.

Summary:
`getEHScopeMembership()` function is used not only for funclet-based
EHs; they apply to all EH schemes that use the scoped IR
(catchpad/cleanuppad/...). D47005 (rL333045) changed some of the uses of
the term 'funclet' to 'EH scopes' in case they apply to all scoped EH,
and this fixes more of them. For `FuncletLayout` pass, I left it as is
because the pass is only used for funclet-based EH.

Reviewers: majnemer

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D47611

llvm-svn: 333711

show more ...