[LLVM][IR][Sanitizers] Add sanitize_numerical_stability attribute (#95051)

Add sanitize_numerical_stability attribute.

[DebugInfo] Remap extracted DIAssignIDs in hotcoldsplit (#91940)

Fix #91814

When instructions are extracted into a new function the `DIAssignID` metadata
uses and attachments need to be remapped

[DebugInfo] Remap extracted DIAssignIDs in hotcoldsplit (#91940)

Fix #91814

When instructions are extracted into a new function the `DIAssignID` metadata
uses and attachments need to be remapped so that the stores and assignment
markers don't link to stores and assignment markers in the original function.

This matches existing inlining behaviour for DIAssignIDs.

show more ...

[CodeExtractor] Resolving the Inconsistency of Compiled Binary Files (#86497)

When the compiler enables ALSR by default, binary inconsistency occurs
when the extractCodeRegion function is called. T

[CodeExtractor] Resolving the Inconsistency of Compiled Binary Files (#86497)

When the compiler enables ALSR by default, binary inconsistency occurs
when the extractCodeRegion function is called. The reason is that the
sequence of traversing ExitBlocks containers of the SmallPtrSet type is
changed due to randomization of BasicBlock pointers.
This fixes https://github.com/llvm/llvm-project/issues/86427

show more ...

[RemoveDIs][NFC] Rename DPLabel->DbgLabelRecord (#85918)

This patch renames DPLabel to DbgLabelRecord, in accordance with the
ongoing DbgRecord rename. This rename was fairly trivial, since DPLabel

[RemoveDIs][NFC] Rename DPLabel->DbgLabelRecord (#85918)

This patch renames DPLabel to DbgLabelRecord, in accordance with the
ongoing DbgRecord rename. This rename was fairly trivial, since DPLabel
isn't as widely used as DPValue and has no real conflicts in either its
full or abbreviated name. As usual, the entire replacement was done
automatically, with `s/DPLabel/DbgLabelRecord/` and `s/DPL/DLR/`.

show more ...

[RemoveDIs][NFC] Rename DPValue -> DbgVariableRecord (#85216)

This is the major rename patch that prior patches have built towards.
The DPValue class is being renamed to DbgVariableRecord, which re

[RemoveDIs][NFC] Rename DPValue -> DbgVariableRecord (#85216)

This is the major rename patch that prior patches have built towards.
The DPValue class is being renamed to DbgVariableRecord, which reflects
the updated terminology for the "final" implementation of the RemoveDI
feature. This is a pure string substitution + clang-format patch. The
only manual component of this patch was determining where to perform
these string substitutions: `DPValue` and `DPV` are almost exclusively
used for DbgRecords, *except* for:

- llvm/lib/target, where 'DP' is used to mean double-precision, and so
appears as part of .td files and in variable names. NB: There is a
single existing use of `DPValue` here that refers to debug info, which
I've manually updated.
- llvm/tools/gold, where 'LDPV' is used as a prefix for symbol
visibility enums.

Outside of these places, I've applied several basic string
substitutions, with the intent that they only affect DbgRecord-related
identifiers; I've checked them as I went through to verify this, with
reasonable confidence that there are no unintended changes that slipped
through the cracks. The substitutions applied are all case-sensitive,
and are applied in the order shown:

```
DPValue -> DbgVariableRecord
DPVal -> DbgVarRec
DPV -> DVR
```

Following the previous rename patches, it should be the case that there
are no instances of any of these strings that are meant to refer to the
general case of DbgRecords, or anything other than the DPValue class.
The idea behind this patch is therefore that pure string substitution is
correct in all cases as long as these assumptions hold.

show more ...

[CodeExtractor] Terminate callsite blocks to new `noreturn` functions with `unreachable`

Since some of the users of `CodeExtractor` like `HotColdSplitting` run
late in the pipeline, returns are not

[CodeExtractor] Terminate callsite blocks to new `noreturn` functions with `unreachable`

Since some of the users of `CodeExtractor` like `HotColdSplitting` run
late in the pipeline, returns are not cleaned to `unreachable`. So,
just emit `unreachable` directly if the function is `noreturn`.

Closes #84682

show more ...

[RemoveDIs][NFC] Rename common interface functions for DPValues->DbgRecords (#84793)

As part of the effort to rename the DbgRecord classes, this patch
renames the widely-used functions that operate

[RemoveDIs][NFC] Rename common interface functions for DPValues->DbgRecords (#84793)

As part of the effort to rename the DbgRecord classes, this patch
renames the widely-used functions that operate on DbgRecords but refer
to DbgValues or DPValues in their names to refer to DbgRecords instead;
all such functions are defined in one of `BasicBlock.h`,
`Instruction.h`, and `DebugProgramInstruction.h`.

This patch explicitly does not change the names of any comments or
variables, except for where they use the exact name of one of the
renamed functions. The reason for this is reviewability; this patch can
be trivially examined to determine that the only changes are direct
string substitutions and any results from clang-format responding to the
changed line lengths. Future patches will cover renaming variables and
comments, and then renaming the classes themselves.

show more ...

Reapply [IR] Add new Range attribute using new ConstantRange Attribute type (#84617)

The only change from https://github.com/llvm/llvm-project/pull/83171 is the
change of the allocator so the destr

Reapply [IR] Add new Range attribute using new ConstantRange Attribute type (#84617)

The only change from https://github.com/llvm/llvm-project/pull/83171 is the
change of the allocator so the destructor is called for
ConstantRangeAttributeImpl.

reverts https://github.com/llvm/llvm-project/pull/84549

show more ...

Revert "[IR] Add new Range attribute using new ConstantRange Attribute type" (#84549)

Reverts llvm/llvm-project#83171

broke sanitizer buildbot
https://lab.llvm.org/buildbot/#/builders/168/builds

Revert "[IR] Add new Range attribute using new ConstantRange Attribute type" (#84549)

Reverts llvm/llvm-project#83171

broke sanitizer buildbot
https://lab.llvm.org/buildbot/#/builders/168/builds/19110/steps/10/logs/stdio

show more ...

[IR] Add new Range attribute using new ConstantRange Attribute type (#83171)

implementation as discussed in
https://discourse.llvm.org/t/rfc-metadata-attachments-for-function-arguments/76420

[RemoveDIs] Reapply 3fda50d3915, insert instructions using iterators

I'd reverted this in 6c7805d5d1 after a bad stage. Original commit
messsage follows:

[NFC][RemoveDIs] Bulk update utilities to i

[RemoveDIs] Reapply 3fda50d3915, insert instructions using iterators

I'd reverted this in 6c7805d5d1 after a bad stage. Original commit
messsage follows:

[NFC][RemoveDIs] Bulk update utilities to insert with iterators

As part of the RemoveDIs project we need LLVM to insert instructions using
iterators wherever possible, so that the iterators can carry a bit of
debug-info. This commit implements some of that by updating the contents of
llvm/lib/Transforms/Utils to always use iterator-versions of instruction
constructors.

There are two general flavours of update:
* Almost all call-sites just call getIterator on an instruction
* Several make use of an existing iterator (scenarios where the code is
actually significant for debug-info)
The underlying logic is that any call to getFirstInsertionPt or similar
APIs that identify the start of a block need to have that iterator passed
directly to the insertion function, without being converted to a bare
Instruction pointer along the way.

I've also switched DemotePHIToStack to take an optional iterator: it needs
to take an iterator, and having a no-insert-location behaviour appears to
be important. The constructors for ICmpInst and FCmpInst have been updated
too. They're the only instructions that take block _references_ rather than
pointers for certain calls, and a future patch is going to make use of
default-null block insertion locations.

All of this should be NFC.

show more ...

Revert "[NFC][RemoveDIs] Bulk update utilities to insert with iterators"

This reverts commit 3fda50d3915b2163a54a37b602be7783a89dd808.

Apparently I've missed a hunk while staging this; will back ou

Revert "[NFC][RemoveDIs] Bulk update utilities to insert with iterators"

This reverts commit 3fda50d3915b2163a54a37b602be7783a89dd808.

Apparently I've missed a hunk while staging this; will back out for now.

Picked up here: https://lab.llvm.org/buildbot/#/builders/139/builds/60429/steps/6/logs/stdio

show more ...

[NFC][RemoveDIs] Bulk update utilities to insert with iterators

As part of the RemoveDIs project we need LLVM to insert instructions using
iterators wherever possible, so that the iterators can carr

[NFC][RemoveDIs] Bulk update utilities to insert with iterators

As part of the RemoveDIs project we need LLVM to insert instructions using
iterators wherever possible, so that the iterators can carry a bit of
debug-info. This commit implements some of that by updating the contents of
llvm/lib/Transforms/Utils to always use iterator-versions of instruction
constructors.

There are two general flavours of update:
* Almost all call-sites just call getIterator on an instruction
* Several make use of an existing iterator (scenarios where the code is
actually significant for debug-info)
The underlying logic is that any call to getFirstInsertionPt or similar
APIs that identify the start of a block need to have that iterator passed
directly to the insertion function, without being converted to a bare
Instruction pointer along the way.

I've also switched DemotePHIToStack to take an optional iterator: it needs
to take an iterator, and having a no-insert-location behaviour appears to
be important. The constructors for ICmpInst and FCmpInst have been updated
too. They're the only instructions that take block _references_ rather than
pointers for certain calls, and a future patch is going to make use of
default-null block insertion locations.

All of this should be NFC.

show more ...

[RemoveDIs] Add DPLabels support [3a/3] (#82633)

Patch 2 of 3 to add llvm.dbg.label support to the RemoveDIs project. The
patch stack adds the DPLabel class, which is the RemoveDIs llvm.dbg.label

[RemoveDIs] Add DPLabels support [3a/3] (#82633)

Patch 2 of 3 to add llvm.dbg.label support to the RemoveDIs project. The
patch stack adds the DPLabel class, which is the RemoveDIs llvm.dbg.label
equivalent.

1. Add DbgRecord base class for DPValue and the not-yet-added
DPLabel class.
2. Add the DPLabel class.
-> 3. Add support to passes.

The next patch, #82639, will enable conversion between dbg.labels and DPLabels.

AssignemntTrackingAnalysis support could have gone two ways:

1. Have the analysis store a DPLabel representation in its results -
SelectionDAGBuilder reads the analysis results and ignores all DbgRecord
kinds.
2. Ignore DPLabels in the analysis - SelectionDAGBuilder reads the analysis
results but still needs to iterate over DPLabels from the IR.

I went with option 2 because it's less work and is no less correct than 1. It's
worth noting that causes labels to sink to the bottom of packs of debug records.
e.g., [value, label, value] becomes [value, value, label]. This shouldn't be a
problem because labels and variable locations don't have an ordering requirement.
The ordering between variable locations is maintained and the label movement is
deterministic

show more ...

[RemoveDIs][NFC] Introduce DbgRecord base class [1/3] (#78252)

Patch 1 of 3 to add llvm.dbg.label support to the RemoveDIs project. The
patch stack adds a new base class

-> 1. Add DbgRecord

[RemoveDIs][NFC] Introduce DbgRecord base class [1/3] (#78252)

Patch 1 of 3 to add llvm.dbg.label support to the RemoveDIs project. The
patch stack adds a new base class

-> 1. Add DbgRecord base class for DPValue and the not-yet-added
DPLabel class.
2. Add the DPLabel class.
3. Enable dbg.label conversion and add support to passes.

Patches 1 and 2 are NFC.

In the near future we also will rename DPValue to DbgVariableRecord and
DPLabel to DbgLabelRecord, at which point we'll overhaul the function
names too. The name DPLabel keeps things consistent for now.

show more ...

[Extractor][DebugInfo] Don't pick DebugLocs from dbg intrinsics (#80863)

When picking the source location for a branch instruction in the
CodeExtractor, we can end up picking the source location of

[Extractor][DebugInfo] Don't pick DebugLocs from dbg intrinsics (#80863)

When picking the source location for a branch instruction in the
CodeExtractor, we can end up picking the source location of a debugging
intrinsic. This never makes sense because any variable assignment
information (or labels) might originate from completely different
lexical scopes that have been inlined, and also makes the line tables
change between -g and -gmlt. Fix this by skipping debug intrinsics when
looking for branch source locations.

Detected because of test differences with RemoveDIs, the non-intrinsinc
form of debug-info -- fixing in intrinsic form to avoid there being
spurious test differences when we turn it on.

show more ...

[RemoveDIs][DebugInfo] Handle DPVAssign in most transforms (#78986)

This patch trivially updates various opt passes to handle DPVAssigns. In
all cases, this means some combination of generifying ex

[RemoveDIs][DebugInfo] Handle DPVAssign in most transforms (#78986)

This patch trivially updates various opt passes to handle DPVAssigns. In
all cases, this means some combination of generifying existing code to
handle DPValues and DbgAssignIntrinsics, iterating over DPValues where
previously we did not, or duplicating code for DbgAssignIntrinsics to
the equivalent DPValue function (in inlining and salvageDebugInfo).

show more ...

[Transforms] Use a range-based for loop (NFC)

[IR] Add dead_on_unwind attribute (#74289)

Add the `dead_on_unwind` attribute, which states that the caller will
not read from this argument if the call unwinds. This allows eliding
stores that co

[IR] Add dead_on_unwind attribute (#74289)

Add the `dead_on_unwind` attribute, which states that the caller will
not read from this argument if the call unwinds. This allows eliding
stores that could otherwise be visible on the unwind path, for example:

```
declare void @may_unwind()

define void @src(ptr noalias dead_on_unwind %out) {
store i32 0, ptr %out
call void @may_unwind()
store i32 1, ptr %out
ret void
}

define void @tgt(ptr noalias dead_on_unwind %out) {
call void @may_unwind()
store i32 1, ptr %out
ret void
}
```

The optimization is not valid without `dead_on_unwind`, because the `i32
0` value might be read if `@may_unwind` unwinds.

This attribute is primarily intended to be used on sret arguments. In
fact, I previously wanted to change the semantics of sret to include
this "no read after unwind" property (see D116998), but based on the
feedback there it is better to keep these attributes orthogonal (sret is
an ABI attribute, dead_on_unwind is an optimization attribute). This is
a reboot of that change with a separate attribute.

show more ...

[DebugInfo][RemoveDIs] Extract DPValues in CodeExtractor like dbg.values (#73252)

CodeExtractor shifts dbg.value intrinsics out of the region being
extracted and updates them to be appropriate in t

[DebugInfo][RemoveDIs] Extract DPValues in CodeExtractor like dbg.values (#73252)

CodeExtractor shifts dbg.value intrinsics out of the region being
extracted and updates them to be appropriate in the extracted function.
With new non-intrinsic variable locations, we need to manually do this
too, with DPValues.

Most of this patch shifts and refactors some utilities in
fixupDebugInfoPostExtraction so that we can add a single extra helper
lambda that iterates over DPValues and applies update-utilities. We also
have to assign the IsNewDbgInfoFormat flag in a bunch of places -- this
normally gets set the moment you insert a block into a function (or
function into a module), however a few blocks are constructed here
before being inserted, thus we have to do some manual setup.

Tested via LoopExtractor_alloca.ll, which invokes debugify.

show more ...

[Coroutines] Introduce [[clang::coro_only_destroy_when_complete]] (#71014)

Close https://github.com/llvm/llvm-project/issues/56980.

This patch tries to introduce a light-weight optimization attri

[Coroutines] Introduce [[clang::coro_only_destroy_when_complete]] (#71014)

Close https://github.com/llvm/llvm-project/issues/56980.

This patch tries to introduce a light-weight optimization attribute for
coroutines which are guaranteed to only be destroyed after it reached
the final suspend.

The rationale behind the patch is simple. See the example:

```C++
A foo() {
dtor d;
co_await something();
dtor d1;
co_await something();
dtor d2;
co_return 43;
}
```

Generally the generated .destroy function may be:

```C++
void foo.destroy(foo.Frame *frame) {
switch(frame->suspend_index()) {
case 1:
frame->d.~dtor();
break;
case 2:
frame->d.~dtor();
frame->d1.~dtor();
break;
case 3:
frame->d.~dtor();
frame->d1.~dtor();
frame->d2.~dtor();
break;
default: // coroutine completed or haven't started
break;
}

frame->promise.~promise_type();
delete frame;
}
```

Since the compiler need to be ready for all the cases that the coroutine
may be destroyed in a valid state.

However, from the user's perspective, we can understand that certain
coroutine types may only be destroyed after it reached to the final
suspend point. And we need a method to teach the compiler about this.
Then this is the patch. After the compiler recognized that the
coroutines can only be destroyed after complete, it can optimize the
above example to:

```C++
void foo.destroy(foo.Frame *frame) {
frame->promise.~promise_type();
delete frame;
}
```

I spent a lot of time experimenting and experiencing this in the
downstream. The numbers are really good. In a real-world coroutine-heavy
workload, the size of the build dir (including .o files) reduces 14%.
And the size of final libraries (excluding the .o files) reduces 8% in
Debug mode and 1% in Release mode.

show more ...

[NFC] Remove Type::getInt8PtrTy (#71029)

Replace this with PointerType::getUnqual().
Followup to the opaque pointer transition. Fixes an in-code TODO item.

[IR] Add writable attribute

This adds a writable attribute, which in conjunction with
dereferenceable(N) states that a spurious store of N bytes is
introduced on function entry. This implies that th

[IR] Add writable attribute

This adds a writable attribute, which in conjunction with
dereferenceable(N) states that a spurious store of N bytes is
introduced on function entry. This implies that this many bytes
are writable without trapping or introducing data races. See
https://llvm.org/docs/Atomics.html#optimization-outside-atomic for
why the second point is important.

This attribute can be added to sret arguments. I believe Rust will
also be able to use it for by-value (moved) arguments. Rust likely
won't be able to use it for &mut arguments (tree borrows does not
appear to allow spurious stores).

In this patch the new attribute is only used by LICM scalar promotion.
However, the actual motivation for this is to fix a correctness issue
in call slot optimization, which needs this attribute to avoid
optimization regressions.

Followup to the discussion on D157499.

Differential Revision: https://reviews.llvm.org/D158081

show more ...

[CodeExtractor] Allow to use 0 addr space for aggregate arg (#66998)

The user of CodeExtractor should be able to specify that
the aggregate argument should be passed as a pointer in zero address
s

[CodeExtractor] Allow to use 0 addr space for aggregate arg (#66998)

The user of CodeExtractor should be able to specify that
the aggregate argument should be passed as a pointer in zero address
space.

CodeExtractor is used to generate outlined functions required by OpenMP
runtime. The arguments of the outlined functions for OpenMP GPU code
are in 0 address space. 0 address space does not need to be the default
address space for GPU device. That's why there is a need to allow
the user of CodeExtractor to specify, that the allocated aggregate parameter
is passed as pointer in zero address space.

show more ...

[LLVM] Add new attribute `optdebug` to optimize for debugging (#66632)

This patch adds a new fn attribute, `optdebug`, that specifies that
optimizations should make decisions that prioritize debug

[LLVM] Add new attribute `optdebug` to optimize for debugging (#66632)

This patch adds a new fn attribute, `optdebug`, that specifies that
optimizations should make decisions that prioritize debug info quality,
potentially at the cost of runtime performance.

This patch does not add any functional changes triggered by this
attribute, only the attribute itself. A subsequent patch will use this
flag to disable the post-RA scheduler.

show more ...