Fixup for #95588, don't assume `align 8`.

This shall fix aarch64 builders.

InstrProf: Mark BiasLI as invariant. (#95588)

Bias doesn't change after startup.

The test is enhanced for optimized sequences and atomic ops.

InstProfiling: Give the name to profc_bias. NFC. (#95587)

[PGO][InstrProf][test] Convert to opaque pointers and fix some stale tests

[test] Migrate legacy InstrProfiling tests to new PM

[InstrProf] Use i32 for GEP index from lowering llvm.instrprof.increment

The `llvm.instrprof.increment` intrinsic uses `i32` for the index. We should use this same type for the index into the GEP in

[InstrProf] Use i32 for GEP index from lowering llvm.instrprof.increment

The `llvm.instrprof.increment` intrinsic uses `i32` for the index. We should use this same type for the index into the GEP instructions.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D114268

show more ...

Revert "[NFC] `IRBuilderBase::CreateAdd()`: place constant onto RHS"

Clang OpenMP codegen tests are failing,
will recommit afterwards.

This reverts commit 4723c9b3c6c46632a5d66e65d198899894b1e2c5.

[NFC] `IRBuilderBase::CreateAdd()`: place constant onto RHS

[InstrProfiling] Use weak alias for bias variable

We need the compiler generated variable to override the weak symbol of
the same name inside the profile runtime, but using LinkOnceODRLinkage
result

[InstrProfiling] Use weak alias for bias variable

We need the compiler generated variable to override the weak symbol of
the same name inside the profile runtime, but using LinkOnceODRLinkage
results in weak symbol being emitted in which case the symbol selected
by the linker is going to depend on the order of inputs which can be
fragile.

This change replaces the use of weak definition inside the runtime with
a weak alias. We place the compiler generated symbol inside a COMDAT
group so dead definition can be garbage collected by the linker.

We also disable the use of runtime counter relocation on Darwin since
Mach-O doesn't support weak external references, but Darwin already uses
a different continous mode that relies on overmapping so runtime counter
relocation isn't needed there.

Differential Revision: https://reviews.llvm.org/D105176

show more ...

Revert "[InstrProfiling] Use external weak reference for bias variable"

This reverts commit 33a7b4d9d8e6a113108aa71ed78ca32a83c68523.
Breaks check-profile on macOS, see comments on https://reviews.l

Revert "[InstrProfiling] Use external weak reference for bias variable"

This reverts commit 33a7b4d9d8e6a113108aa71ed78ca32a83c68523.
Breaks check-profile on macOS, see comments on https://reviews.llvm.org/D105176

show more ...

[InstrProfiling] Use external weak reference for bias variable

We need the compiler generated variable to override the weak symbol of
the same name inside the profile runtime, but using LinkOnceODRL

[InstrProfiling] Use external weak reference for bias variable

We need the compiler generated variable to override the weak symbol of
the same name inside the profile runtime, but using LinkOnceODRLinkage
results in weak symbol being emitted which leads to an issue where the
linker might choose either of the weak symbols potentially disabling the
runtime counter relocation.

This change replaces the use of weak definition inside the runtime with
an external weak reference to address the issue. We also place the
compiler generated symbol inside a COMDAT group so dead definition can
be garbage collected by the linker.

Differential Revision: https://reviews.llvm.org/D105176

show more ...

[InstrProfiling][test] Fix stale tests

* Change linkage/visibility of __profn_ variables to match the reality
* alwaysinline.ll: Add "EnableValueProfiling", otherwise it doesn't test available_exter

[InstrProfiling][test] Fix stale tests

* Change linkage/visibility of __profn_ variables to match the reality
* alwaysinline.ll: Add "EnableValueProfiling", otherwise it doesn't test available_externally alwaysinline.
* Delete PR23499.ll - covered by other comdat tests.

show more ...

[Instrumentation] Set hidden visibility for the bias variable

We have to avoid using a GOT relocation to access the bias variable,
setting the hidden visibility achieves that.

Differential Revision

[Instrumentation] Set hidden visibility for the bias variable

We have to avoid using a GOT relocation to access the bias variable,
setting the hidden visibility achieves that.

Differential Revision: https://reviews.llvm.org/D73529

show more ...

[profile] Support counter relocation at runtime

This is an alternative to the continous mode that was implemented in
D68351. This mode relies on padding and the ability to mmap a file over
the exist

[profile] Support counter relocation at runtime

This is an alternative to the continous mode that was implemented in
D68351. This mode relies on padding and the ability to mmap a file over
the existing mapping which is generally only available on POSIX systems
and isn't suitable for other platforms.

This change instead introduces the ability to relocate counters at
runtime using a level of indirection. On every counter access, we add a
bias to the counter address. This bias is stored in a symbol that's
provided by the profile runtime and is initially set to zero, meaning no
relocation. The runtime can mmap the profile into memory at abitrary
location, and set bias to the offset between the original and the new
counter location, at which point every subsequent counter access will be
to the new location, which allows updating profile directly akin to the
continous mode.

The advantage of this implementation is that doesn't require any special
OS support. The disadvantage is the extra overhead due to additional
instructions required for each counter access (overhead both in terms of
binary size and performance) plus duplication of counters (i.e. one copy
in the binary itself and another copy that's mmapped).

Differential Revision: https://reviews.llvm.org/D69740

show more ...