"Reapply "[llvm-jitlink] Use concurrent linking by default." with more fixes.

This reapplies edca1d9bad2 which was reverted in 7ec139ad4bc due to bot
failures.

LocalDependencyPropagation.s is updat

"Reapply "[llvm-jitlink] Use concurrent linking by default." with more fixes.

This reapplies edca1d9bad2 which was reverted in 7ec139ad4bc due to bot
failures.

LocalDependencyPropagation.s is updated to use -num-threads=0 in order to
avoid interleaving debugging output.

ELFNixPlatform.h is updated to protect the deferred runtime function calls
map during bootstrap.

show more ...

Revert "Reapply "[llvm-jitlink] Use concurrent linking by default." with fixes. (#120958)"

Caused random failures.

This reverts commit 93d4b1f7a72f366c1ea91b2d65991266053be8d9.
(llvmorg-20-init-162

Revert "Reapply "[llvm-jitlink] Use concurrent linking by default." with fixes. (#120958)"

Caused random failures.

This reverts commit 93d4b1f7a72f366c1ea91b2d65991266053be8d9.
(llvmorg-20-init-16299-g93d4b1f7a72f)

show more ...

Reapply "[llvm-jitlink] Use concurrent linking by default." with fixes. (#120958)

Reapplies commit edca1d9bad2 which was reverted in 34531cff638 while I
investigated bot failures, (e.g.
https://la

Reapply "[llvm-jitlink] Use concurrent linking by default." with fixes. (#120958)

Reapplies commit edca1d9bad2 which was reverted in 34531cff638 while I
investigated bot failures, (e.g.
https://lab.llvm.org/buildbot/#/builders/137/builds/10791).

Commit 158a60051d2 should address the -check failures on the bots, which
were caused by checks running earlier under the concurrent linking
scheme before all files referenced by the checks had been fully linked.
This patch also fixes the -threads option failure by renaming the option
to -num-threads to avoid clashing with the ThreadCount cl::opt variable
defined in ThinLTOCodeGenerator.cpp.

show more ...

Revert "[llvm-jitlink] Use concurrent linking by default."

This reverts commit edca1d9bad2 while I investigate bot failures,
e.g. https://lab.llvm.org/buildbot/#/builders/137/builds/10791.

[llvm-jitlink] Use concurrent linking by default.

Adds a -threads option to llvm-jitlink. By default llvm-jitlink will now use a
DynamicThreadPoolTaskDispatcher with the number of materialization th

[llvm-jitlink] Use concurrent linking by default.

Adds a -threads option to llvm-jitlink. By default llvm-jitlink will now use a
DynamicThreadPoolTaskDispatcher with the number of materialization threads set
to whatever is returned by std::hardware_concurrency(). This brings the default
in-place linking behavior in line with the concurrent linking that is used for
-oop-executor and -oop-executor-connect mode.

In-place linking on the main thread can be forced by passing -threads=0.

show more ...

[llvm-jitlink] Refactor GOT and stubs registration (NFC) (#78365)

Add methods `registerGOTEntry()` and `registerStubEntry()` in
`Session::FileInfo` to factor out generic code from the individual
o

[llvm-jitlink] Refactor GOT and stubs registration (NFC) (#78365)

Add methods `registerGOTEntry()` and `registerStubEntry()` in
`Session::FileInfo` to factor out generic code from the individual
object type implementations.

show more ...

[jitlink][rtdyld][checker] Re-apply 4b17c81d5a5 with fixes.

This re-applies 4b17c81d5a5, "[jitlink/rtdydl][checker] Add TargetFlag
dependent disassembler switching support", which was reverted in
48

[jitlink][rtdyld][checker] Re-apply 4b17c81d5a5 with fixes.

This re-applies 4b17c81d5a5, "[jitlink/rtdydl][checker] Add TargetFlag
dependent disassembler switching support", which was reverted in
4871a9ca546 due to bot failures.

The patch has been updated to add missing plumbing for Subtarget Features and
a CPU string, which should fix the failing tests.

https://reviews.llvm.org/D158280

show more ...

Revert "[jitlink/rtdydl][checker] Add TargetFlag dependent disassembler switching support"

This reverts commit 4b17c81d5a5d3e0f514026c2b7f9b623d901cc04.

Caused buildbot failures:
https://lab.llvm.o

Revert "[jitlink/rtdydl][checker] Add TargetFlag dependent disassembler switching support"

This reverts commit 4b17c81d5a5d3e0f514026c2b7f9b623d901cc04.

Caused buildbot failures:
https://lab.llvm.org/buildbot/#/builders/230/builds/18341
https://lab.llvm.org/buildbot/#/builders/109/builds/73169
https://lab.llvm.org/buildbot/#/builders/67/builds/12597

show more ...

[jitlink/rtdydl][checker] Add TargetFlag dependent disassembler switching support

Some targets such as AArch32 make use of TargetFlags to indicate ISA mode. Depending
on the TargetFlag, MCDisassembl

[jitlink/rtdydl][checker] Add TargetFlag dependent disassembler switching support

Some targets such as AArch32 make use of TargetFlags to indicate ISA mode. Depending
on the TargetFlag, MCDisassembler and similar target specific objects should be
reinitialized with the correct Target Triple. Backends with similar needs can
easily extend this implementation for their usecase.

The drivers llvm-rtdyld and llvm-jitlink have their SymbolInfo's extended to take
TargetFlag into account. RuntimeDyldChecker can now create necessary TargetInfo
to reinitialize MCDisassembler and MCInstPrinter. The required triple is obtained
from the new getTripleFromTargetFlag function by checking the TargetFlag.

In addition, breaking changes for RuntimeDyld COFF Thumb tests are fixed by making
the backend emit a TargetFlag.

Reviewed By: lhames, sgraenitz

Differential Revision: https://reviews.llvm.org/D158280

show more ...

[llvm] Use x.empty() instead of llvm::empty(x) (NFC)

I'm planning to deprecate and eventually remove llvm::empty.

I thought about replacing llvm::empty(x) with std::empty(x), but it
turns out that

[llvm] Use x.empty() instead of llvm::empty(x) (NFC)

I'm planning to deprecate and eventually remove llvm::empty.

I thought about replacing llvm::empty(x) with std::empty(x), but it
turns out that all uses can be converted to x.empty(). That is, no
use requires the ability of std::empty to accept C arrays and
std::initializer_list.

Differential Revision: https://reviews.llvm.org/D133677

show more ...

[llvm] Use llvm::find_if (NFC)

Re-apply "[JITLink] Update JITLink to use ExecutorAddr rather... " with fixes.

This re-applies 133f86e95492b2a00b944e070878424cfa73f87c, which was reverted in
c5965a411c635106a47738b8d2e24db822b7416

Re-apply "[JITLink] Update JITLink to use ExecutorAddr rather... " with fixes.

This re-applies 133f86e95492b2a00b944e070878424cfa73f87c, which was reverted in
c5965a411c635106a47738b8d2e24db822b7416f while I investigated bot failures.

The original failure contained an arithmetic conversion think-o (on line 419 of
EHFrameSupport.cpp) that could cause failures on 32-bit platforms. The issue
should be fixed in this patch.

show more ...

Revert "[JITLink] Update JITLink to use ExecutorAddr rather than..."

This reverts commit 133f86e95492b2a00b944e070878424cfa73f87c while I investigate
the bot failures at https://lab.llvm.org/buildbo

Revert "[JITLink] Update JITLink to use ExecutorAddr rather than..."

This reverts commit 133f86e95492b2a00b944e070878424cfa73f87c while I investigate
the bot failures at https://lab.llvm.org/buildbot#builders/186/builds/3370.

show more ...

[JITLink] Update JITLink to use ExecutorAddr rather than JITTargetAddress.

ExecutorAddr is the preferred representation for executor process addresses now.

[JITLink] Switch from StringRef to ArrayRef<char>, add some generic x86-64 utils

Adds utilities for creating anonymous pointers and jump stubs to x86_64.h. These
are used by the GOT and Stubs builde

[JITLink] Switch from StringRef to ArrayRef<char>, add some generic x86-64 utils

Adds utilities for creating anonymous pointers and jump stubs to x86_64.h. These
are used by the GOT and Stubs builder, but may also be used by pass writers who
want to create pointer stubs for indirection.

This patch also switches the underlying type for LinkGraph content from
StringRef to ArrayRef<char>. This avoids any confusion when working with buffers
that contain null bytes in the middle like, for example, a newly added null
pointer content array. ;)

show more ...

[llvm-jitlink] Add suppport for testing GOT entries and stubs for ELF.

This enables regression testing of GOT and stub handling with
llvm-jitlink.

[JITLink] Improve llvm-jitlink regression testing support for ELF.

This patch adds a jitlink pass, 'registerELFGraphInfo', that records section
and symbol information about each LinkGraph in the llv

[JITLink] Improve llvm-jitlink regression testing support for ELF.

This patch adds a jitlink pass, 'registerELFGraphInfo', that records section
and symbol information about each LinkGraph in the llvm-jitlink session object.
This allows symbols and sections to be referred to by name in llvm-jitlink
regression tests. This will enable a testcase to be written for
https://reviews.llvm.org/D80613.

show more ...

[JITLink] Fix 80-column rule violation.

[llvm-jitlink] Fix DEBUG_TYPE string.

This updates the DEBUG_TYPE string in llvm-jitlink-macho.cpp to reflect the
change made to llvm-jitlink.cpp in 66128c48400f03f2b423561a63b3e78a355fd349.

[JITLink] Remove the Section::symbols_empty() method.

llvm::empty(Sec.symbols()) can be used instead.

[JITLink] Switch from an atom-based model to a "blocks and symbols" model.

In the Atom model the symbols, content and relocations of a relocatable object
file are represented as a graph of atoms, wh

[JITLink] Switch from an atom-based model to a "blocks and symbols" model.

In the Atom model the symbols, content and relocations of a relocatable object
file are represented as a graph of atoms, where each Atom represents a
contiguous block of content with a single name (or no name at all if the
content is anonymous), and where edges between Atoms represent relocations.
If more than one symbol is associated with a contiguous block of content then
the content is broken into multiple atoms and layout constraints (represented by
edges) are introduced to ensure that the content remains effectively contiguous.
These layout constraints must be kept in mind when examining the content
associated with a symbol (it may be spread over multiple atoms) or when applying
certain relocation types (e.g. MachO subtractors).

This patch replaces the Atom model in JITLink with a blocks-and-symbols model.
The blocks-and-symbols model represents relocatable object files as bipartite
graphs, with one set of nodes representing contiguous content (Blocks) and
another representing named or anonymous locations (Symbols) within a Block.
Relocations are represented as edges from Blocks to Symbols. This scheme
removes layout constraints (simplifying handling of MachO alt-entry symbols,
and hopefully ELF sections at some point in the future) and simplifies some
relocation logic.

llvm-svn: 373689

show more ...

[JITLink] Add a test for zero-filled content.

Also updates RuntimeDyldChecker and llvm-rtdyld to support zero-fill tests by
returning a content address of zero (but no error) for zero-fill atoms, an

[JITLink] Add a test for zero-filled content.

Also updates RuntimeDyldChecker and llvm-rtdyld to support zero-fill tests by
returning a content address of zero (but no error) for zero-fill atoms, and
treating loads from zero as returning zero.

llvm-svn: 360547

show more ...

Initial implementation of JITLink - A replacement for RuntimeDyld.

Summary:

JITLink is a jit-linker that performs the same high-level task as RuntimeDyld:
it parses relocatable object files and mak

Initial implementation of JITLink - A replacement for RuntimeDyld.

Summary:

JITLink is a jit-linker that performs the same high-level task as RuntimeDyld:
it parses relocatable object files and makes their contents runnable in a target
process.

JITLink aims to improve on RuntimeDyld in several ways:

(1) A clear design intended to maximize code-sharing while minimizing coupling.

RuntimeDyld has been developed in an ad-hoc fashion for a number of years and
this had led to intermingling of code for multiple architectures (e.g. in
RuntimeDyldELF::processRelocationRef) in a way that makes the code more
difficult to read, reason about, extend. JITLink is designed to isolate
format and architecture specific code, while still sharing generic code.

(2) Support for native code models.

RuntimeDyld required the use of large code models (where calls to external
functions are made indirectly via registers) for many of platforms due to its
restrictive model for stub generation (one "stub" per symbol). JITLink allows
arbitrary mutation of the atom graph, allowing both GOT and PLT atoms to be
added naturally.

(3) Native support for asynchronous linking.

JITLink uses asynchronous calls for symbol resolution and finalization: these
callbacks are passed a continuation function that they must call to complete the
linker's work. This allows for cleaner interoperation with the new concurrent
ORC JIT APIs, while still being easily implementable in synchronous style if
asynchrony is not needed.

To maximise sharing, the design has a hierarchy of common code:

(1) Generic atom-graph data structure and algorithms (e.g. dead stripping and
| memory allocation) that are intended to be shared by all architectures.
|
+ -- (2) Shared per-format code that utilizes (1), e.g. Generic MachO to
| atom-graph parsing.
|
+ -- (3) Architecture specific code that uses (1) and (2). E.g.
JITLinkerMachO_x86_64, which adds x86-64 specific relocation
support to (2) to build and patch up the atom graph.

To support asynchronous symbol resolution and finalization, the callbacks for
these operations take continuations as arguments:

using JITLinkAsyncLookupContinuation =
std::function<void(Expected<AsyncLookupResult> LR)>;

using JITLinkAsyncLookupFunction =
std::function<void(const DenseSet<StringRef> &Symbols,
JITLinkAsyncLookupContinuation LookupContinuation)>;

using FinalizeContinuation = std::function<void(Error)>;

virtual void finalizeAsync(FinalizeContinuation OnFinalize);

In addition to its headline features, JITLink also makes other improvements:

- Dead stripping support: symbols that are not used (e.g. redundant ODR
definitions) are discarded, and take up no memory in the target process
(In contrast, RuntimeDyld supported pointer equality for weak definitions,
but the redundant definitions stayed resident in memory).

- Improved exception handling support. JITLink provides a much more extensive
eh-frame parser than RuntimeDyld, and is able to correctly fix up many
eh-frame sections that RuntimeDyld currently (silently) fails on.

- More extensive validation and error handling throughout.

This initial patch supports linking MachO/x86-64 only. Work on support for
other architectures and formats will happen in-tree.

Differential Revision: https://reviews.llvm.org/D58704

llvm-svn: 358818

show more ...