[dsymutil] Share one BinaryHolder between debug map parsing & linking (#113234)

I (re)discovered that dsymutil was instantiating two BinaryHolders: one
for parsing the debug map and one for linking

[dsymutil] Share one BinaryHolder between debug map parsing & linking (#113234)

I (re)discovered that dsymutil was instantiating two BinaryHolders: one
for parsing the debug map and one for linking. That really defeats the
purpose of the BinaryHolder as it serves as a cache. Fix the issue and
remove an old FIXME.

show more ...

[llvm][tools] Strip unneeded uses of raw_string_ostream::str() (NFC)

Remove excess layer of indirection.

[llvm] Construct SmallVector with ArrayRef (NFC) (#101872)

[dsymutil] Use operator==(StringRef, StringRef) (NFC)

[dsymutil] Fix spurious warnings in MachODebugMapParser (#78794)

When the MachODebugMapParser encounters an object file that cannot be
found on disk, it currently leaves the parser in an incoherent

[dsymutil] Fix spurious warnings in MachODebugMapParser (#78794)

When the MachODebugMapParser encounters an object file that cannot be
found on disk, it currently leaves the parser in an incoherent state,
resulting in spurious warnings that can in turn slow down dsymutil.

This fixes #78411.

rdar://117515153

show more ...

[dsymutil] Improve missing symbol warning message (#75378)

The current warning emitted by dsymutil when it can't find a symbol in
an object file is worded rather poorly:

```
could not find obje

[dsymutil] Improve missing symbol warning message (#75378)

The current warning emitted by dsymutil when it can't find a symbol in
an object file is worded rather poorly:

```
could not find object file symbol for symbol _foo
```

It's also lacking information that makes the warning actionable, such as
the object file it's looking at. This patch rewords the warning and adds
the object file path to the warning:

```
could not find symbol '_foo' in object file 'test.o'
```

rdar://119621065

show more ...

[llvm] Use StringRef::{starts,ends}_with (NFC) (#74956)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::

[llvm] Use StringRef::{starts,ends}_with (NFC) (#74956)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.

I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.

show more ...

Reland [dsymutil] Add support for mergeable libraries (#70256)

Reland https://reviews.llvm.org/D158124
Fixed `-fpermissive` error reported by gcc only.

Revert "[dsymutil] Add support for mergeable libraries"

This reverts commit 122c89b271af30b86536cad7bac64ea9c56615ed. Change does not build, with errors such as:

In file included from ../llvm-proje

Revert "[dsymutil] Add support for mergeable libraries"

This reverts commit 122c89b271af30b86536cad7bac64ea9c56615ed. Change does not build, with errors such as:

In file included from ../llvm-project/llvm/tools/dsymutil/DebugMap.h:24,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
../llvm-project/llvm/tools/dsymutil/RelocationMap.h:60:17: error: declaration of ‘llvm::dsymutil::SymbolMapping llvm::dsymutil::ValidReloc::SymbolMapping’ changes meaning of ‘SymbolMapping’ [-fpermissive]
60 | SymbolMapping SymbolMapping;
| ^~~~~~~~~~~~~
../llvm-project/llvm/tools/dsymutil/RelocationMap.h:36:8: note: ‘SymbolMapping’ declared here as ‘struct llvm::dsymutil::SymbolMapping’
36 | struct SymbolMapping {
| ^~~~~~~~~~~~~
In file included from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
../llvm-project/llvm/tools/dsymutil/DebugMap.h:198:32: error: declaration of ‘std::optional<llvm::dsymutil::RelocationMap> llvm::dsymutil::DebugMapObject::RelocationMap’ changes meaning of ‘RelocationMap’ [-fpermissive]
198 | std::optional<RelocationMap> RelocationMap;
| ^~~~~~~~~~~~~
In file included from ../llvm-project/llvm/tools/dsymutil/DebugMap.h:24,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
../llvm-project/llvm/tools/dsymutil/RelocationMap.h:76:7: note: ‘RelocationMap’ declared here as ‘class llvm::dsymutil::RelocationMap’
76 | class RelocationMap {
| ^~~~~~~~~~~~~

show more ...

[dsymutil] Add support for mergeable libraries

This adds support in dsymutil for mergeable libraries [1].

dsymutil reads a new stab emitted by ld, allowing it to operate on
dynamic libraries instea

[dsymutil] Add support for mergeable libraries

This adds support in dsymutil for mergeable libraries [1].

dsymutil reads a new stab emitted by ld, allowing it to operate on
dynamic libraries instead of object files. It also now loads the DWARF
files associated to the libraries, and build the debug map for each
binary from the list of symbols exported by the library. For each Debug
Map Object, there is a new associated Relocation Map which is serialized
from the information retrieved in the original debug_info (or
debug_addr) section of the .o file.

The final DWARF file has multiple compile units, so the offsets
information of the relocations are adjusted relatively to the compile
unit they will end up belonging to, inside the final linked DWARF file.

[1] https://developer.apple.com/documentation/xcode/configuring-your-project-to-use-mergeable-libraries

Differential revision: https://reviews.llvm.org/D158124

show more ...

[dsymutil] Remove paper trail warnings (#67041)

Remove the paper trail warning from dsymutil and the DWARF linker. The
original purpose of this functionality was to leave a paper trail in the
outp

[dsymutil] Remove paper trail warnings (#67041)

Remove the paper trail warning from dsymutil and the DWARF linker. The
original purpose of this functionality was to leave a paper trail in the
output artifact about missing object files. The current implementation
however has diverged and is the source of a pretty serious bug:

- In the debug map parser, missing object files are not the only
warnings we emit. When paper trail warnings are enabled, all of them end
up in the dSYM which wasn't the goal.

- When warnings are associated with a object file in the debug map, it
is skipped by the DWARF linker. This only makes sense if the object file
is missing and is obviously incorrect for any other type of warning
(such as a missing symbol).

The combination of the two means that we can generate broken DWARF when
the feature is enabled. AFAIK it was only used by Apple and nobody I
spoke to has relied on it, so rather than fixing the broken behavior I
propose we remove it.

show more ...

[dsymutil] Skip duplicates files with identical time stamps in the debug map

Static archives can contain multiple files with the same file name, in
which case the timestamp is used to disambiguate.

[dsymutil] Skip duplicates files with identical time stamps in the debug map

Static archives can contain multiple files with the same file name, in
which case the timestamp is used to disambiguate. Because timestamps are
expressed in seconds since epoch timestamp collisions are far from
impossible. Furthermore, to facilitate reproducible builds, the static
linker can be told to emit no timestamps at all.

dsymutil already detects timestamp mismatches between the debug map and
the object files. However, it does not handle timestamp collisions
within the debug maps (STABS). Currently, we arbitrarily pick the first
debug map entry and ignore the rest. This is incorrect: if a symbol
exists in multiple object files, the linker might not have picked the
one from the first object file. This also results in missing symbol
warnings for all the symbols not defined in the first object file.

Given that in this scenario, dsymutil does not have enough information
to disambiguate, it should print a single informative warning and skip
the ambiguous debug map objects.

rdar://110374836

Differential revision: https://reviews.llvm.org/D152585

show more ...

[dsymutil] Initialize the debug map before loading the main binary

Fix a crash when a warning is emitted while loading the symbols from the
main binary. The warning helper assumes that the resulting

[dsymutil] Initialize the debug map before loading the main binary

Fix a crash when a warning is emitted while loading the symbols from the
main binary. The warning helper assumes that the resulting debug map is
initialized, but this happened after loading the main binary. Since
there's no dependency between the two the initialization can be moved
up.

rdar://107298776

show more ...

[YAML] Convert Optional to std::optional

[tools] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of m

[tools] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[dsymutil] Fix a few TODOs about reporting errors to the user

I saw the TODOs while reading this file and figured I'd do them.
I haven't seen these happen in practice.

No expected behavior change.

[dsymutil] Fix a few TODOs about reporting errors to the user

I saw the TODOs while reading this file and figured I'd do them.
I haven't seen these happen in practice.

No expected behavior change.

Differential Revision: https://reviews.llvm.org/D123215

show more ...

[dsymutil] Fix O(n^2) behavior when running on ld64.lld's current ICF output

STABS information consists of a list of records in the linked binary
that look like this:

OSO: path/to/some.o
SO: pa

[dsymutil] Fix O(n^2) behavior when running on ld64.lld's current ICF output

STABS information consists of a list of records in the linked binary
that look like this:

OSO: path/to/some.o
SO: path/to/some.c
FUN: sym1
FUN: sym2
...

The linked binary has one such set of records for every .o file linked
into it.

When dsymutil processes the binary's STABS information, it:

1. Reads the .o file mentioned in the OSO line
2. For each FUN entry after it in the main executable's STABS info:
a) it looks up that symbol in the symbol of that .o file
b) if it doesn't find it there, it goes through all symbols in the
main binary at the same address and sees if any of those match

With ICF, ld64.lld's STABS output claims that all identical functions
that were folded are in the .o file of the one that's deemed the
canonical one. Many small functions might be folded into a single
function, so there are .o OSO entries that end up with many FUN lines,
but almost none of them exist in the .o file's symbol table.

Previously, dsymutil would do a full scan of all symbols in the main
executable _for every of these entries_.

This patch instead scans all aliases once and remembers them per name.
This reduces the alias resolution complexity from
O(number_of_aliases_in_o_file * number_of_symbols_in_main_executable) to
O(number_of_aliases_in_o_file * log(number_of_aliases_in_o_file)).

In practice, it reduces the time spent to run dsymutil on
Chromium Framework from 26 min (after https://reviews.llvm.org/D89444)
or 12 min (before https://reviews.llvm.org/D89444) to ~8m30s.

We probably want to change how ld64.lld writes STABS entries when ICF
is enabled, but making dsymutil not have pathological performance for
this input seems like a good change as well.

No expected behavior change (other than it's faster). I verified that
for Chromium Framework, the generated .dSYM is identical with and
without this patch.

Differential Revision: https://reviews.llvm.org/D123218

show more ...

[tools] Delete redundant 'static' from namespace scope 'static const'. NFC

[dsymutil] Only look for ThinLTO suffixes if we failed to find symbol.

Only look for symbols with the ThinLTO suffix if we fail to find the
symbol.

[dsymutil] Fix spurious warnings for missing symbols with thinLTO

Fix spurious warnings for missing symbols with thinLTO. The latter
appends a unique suffix to avoid collisions for exported private

[dsymutil] Fix spurious warnings for missing symbols with thinLTO

Fix spurious warnings for missing symbols with thinLTO. The latter
appends a unique suffix to avoid collisions for exported private
symbols, resulting in dsymutil complaining it couldn't find the symbol
in the object file.

rdar://75434058

Differential revision: https://reviews.llvm.org/D99125

show more ...

[dsymutil] Generalize handling of aliases external symbols

In f9fb9da36c34d2745b74dc30e6c26f7d3f48466c we fixed spurious warnings
caused by aliases to private extern symbols. This patch generalizes

[dsymutil] Generalize handling of aliases external symbols

In f9fb9da36c34d2745b74dc30e6c26f7d3f48466c we fixed spurious warnings
caused by aliases to private extern symbols. This patch generalizes that
to regular external symbols as well.

show more ...

[dsymutil] Fix handling of aliases to private external symbols

dsymutil was incorrectly ignoring aliases to private extern symbols in
the MachODebugMapParser. This resulted in spurious warnings abou

[dsymutil] Fix handling of aliases to private external symbols

dsymutil was incorrectly ignoring aliases to private extern symbols in
the MachODebugMapParser. This resulted in spurious warnings about not
being able to find symbols.

rdar://49652389

Differential revision: https://reviews.llvm.org/D89444

show more ...

[dsymutil] Thread the VFS through dsymutil (NFC)

This patch threads the virtual file system through dsymutil.

Currently there is no good way to find out exactly what files are
necessary in order to

[dsymutil] Thread the VFS through dsymutil (NFC)

This patch threads the virtual file system through dsymutil.

Currently there is no good way to find out exactly what files are
necessary in order to reproduce a dsymutil link, at least not without
knowledge of how dsymutil's internals. My motivation for this change is
to add lightweight "reproducers" that automatically gather the input
object files through the FileCollectorFileSystem. The files together
with the YAML mapping will allow us to transparently reproduce a
dsymutil link, even without having to mess with the OSO path prefix.

Differential revision: https://reviews.llvm.org/D79376

show more ...

[Object] Change ObjectFile::getSymbolValue() return type to Expected<uint64_t>

Summary:
In D77860, we have changed `getSymbolFlags()` return type to `Expected<uint32_t>`.
This change helps bubble th

[Object] Change ObjectFile::getSymbolValue() return type to Expected<uint64_t>

Summary:
In D77860, we have changed `getSymbolFlags()` return type to `Expected<uint32_t>`.
This change helps bubble the error further up the stack.

Reviewers: jhenderson, grimar, JDevlieghere, MaskRay

Reviewed By: jhenderson

Subscribers: hiraditya, MaskRay, rupprecht, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D79075

show more ...

[Object] Change uint32_t getSymbolFlags() to Expected<uint32_t> getSymbolFlags().

This change enables getSymbolFlags() to return errors which benefit error reporting in clients.

Differential Revisi

[Object] Change uint32_t getSymbolFlags() to Expected<uint32_t> getSymbolFlags().

This change enables getSymbolFlags() to return errors which benefit error reporting in clients.

Differential Revision: https://reviews.llvm.org/D77860

show more ...