Cleanup llvm/DebugInfo/PDB headers

accumulated preprocessed size:
before: 1065515095
after: 1065629059

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differenti

Cleanup llvm/DebugInfo/PDB headers

accumulated preprocessed size:
before: 1065515095
after: 1065629059

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120195

show more ...

[llvm] Use = default (NFC)

Re-land "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit bacf9cf2c5cdec3567580e5030c4c82f42b3d745 and
reinstates commit 1a9bd5b81328adf0dd5a8b4f3ad5949463e6

Re-land "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit bacf9cf2c5cdec3567580e5030c4c82f42b3d745 and
reinstates commit 1a9bd5b81328adf0dd5a8b4f3ad5949463e66da3.

Reverting this commit did not appear to make the problem go away, so we
can go ahead and reland it.

show more ...

Revert "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit 1a9bd5b81328adf0dd5a8b4f3ad5949463e66da3.

I suspect that this patch may have caused https://crbug.c

Revert "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit 1a9bd5b81328adf0dd5a8b4f3ad5949463e66da3.

I suspect that this patch may have caused https://crbug.com/1171438.

show more ...

Reland "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit 5b7aef6eb4b2930971029b984cb2360f7682e5a5 and relands
6529d7c5a45b1b9588e512013b02f891d71bc134.

The

Reland "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit 5b7aef6eb4b2930971029b984cb2360f7682e5a5 and relands
6529d7c5a45b1b9588e512013b02f891d71bc134.

The ASan error was debugged and determined to be the fault of an invalid
object file input in our test suite, which was fixed by my last change.
LLD's project policy is that it assumes input objects are valid, so I
have added a comment about this assumption to the relocation bounds
check.

show more ...

Revert "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit 6529d7c5a45b1b9588e512013b02f891d71bc134.

Reason: Broke the ASan buildbots.
http://lab.llvm.org:801

Revert "[PDB] Defer relocating .debug$S until commit time and parallelize it"

This reverts commit 6529d7c5a45b1b9588e512013b02f891d71bc134.

Reason: Broke the ASan buildbots.
http://lab.llvm.org:8011/#/builders/99/builds/1567

show more ...

[PDB] Defer relocating .debug$S until commit time and parallelize it

This is a pretty classic optimization. Instead of processing symbol
records and copying them to temporary storage, do a first pas

[PDB] Defer relocating .debug$S until commit time and parallelize it

This is a pretty classic optimization. Instead of processing symbol
records and copying them to temporary storage, do a first pass to
measure how large the module symbol stream will be, and then copy the
data into place in the PDB file. This requires defering relocation until
much later, which accounts for most of the complexity in this patch.

This patch avoids copying the contents of all live .debug$S sections
into heap memory, which is worth about 20% of private memory usage when
making PDBs. However, this is not an unmitigated performance win,
because it can be faster to read dense, temporary, heap data than it is
to iterate symbol records in object file backed memory a second time.

Results on release chrome.dll:
peak mem: 5164.89MB -> 4072.19MB (-1,092.7MB, -21.2%)
wall-j1: 0m30.844s -> 0m32.094s (slightly slower)
wall-j3: 0m20.968s -> 0m20.312s (slightly faster)
wall-j8: 0m19.062s -> 0m17.672s (meaningfully faster)

I gathered similar numbers for a debug, component build of content.dll
in Chrome, and the performance impact of this change was in the noise.
The memory usage reduction was visible and similar.

Because of the new parallelism in the PDB commit phase, more cores makes
the new approach faster. I'm assuming that most C++ developer machines
these days are at least quad core, so I think this is a win.

Differential Revision: https://reviews.llvm.org/D94267

show more ...

[PDB] Remove a couple asserts that are no longer valid now that C13Builders does not use unique_ptr.

These asserts used to check that unique_ptr was not null.

This fixes failures from 7af4bb16417de

[PDB] Remove a couple asserts that are no longer valid now that C13Builders does not use unique_ptr.

These asserts used to check that unique_ptr was not null.

This fixes failures from 7af4bb16417deeb1d01e7dbbbb2272f1f46753c6

show more ...

[PDB] Remove unique_ptr wrapper around C13 line table subsections

This accounts for a large portion of the memory allocations in LLD.
This DebugSubsectionRecordBuilder object can be stored directly

[PDB] Remove unique_ptr wrapper around C13 line table subsections

This accounts for a large portion of the memory allocations in LLD.
This DebugSubsectionRecordBuilder object can be stored directly in
C13Builders, it mostly wraps other subsections.

Remove the container kind field from the object. It is always the same
for all elements in the vector, and we can pass it in during writing.

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[llvm] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of

[llvm] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

llvm-svn: 369013

show more ...

[DebugInfo][PDB] Don't write empty debug streams

Before, empty debug streams were written as 8 bytes (4 bytes signature + 4 bytes for the GlobalRefs count).

With this patch, unused empty streams ar

[DebugInfo][PDB] Don't write empty debug streams

Before, empty debug streams were written as 8 bytes (4 bytes signature + 4 bytes for the GlobalRefs count).

With this patch, unused empty streams aren't emitted anymore. Modules now encode 65535 as an 'unused stream' value, by convention.
Also fix the * Linker * contrib section which wasn't correctly emitted previously.

Differential Revision: https://reviews.llvm.org/D59502

llvm-svn: 356395

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Add missing error checking code intended for r347687

llvm-svn: 347690

[PDB] Add symbol records in bulk

Summary:
This speeds up linking clang.exe/pdb with /DEBUG:GHASH by 31%, from
12.9s to 9.8s.

Symbol records are typically small (16.7 bytes on average), but we
proce

[PDB] Add symbol records in bulk

Summary:
This speeds up linking clang.exe/pdb with /DEBUG:GHASH by 31%, from
12.9s to 9.8s.

Symbol records are typically small (16.7 bytes on average), but we
processed them one at a time. CVSymbol is a relatively "large" type. It
wraps an ArrayRef<uint8_t> with a kind an optional 32-bit hash, which we
don't need. Before this change, each DbiModuleDescriptorBuilder would
maintain an array of CVSymbols, and would write them individually with a
BinaryItemStream.

With this change, we now add symbols that happen to appear contiguously
in bulk. For each .debug$S section (roughly one per function), we
allocate two copies, one for relocation, and one for realignment
purposes. For runs of symbols that go in the module stream, which is
most symbols, we now add them as a single ArrayRef<uint8_t>, so the
vector DbiModuleDescriptorBuilder is roughly linear in the number of
.debug$S sections (O(# funcs)) instead of the number of symbol records
(very large).

Some stats on symbol sizes for the curious:
PDB size: 507M
sym bytes: 316,508,016
sym count: 18,954,971
sym byte avg: 16.7

As future work, we may be able to skip copying symbol records in the
linker for realignment purposes if we make LLVM write them aligned into
the object file. We need to double check that such symbol records are
still compatible with link.exe, but if so, it's definitely worth doing,
since my profile shows we spend 500ms in memcpy in the symbol merging
code. We could potentially cut that in half by saving a copy.
Alternatively, we could apply the relocations *after* we iterate the
symbols. This would require some careful re-engineering of the
relocation processing code, though.

Reviewers: zturner, aganea, ruiu

Subscribers: hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D54554

llvm-svn: 347687

show more ...

[LLD/PDB] Emit first section contribution for DBI Module Descriptor.

Part of the DBI stream is a list of variable length structures
describing each module that contributes to the final executable.

[LLD/PDB] Emit first section contribution for DBI Module Descriptor.

Part of the DBI stream is a list of variable length structures
describing each module that contributes to the final executable.

One member of this structure is a section contribution entry that
describes the first section contribution in the output file for
the given module.

We have been leaving this structure unpopulated until now, so with
this patch it is now filled out correctly.

Differential Revision: https://reviews.llvm.org/D45832

llvm-svn: 330457

show more ...

[llvm-pdbutil] Dump first section contribution for each module.

The DBI stream contains a list of module descriptors. At the
beginning of each descriptor is a structure representing the first
secti

[llvm-pdbutil] Dump first section contribution for each module.

The DBI stream contains a list of module descriptors. At the
beginning of each descriptor is a structure representing the first
section contribution in the output file for that module. LLD
currently doesn't fill out this structure at all, but link.exe
does. So as a precursor to emitting this data in LLD, we first
need a way to dump it so that it can be checked.

This patch adds support for the dumping, and verifies via a test
that LLD emits bogus information.

llvm-svn: 330208

show more ...

Remove dead code. NFCI.

llvm-svn: 312740

[PDB] Merge Global and Publics Builders.

The publics stream and globals stream are very similar. They both
contain a list of hash buckets that refer into a single shared stream,
the symbol record st

[PDB] Merge Global and Publics Builders.

The publics stream and globals stream are very similar. They both
contain a list of hash buckets that refer into a single shared stream,
the symbol record stream. Because of the need for each builder to manage
both an independent hash stream as well as a single shared record
stream, making the two builders be independent entities is not the right
design. This patch merges them into a single class, of which only a
single instance is needed to create all 3 streams. PublicsStreamBuilder
and GlobalsStreamBuilder are now merged into the single GSIStreamBuilder
class, which writes all 3 streams at once.

Note that this patch does not contain any functionality change. So we're
still not yet writing any records to the globals stream. All we're doing
is making it so that when we do start writing records to the globals,
this refactor won't have to be part of that patch.

Differential Revision: https://reviews.llvm.org/D36489

llvm-svn: 310438

show more ...

[PDB] Write public symbol records and the publics hash table

Summary:
MSVC link.exe records all external symbol names in the publics stream.
It provides similar functionality to an ELF .symtab.

Rev

[PDB] Write public symbol records and the publics hash table

Summary:
MSVC link.exe records all external symbol names in the publics stream.
It provides similar functionality to an ELF .symtab.

Reviewers: zturner, ruiu

Subscribers: hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D35871

llvm-svn: 309303

show more ...

[PDB] Teach libpdb to write DBI Stream ECNames.

Based strictly on the name, this seems to have something to do
width edit & continue. The goal of this patch has nothing to do
with supporting edit a

[PDB] Teach libpdb to write DBI Stream ECNames.

Based strictly on the name, this seems to have something to do
width edit & continue. The goal of this patch has nothing to do
with supporting edit and continue though. msvc link.exe writes
very basic information into this area even when *not* compiling
with support for E&C, and so the goal here is to bring lld-link
to parity. Since we cannot know what assumptions standard tools
make about the content of PDB files, we need to be as close as
possible.

This ECNames data structure is a standard PDB string hash table.
link.exe puts a single string into this hash table, which is the
full path to the PDB file on disk. It then references this string
from the module descriptor for the compiler generated `* Linker *`
module.

With this patch, lld-link will generate the exact same sequence of
bytes as MSVC link for this subsection for a given object file
input (as reported by `llvm-pdbutil bytes -ec`).

llvm-svn: 307356

show more ...

[PDB] Don't write uninitialized bytes to a PDB file.

There were certain fields that we didn't know how to write, as
well as various padding bytes that we would ignore. This leads
to garbage data in

[PDB] Don't write uninitialized bytes to a PDB file.

There were certain fields that we didn't know how to write, as
well as various padding bytes that we would ignore. This leads
to garbage data in the PDB. While not strictly necessary, we
should initialize these bytes to something meaningful, as it
makes for easier binary comparison between PDBs.

llvm-svn: 305819

show more ...

[PDB] Start emitting source file and line information

Summary:
This is a first step towards getting line info to show up in VS and
windbg. So far, only llvm-pdbutil can parse the PDBs that we produc

[PDB] Start emitting source file and line information

Summary:
This is a first step towards getting line info to show up in VS and
windbg. So far, only llvm-pdbutil can parse the PDBs that we produce.
cvdump doesn't like something about our file checksum tables. I'll have
to dig into that next.

This patch adds a new DebugSubsectionRecordBuilder which takes bytes
directly from some other producer, such as a linker, and sticks it into
the PDB. Line tables only need to be relocated. No data needs to be
rewritten.

File checksums and string tables, on the other hand, need to be re-done.

Reviewers: zturner, ruiu

Subscribers: llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D34257

llvm-svn: 305713

show more ...

Resubmit "[codeview] Make obj2yaml/yaml2obj support .debug$S..."

This was originally reverted because of some non-deterministic
failures on certain buildbots. Luckily ASAN eventually caught
this as

Resubmit "[codeview] Make obj2yaml/yaml2obj support .debug$S..."

This was originally reverted because of some non-deterministic
failures on certain buildbots. Luckily ASAN eventually caught
this as a stack-use-after-scope, so the fix is included in
this patch.

llvm-svn: 305393

show more ...

Revert "[codeview] Make obj2yaml/yaml2obj support .debug$S..."

This is causing failures on linux bots with an invalid stream
read. It doesn't repro in any configuration on Windows, so
reverting unt

Revert "[codeview] Make obj2yaml/yaml2obj support .debug$S..."

This is causing failures on linux bots with an invalid stream
read. It doesn't repro in any configuration on Windows, so
reverting until I have a chance to investigate on Linux.

llvm-svn: 305371

show more ...