[BOLT][DWARF][NFC] Fix DebugStrOffsetsWriter (#100672)

Fix DebugStrOffsetsWriter so updateAddressMap can't be called after it
is finalized.

[BOLT][DWARF][NFC] Refactor address writers (#98094)

Refactors address writers to create an instance for each CU and its DWO
CU.

[BOLT][DWARF] Cleanup buffer initialization for DWO range writer (#97843)

Cleanup buffer initialization for DWO range writer instances to remove
empty buffer at the beginning.

[BOLT][DWARF][NFC] Remove unnecessary SectionOffset (#97841)

Removes unnecessary SectionOffset variable from DebugData.

[BOLT][DWARF] Refactor legacy ranges writers (#96006)

Refactors legacy ranges writers to create a writer for each instance of
a DWO file.

We now write out everything into .debug_ranges after the

[BOLT][DWARF] Refactor legacy ranges writers (#96006)

Refactors legacy ranges writers to create a writer for each instance of
a DWO file.

We now write out everything into .debug_ranges after the all the DWO
files are processed. This also changes the order that ranges is written
out in, as before we wrote out while in the main CU processing loop and
we now iterate through the CU buckets created by partitionCUs, after the
main processing loop.

show more ...

[BOLT][DWARF] Update DW_AT_comp_dir/DW_AT_dwo_name for DWO TUs (#91486)

Type unit DIE generated by clang contains DW_AT_comp_dir/DW_AT_dwo_name.
This was added to clang to help LLDB to figure out w

[BOLT][DWARF] Update DW_AT_comp_dir/DW_AT_dwo_name for DWO TUs (#91486)

Type unit DIE generated by clang contains DW_AT_comp_dir/DW_AT_dwo_name.
This was added to clang to help LLDB to figure out where type unit come
from when accessing an entry in a .debug_names accelerator table and
type units in .dwp file.

When BOLT writes out .dwo files it changes the name of them. User can
also specify directory of where they can be written out. Added support
to BOLT to update those attributes.

show more ...

[BOLT][NFC] Clean includes, add license headers (#87200)

[BOLT][DWARF][NFC] Expose DebugStrOffsetsWriter::clear (#82548)

Refactored cod that clears data-structures in DebugStrOffsetsWriter into
clear() function and made initialize() public. This is for

[BOLT][DWARF][NFC] Expose DebugStrOffsetsWriter::clear (#82548)

Refactored cod that clears data-structures in DebugStrOffsetsWriter into
clear() function and made initialize() public. This is for
https://github.com/llvm/llvm-project/pull/81062.

show more ...

[BOLT][DWARF] Fix out of order rangelists/loclists (#81645)

GCC can generate rangelists/loclists that are out of order. Fixed so
that we don't assert, and instead generate partially optimized list.

[BOLT][DWARF] Fix out of order rangelists/loclists (#81645)

GCC can generate rangelists/loclists that are out of order. Fixed so
that we don't assert, and instead generate partially optimized list.
Through most code paths we do sort rnglists/loclists, but not for
loclist for a path where BOLT does not modify a function. Although it's
nice to have lists sorted, this implementation shouldn't rely on it.
This also fixes an issue if we partially capture a list we would write
out *end_of_list in helper function. So tools won't see the rest of the
addresses being written out.

show more ...

[BOLT][NFC] Log through JournalingStreams (#81524)

Make core BOLT functionality more friendly to being used as a
library instead of in our standalone driver llvm-bolt. To
accomplish this, we augme

[BOLT][NFC] Log through JournalingStreams (#81524)

Make core BOLT functionality more friendly to being used as a
library instead of in our standalone driver llvm-bolt. To
accomplish this, we augment BinaryContext with journaling streams
that are to be used by most BOLT code whenever something needs to
be logged to the screen. Users of the library can decide if logs
should be printed to a file, no file or to the screen, as
before. To illustrate this, this patch adds a new option
`--log-file` that allows the user to redirect BOLT logging to a
file on disk or completely hide it by using
`--log-file=/dev/null`. Future BOLT code should now use
`BinaryContext::outs()` for printing important messages instead of
`llvm::outs()`. A new test log.test enforces this by verifying that
no strings are print to screen once the `--log-file` option is
used.

In previous patches we also added a new BOLTError class to report
common and fatal errors, so code shouldn't call exit(1) now. To
easily handle problems as before (by quitting with exit(1)),
callers can now use
`BinaryContext::logBOLTErrorsAndQuitOnFatal(Error)` whenever code
needs to deal with BOLT errors. To test this, we have fatal.s
that checks we are correctly quitting and printing a fatal error
to the screen.

Because this is a significant change by itself, not all code was
yet ported. Code from Profiler libs (DataAggregator and friends)
still print errors directly to screen.

Co-authored-by: Rafael Auler <rafaelauler@fb.com>

Test Plan: NFC

show more ...

[BOLT][DWARF] Fix handling .debug_str_offsets for type units (#75522)

There was an assumpiton that TUs and CUs share .debug_str_offsets
contribution. For ThinLTO builds it is not the case. Changed

[BOLT][DWARF] Fix handling .debug_str_offsets for type units (#75522)

There was an assumpiton that TUs and CUs share .debug_str_offsets
contribution. For ThinLTO builds it is not the case. Changed so that we
parse contributions for TUs also, and did some refactoring so that we
don't re-parse contributions that were not modified.

show more ...

[BOLT][DWARF] Fix handling of debug_str_offsets (#75100)

We were not setting size field of .debug_str_offsets correctly. Fixed
it, and added a test.

[BOLT] Fix typos (#68121)

Closes https://github.com/llvm/llvm-project/issues/63097

Before merging please make sure the change to
bolt/include/bolt/Passes/StokeInfo.h is correct.

bolt/include/

[BOLT] Fix typos (#68121)

Closes https://github.com/llvm/llvm-project/issues/63097

Before merging please make sure the change to
bolt/include/bolt/Passes/StokeInfo.h is correct.

bolt/include/bolt/Passes/StokeInfo.h

```diff
// This Pass solves the two major problems to use the Stoke program without
- // proting its code:
+ // probing its code:
```

I'm still not happy about the awkward wording in this comment.

bolt/include/bolt/Passes/FixRelaxationPass.h

```
$ ed -s bolt/include/bolt/Passes/FixRelaxationPass.h <<<'9,12p'
// This file declares the FixRelaxations class, which locates instructions with
// wrong targets and fixes them. Such problems usually occures when linker
// relaxes (changes) instructions, but doesn't fix relocations types properly
// for them.
$
```


bolt/docs/doxygen.cfg.in
bolt/include/bolt/Core/BinaryContext.h
bolt/include/bolt/Core/BinaryFunction.h
bolt/include/bolt/Core/BinarySection.h
bolt/include/bolt/Core/DebugData.h
bolt/include/bolt/Core/DynoStats.h
bolt/include/bolt/Core/Exceptions.h
bolt/include/bolt/Core/MCPlusBuilder.h
bolt/include/bolt/Core/Relocation.h
bolt/include/bolt/Passes/FixRelaxationPass.h
bolt/include/bolt/Passes/InstrumentationSummary.h
bolt/include/bolt/Passes/ReorderAlgorithm.h
bolt/include/bolt/Passes/StackReachingUses.h
bolt/include/bolt/Passes/StokeInfo.h
bolt/include/bolt/Passes/TailDuplication.h
bolt/include/bolt/Profile/DataAggregator.h
bolt/include/bolt/Profile/DataReader.h
bolt/lib/Core/BinaryContext.cpp
bolt/lib/Core/BinarySection.cpp
bolt/lib/Core/DebugData.cpp
bolt/lib/Core/DynoStats.cpp
bolt/lib/Core/Relocation.cpp
bolt/lib/Passes/Instrumentation.cpp
bolt/lib/Passes/JTFootprintReduction.cpp
bolt/lib/Passes/ReorderData.cpp
bolt/lib/Passes/RetpolineInsertion.cpp
bolt/lib/Passes/ShrinkWrapping.cpp
bolt/lib/Passes/TailDuplication.cpp
bolt/lib/Rewrite/BoltDiff.cpp
bolt/lib/Rewrite/DWARFRewriter.cpp
bolt/lib/Rewrite/RewriteInstance.cpp
bolt/lib/Utils/CommandLineOpts.cpp
bolt/runtime/instr.cpp
bolt/test/AArch64/got-ld64-relaxation.test
bolt/test/AArch64/unmarked-data.test
bolt/test/X86/Inputs/dwarf5-cu-no-debug-addr-helper.s
bolt/test/X86/Inputs/linenumber.cpp
bolt/test/X86/double-jump.test
bolt/test/X86/dwarf5-call-pc-function-null-check.test
bolt/test/X86/dwarf5-split-dwarf4-monolithic.test
bolt/test/X86/dynrelocs.s
bolt/test/X86/fallthrough-to-noop.test
bolt/test/X86/tail-duplication-cache.s
bolt/test/runtime/X86/instrumentation-ind-calls.s

show more ...

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an
enum. This patch replaces suppo

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an
enum. This patch replaces support::{big,little,native} with
llvm::endianness::{big,little,native}.

show more ...

[BOLT] Fix the initialization of DWARFDataExtractor

Without this patch, we pass Endian as one of the parameters to the
constructor of DWARFDataExtractor. The problem is that Endian is of:

enum e

[BOLT] Fix the initialization of DWARFDataExtractor

Without this patch, we pass Endian as one of the parameters to the
constructor of DWARFDataExtractor. The problem is that Endian is of:

enum endianness {big, little, native};

whereas the constructor is expecting "bool IsLittleEndian". That is,
we are relying on an implicit conversion to convert big and little to
false and true, respectively.

When we migrate llvm::support::endianness to std::endian in future, we
can no longer rely on an implicit conversion because std::endian is
declared with "enum class". Even if we could, the conversion would
not be guaranteed to work because, for example, libcxx defines:

enum class endian {
little = 0xDEAD,
big = 0xFACE,
:

where big and little are not boolean values.

This patch fixes the problem by properly converting Endian to a
boolean value.

show more ...

[BOLT][DWARF][NFC] Remove addIndexAddress

Removed unused API DebugAddrWriter::addIndexAddress.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D157357

[BOLT][DWARF] Fix performance regression running BOLT on binaries build with DWARF4

In one of the previous diffs LocBuffer was changed to pass by value. This lead to
performance regression running B

[BOLT][DWARF] Fix performance regression running BOLT on binaries build with DWARF4

In one of the previous diffs LocBuffer was changed to pass by value. This lead to
performance regression running BOLT on binaries with DWARF4 split dwarf.

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D155763

show more ...

[BOLT][DWARF] Change to process and write out TUs first then CUs in batches

To reduce memory footprint changed so that we process and write out TUs first,
reset DIEBuilder and process CUs. CUs are p

[BOLT][DWARF] Change to process and write out TUs first then CUs in batches

To reduce memory footprint changed so that we process and write out TUs first,
reset DIEBuilder and process CUs. CUs are processed in buckets. First bucket
contains all the CUs with cross CU references. Rest processd one at a time.

clang-17 build in debug mode, by clang-17.
before
8:25.81 real, 834.37 user, 86.03 sys, 0 amem, 79525064 mmem
8:02.20 real, 820.46 user, 81.81 sys, 0 amem, 79501616 mmem
7:52.69 real, 802.01 user, 83.99 sys, 0 amem, 79534392 mmem

after
7:49.35 real, 822.04 user, 66.19 sys, 0 amem, 34934260 mmem
7:42.16 real, 825.46 user, 63.52 sys, 0 amem, 34951660 mmem
7:46.71 real, 821.11 user, 63.14 sys, 0 amem, 34981164 mmem

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D151909

show more ...

[BOLT][DWARF] Numerous fixes for a new DWARFRewriter

* Some cleanup and minor fixes for the new debug information re-writer before moving on
to productatization.

* The new rewriter wasn't handling

[BOLT][DWARF] Numerous fixes for a new DWARFRewriter

* Some cleanup and minor fixes for the new debug information re-writer before moving on
to productatization.

* The new rewriter wasn't handling binary with DWARF5 and DWARF4 with
-fdebug-types-sections.

* Removed dead cross cu reference code.

* Added support for DW_AT_sibling.

* With the new re-writer abbrev number can change which can lead to offset of Type
Units changing. Before we would just copy raw data. Changed to write out Type
Unit List. This is generated by gdb-add-index.

* Fixed how bolt handles gdb-index generated by gdb-11 with types sections.
Simplified logic that handles variations of gdb-index.

* Clang can generate two type units with the same hash, but different content. LLD
does not de-duplicate when ThinLTO is involved. Changed so that TU hash and
offset are used to make TU's unique.

* It is possible to have references within location expression to another DIE.
Fixed it so that relative offset is updated correctly.

* Removed all the code related to patching.

* Removed dead code. Changed how we handling writting out TUs and TU Index. It now
should fully work for DWARF4 and DWARF5.

* Removed unused arguments from some APIs, changed return type to void, and other
small cleanups.

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D151906

show more ...

[BOLT][DWARF] Implement new mechanism for DWARFRewriter

This revision implement new mechanism for DWARFRewriter.
In the new mechanism, we adopt the same way with DWARFLinker did.
By parsing Debug in

[BOLT][DWARF] Implement new mechanism for DWARFRewriter

This revision implement new mechanism for DWARFRewriter.
In the new mechanism, we adopt the same way with DWARFLinker did.
By parsing Debug information into IR, we are allowed to handle debug information more flexible.
Now the debug information updating process relies on IR and IR will be written out to binary once the updating finished.

A new class was added: DIEBuilder. This class is responsible for parsing debug information and raising it to the IR level.
This class is also used to write out the .debug_info and .debug_abbrev sections.
Since we output brand new Abbrev section we won't need to always convert low_pc/high_pc into ranges.
When conversion does happen we can also remove low_pc entry.

Reviewed By: maksfb, ayermolo

Differential Revision: https://reviews.llvm.org/D130315

show more ...

[BOLT] Fix unused-variable warnings

This patch fixes:

bolt/lib/Core/DebugData.cpp:1669:20: error: unused variable
'StrOffset' [-Werror,-Wunused-variable]

bolt/lib/Core/DebugData.cpp:1676:18:

[BOLT] Fix unused-variable warnings

This patch fixes:

bolt/lib/Core/DebugData.cpp:1669:20: error: unused variable
'StrOffset' [-Werror,-Wunused-variable]

bolt/lib/Core/DebugData.cpp:1676:18: error: unused variable
'NewOffset' [-Werror,-Wunused-variable]

show more ...

Revert "[DWARF][BOLT] Implement new mechanism for DWARFRewriter"

This reverts commit 460a2244430fae192298a5fd9fa2a269e540e8c1.
It breaks building on macOS, and it was landed with a review URL
pointi

Revert "[DWARF][BOLT] Implement new mechanism for DWARFRewriter"

This reverts commit 460a2244430fae192298a5fd9fa2a269e540e8c1.
It breaks building on macOS, and it was landed with a review URL
pointing to some Facebook-internal service.

Also reverts a bunch of follow-ups:

Revert "[BOLT][DWARF] Don't check string offsets"
This reverts commit f9d6f48c8bf5acaac07502403c41cf0b0d89c8d2.

Revert "[BOLT][DWARF] Change to process and write out TUs first then CUs in batches"
This reverts commit 88e95c1e4bb6e2ad3bfd185b96341ad5c09eff6b.

Revert "[BOLT][DWARF] Output DWO files as they are being processed"
This reverts commit 46ca2e3fcd419b1246357ed3b9cd36630f16e64d.

Revert "[BOLT][DWARF] Don't check string offsets"
This reverts commit cfe4a4b04f219a9dbb4e3fc01883437b6ff0e702.

Revert "[BOLT][DWARF] Numerous fixes for a new DWARFRewriter"
This reverts commit 2701a661daa393ad5901ac88d420d7aa931eda0d.

show more ...

[BOLT][DWARF] Change to process and write out TUs first then CUs in batches

Summary:
To reduce memory footprint changed so that we process and write out TUs first,
reset DIEBuilder and process CUs.

[BOLT][DWARF] Change to process and write out TUs first then CUs in batches

Summary:
To reduce memory footprint changed so that we process and write out TUs first,
reset DIEBuilder and process CUs. CUs are processed in buckets. First bucket
contains all the CUs with cross CU references. Rest processd one at a time.

clang-17 build in debug mode, by clang-17.
before
8:25.81 real, 834.37 user, 86.03 sys, 0 amem, 79525064 mmem
8:02.20 real, 820.46 user, 81.81 sys, 0 amem, 79501616 mmem
7:52.69 real, 802.01 user, 83.99 sys, 0 amem, 79534392 mmem

after
7:49.35 real, 822.04 user, 66.19 sys, 0 amem, 34934260 mmem
7:42.16 real, 825.46 user, 63.52 sys, 0 amem, 34951660 mmem
7:46.71 real, 821.11 user, 63.14 sys, 0 amem, 34981164 mmem

Differential Revision: https://phabricator.intern.facebook.com/D45883198

show more ...

[BOLT][DWARF] Numerous fixes for a new DWARFRewriter

Summary:

* Some cleanup and minor fixes for the new debug information re-writer before moving on
to productatization.

* The new rewriter wasn't

[BOLT][DWARF] Numerous fixes for a new DWARFRewriter

Summary:

* Some cleanup and minor fixes for the new debug information re-writer before moving on
to productatization.

* The new rewriter wasn't handling binary with DWARF5 and DWARF4 with
-fdebug-types-sections.

* Removed dead cross cu reference code.

* Added support for DW_AT_sibling.

* With the new re-writer abbrev number can change which can lead to offset of Type
Units changing. Before we would just copy raw data. Changed to write out Type
Unit List. This is generated by gdb-add-index.

* Fixed how bolt handles gdb-index generated by gdb-11 with types sections.
Simplified logic that handles variations of gdb-index.

* Clang can generate two type units with the same hash, but different content. LLD
does not de-duplicate when ThinLTO is involved. Changed so that TU hash and
offset are used to make TU's unique.

* It is possible to have references within location expression to another DIE.
Fixed it so that relative offset is updated correctly.

* Removed all the code related to patching.

* Removed dead code. Changed how we handling writting out TUs and TU Index. It now
should fully work for DWARF4 and DWARF5.

* Removed unused arguments from some APIs, changed return type to void, and other
small cleanups.

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: https://phabricator.intern.facebook.com/D46168257

show more ...

[DWARF][BOLT] Implement new mechanism for DWARFRewriter

Summary:
This revision implement new mechanism for DWARFRewriter.
In the new mechanism, we adopt the same way with DWARFLinker did.
By parsing

[DWARF][BOLT] Implement new mechanism for DWARFRewriter

Summary:
This revision implement new mechanism for DWARFRewriter.
In the new mechanism, we adopt the same way with DWARFLinker did.
By parsing Debug information into IR, we are allowed to handle debug information more flexible.
Now the debug information updating process relies on IR and IR will be written out to binary once the updating finished.

A new class was added: DIEBuilder. This class is responsible for parsing debug information and raising it to the IR level.
This class is also used to write out the .debug_info and .debug_abbrev sections.
Since we output brand new Abbrev section we won't need to always convert low_pc/high_pc into ranges.
When conversion does happen we can also remove low_pc entry.

Differential Revision: https://phabricator.intern.facebook.com/D39484421

Tasks: T117448832

show more ...