[GSYM] Callsites: Add data format support and loading from YAML (#109781)

This PR adds support in the gSYM format for call site information and
adds support for loading call sites from a YAML file.

[GSYM] Callsites: Add data format support and loading from YAML (#109781)

This PR adds support in the gSYM format for call site information and
adds support for loading call sites from a YAML file. The support for
YAML input is mostly for testing purposes - so we have a way to test the
functionality.

Note that this data is not currently used in the gSYM tooling - the
logic to use call sites will be added in a later PR.

The reason why we need call site information in gSYM files is so that we
can support better call stack function disambiguation in the case where
multiple functions have been merged due to optimization (linker ICF).
When resolving a merged function on the callstack, we can use the call
site information of the calling function to narrow down the actual
function that is being called, from the set of all merged functions.

See [this
RFC](https://discourse.llvm.org/t/rfc-extending-gsym-format-with-call-site-information-for-merged-function-disambiguation/80682)
for more details on this change.

show more ...

[llvm][unittests] Don't call raw_string_ostream::flush() (NFC)

raw_string_ostream::flush() is essentially a no-op (also specified in docs).
Don't call it in tests that aren't meant to test 'raw_stri

[llvm][unittests] Don't call raw_string_ostream::flush() (NFC)

raw_string_ostream::flush() is essentially a no-op (also specified in docs).
Don't call it in tests that aren't meant to test 'raw_string_ostream' itself.

p.s. remove a few redundant calls to raw_string_ostream::str()

show more ...

Gsymutil aggregation similar to DwarfDump --verify (#81154)

GsymUtil, like DwarfDump --verify, spews a *lot* of data necessary to
understand/diagnose issues with DWARF data. The trouble is that the

Gsymutil aggregation similar to DwarfDump --verify (#81154)

GsymUtil, like DwarfDump --verify, spews a *lot* of data necessary to
understand/diagnose issues with DWARF data. The trouble is that the kind
of information necessary to make the messages useful also makes them
nearly impossible to easily categorize. I put together a similar output
categorizer (https://github.com/llvm/llvm-project/pull/79648) that will
emit a summary of issues identified at the bottom of the (very verbose)
output, enabling easier tracking of issues as they arise or are
addressed.

There's a single output change, where a message "warning: Unable to
retrieve DWO .debug_info section for some object files. (Remove the
--quiet flag for full output)" was being dumped the first time it was
encountered (in what looks like an attempt to make something easily
grep-able), but rather than keep the output in the same order, that
message is now a 'category' so gets emitted at the end of the output.
The test 'tools/llvm-gsymutil/X86/elf-dwo.yaml' was changed to reflect
this difference.

---------

Co-authored-by: Kevin Frei <freik@meta.com>

show more ...

Modify llvm-gsymutil lookups to handle overlapping ranges correctly. (#72350)

llvm-gsymutil allows address ranges to overlap. There was a bug where if
we had debug info for a function with a range

Modify llvm-gsymutil lookups to handle overlapping ranges correctly. (#72350)

llvm-gsymutil allows address ranges to overlap. There was a bug where if
we had debug info for a function with a range like [0x100-0x200) and a
symbol at the same start address yet with a larger range like
[0x100-0x300), we would randomly get either only information from the
first or second entry. This could cause lookups to fail due to the way
the binary search worked.

This patch makes sure that when lookups happen we find the first address
table entry that can match an address, and also ensures that we always
select the first FunctionInfo that could match. FunctionInfo entries are
sorted such that the most debug info rich entries come first. And if we
have two ranges that have the same start address, the smaller range
comes first and the larger one comes next. This patch also adds the
ability to iterate over all function infos with the same start address
to always find a range that contains the address.

Added a unit test to test this functionality that failed prior to this
fix and now succeeds.

Also fix an issue when dumping an entire GSYM file that has duplicate address entries where it used to always print out the binary search match for the FunctionInfo, not the actual data for the address index.

show more ...

Modify llvm-gsymutil to ignore invalid file indexes (#71431)

DWARF produced by LTO and BOLT can sometimes be broken where file
indexes are beyond the end of the line table's file list in the
prolo

Modify llvm-gsymutil to ignore invalid file indexes (#71431)

DWARF produced by LTO and BOLT can sometimes be broken where file
indexes are beyond the end of the line table's file list in the
prologue. This patch allows llvm-gsymutil to convert this DWARF without
crashing, and emits errors when:

line table contains entries with an invalid file index (line entry will
be removed) inline functions that have invalid DW_AT_call_file file
indexes when there are no line table entries for a function and we fall
back to making a single line table entry from the functions
DW_AT_decl_file/DW_AT_decl_line attributes, we make sure the
DW_AT_decl_file attribute is valid before emitting it.

show more ...

Remove extra "git" word that made it into the GSYMTest.cpp file.

Fix line table lookups in line tables with multiple lines with the sa… (#70879)

Fix line table lookups in line tables with multiple lines with the same
address.

Compilers emit line tables that h

Fix line table lookups in line tables with multiple lines with the sa… (#70879)

Fix line table lookups in line tables with multiple lines with the same
address.

Compilers emit line tables that have multiple line table entries with
the same address. When doing lookups, we always need to use the last
line entry if a lookup address matches the address of one or more line
entries. This is because the size of an address range for a line uses
the next line entry to figure out how big the current line entry is. If
the next line entry has the same address, that means the current line
entry has a size of zero, so no bytes correspond to the line entry.

This patch ensures that lookups will always pick the last matching line
entry when the lookup address matches more than one line entry.

show more ...

[llvm] Stop including llvm/Support/Endian.h (NFC)

Identified with misc-include-cleaner.

llvm-gsymutil now handles empty linkage names correctly. (#68931)

Previous to this fix, if we had a DW_TAG_subprogram that had a
DW_AT_linkage_name that was empty, it would attempt to use this name

llvm-gsymutil now handles empty linkage names correctly. (#68931)

Previous to this fix, if we had a DW_TAG_subprogram that had a
DW_AT_linkage_name that was empty, it would attempt to use this name
which would cause an error to be emitted when saving the gsym file to
disk:

error: DWARF conversion failed: : attempted to encode invalid
FunctionInfo object

This patch fixes this issue and adds a unit test case.

show more ...

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an enum.
This patch replaces llvm:

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an enum.
This patch replaces llvm::support::{big,little,native} with
llvm::endianness::{big,little,native}.

show more ...

Use llvm::endianness (NFC)

Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
support::endianness with llvm::endianness.

Use llvm::endianness{,::little,::native} (NFC)

Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
llvm::support::endianness wi

Use llvm::endianness{,::little,::native} (NFC)

Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
llvm::support::endianness with llvm::endianness.

show more ...

[Support] Deprecate system_endianness (#68279)

system_endianness() just returns llvm::endianness::native, a
compile-time constant equivalent to std::native in C++20. This patch
deprecates system_

[Support] Deprecate system_endianness (#68279)

system_endianness() just returns llvm::endianness::native, a
compile-time constant equivalent to std::native in C++20. This patch
deprecates system_endianness() while replacing all invocations of
system_endianness() with llvm::endianness::native.

While we are at it, this patch replaces
llvm::support::endianness::{big,little} with
llvm::endianness::{big,little} in those statements that happen to call
system_endianness(). It does not go out of its way to replace other
occurrences of llvm::support::endianness::{big,little}.

show more ...

Fix spurious errors that would be emitted when DW_TAG_subprogram DIEs had mutliple ranges in DW_AT_ranges.

llvm-gsymutil would emit errors about address ranges for DW_TAG_inlined_subroutine DIEs who

Fix spurious errors that would be emitted when DW_TAG_subprogram DIEs had mutliple ranges in DW_AT_ranges.

llvm-gsymutil would emit errors about address ranges for DW_TAG_inlined_subroutine DIEs whose address range didn't exist in the parent inline information. When a DW_TAG_subprogram DIE has more than one address range with a DW_AT_ranges attribute, we emit multiple FunctionInfo objets, one for each range of a function. When we parsed the inline information, it might have inline contribution that appear in any of the function's ranges, and if we were parsing the first range of a function, all inline entries that appeared in other valid ranges of the functions would end up emitting error messages. This patch fixes this by always passing down the full list of ranges, even if they aren't being used in the parse of the information. This eliminates reporting of errors when we shouldn't have been emitting error messages. Added a test to track this and ensure this doesn't regress.

Also we don't warn if we end up with empty inline information if the only top level inline function have been elided where the high and low PC values are the same which indicates that the inline function was elided.

Differential Revision: https://reviews.llvm.org/D157669

show more ...

Improve llvm-gsymutil performance in quiet mode.

The GSYM code alwasy logging to streams even in quiet mode. When in quiet mode we would use the "nulls()" stream to avoid logging to the terminal, bu

Improve llvm-gsymutil performance in quiet mode.

The GSYM code alwasy logging to streams even in quiet mode. When in quiet mode we would use the "nulls()" stream to avoid logging to the terminal, but this still caused logging functions to be called on DWARFDie objects and other messages which were quite expensive and not needed if we weren't logging anything. This patch switches some logs in performant areas to be "raw_ostream *" values and if the ostream pointer is NULL, then we don't call the expensive logging functions on DWARFDie and other objects which will improve performance.

Differential Revision: https://reviews.llvm.org/D157466

show more ...

Increase performance of llvm-gsymutil by up to 200%.

llvm-gsymutil was maintaining an address ranges collection behind a mutex and having the multi-threaded code access this and hold the mutex was c

Increase performance of llvm-gsymutil by up to 200%.

llvm-gsymutil was maintaining an address ranges collection behind a mutex and having the multi-threaded code access this and hold the mutex was causing slowdown when converting DWARF to GSYM. This patch does the following:
- removes the "Ranges" variable from the GsymCreator and any functions and places that used it
- clients don't try to detect if a function has been added for an address range, we now remove any inferior copies of information in the GsymCreator::finalize() routine as was done before, we just have more items to remove, though performance is greator due to less mutex thread locking
- after I started adding all of the inferior funtion info objects the previous patch that tried to remove infrior debug info had bugs in it, so I replace the removeIfBinary() function in GsymCreator with a more efficient and easier to debug way to do things which copies items from the GsymCreator::Funcs into a new vector of FunctionInfo objects and then replaces GsymCreator::Funcs at the end.
- Sorting of FunctionInfo objects has been modified to also compare InlineInfo objects. We found cases where LTO was ruining inline function address ranges and we ended up with a variety of FunctionInfo objects for the same range that had varying amounts of valid debug info. This patch now ensure that two function info objects with different inline info for the same function address range, the best one will be picked to ensure the greatest fidelity.
- If we detect that a DW_TAG_subprogram has inline functions and after parsing it, we don't end up with any valid inline information, we set the optional to std::nullopt to avoid emitting empty inline information and wasting space.

My tests show a 200% perf increase on M1 macs and a 100% performance increase on linux machines for the same complex large DWARF input binary.

Differential Revision: https://reviews.llvm.org/D156773

show more ...

Fixed an issue where llvm-gsymutil would crash when parsing bad inline ranges.

If a function contains inline function ranges whose address ranges are not contained in the parent scope, then emit an

Fixed an issue where llvm-gsymutil would crash when parsing bad inline ranges.

If a function contains inline function ranges whose address ranges are not contained in the parent scope, then emit an error message and omit them from the final GSYM. Prior to this we would only test if an inline function's address range was within the concrete function's ranges. If we ran into a case where the inline range was within the function's ranges, but not within one of the parent inline function's ranges, then we would fail to produce a GSYM file and exit with an error.

The current code will emit full details on invalid inline ranges as they are being parsed and will omit any bad ranges from the final GSYM file.

Differential Revision: https://reviews.llvm.org/D155254

show more ...

Add the ability to segment GSYM files.

Some workflows can generate large GSYM files and sharding GSYM files into segments can help some performant workflows that can take advantage of smaller GSYM f

Add the ability to segment GSYM files.

Some workflows can generate large GSYM files and sharding GSYM files into segments can help some performant workflows that can take advantage of smaller GSYM files. This patch add a new --segment-size option to llvm-gsymutil. This option can specify a rough size in bytes of how large each segment should be.

Segmented GSYM files contain only the strings and files that are needed for the FunctionInfo objects that are added to each shard. The output file path gets the first address of the first contained function info appended as a suffix to the filename. If a base address of an image is set in the GsymCreator, then all segments will use this same base address which allows lookups for symbolication to happen correctly when the image has been slid in memory.

Code has been addeed to refactor and re-use methods within the GsymCreator to allow for segments to be created easily and tested.

Example of segmenting GSYM files:

$ llvm-gsymutil --convert llvm-gsymutil.dSYM -o llvm-gsymutil.gsym --segment-size 10485760
$ ls -l llvm-gsymutil.gsym-*
-rw-r--r-- 1 gclayton staff 10485839 Feb 9 10:45 llvm-gsymutil.gsym-0x1000030c0
-rw-r--r-- 1 gclayton staff 10485765 Feb 9 10:45 llvm-gsymutil.gsym-0x100668888
-rw-r--r-- 1 gclayton staff 10485881 Feb 9 10:45 llvm-gsymutil.gsym-0x100c948b8
-rw-r--r-- 1 gclayton staff 10485954 Feb 9 10:45 llvm-gsymutil.gsym-0x101659e70
-rw-r--r-- 1 gclayton staff 10485792 Feb 9 10:45 llvm-gsymutil.gsym-0x1022b1dc0
-rw-r--r-- 1 gclayton staff 10485889 Feb 9 10:45 llvm-gsymutil.gsym-0x102a18b10
-rw-r--r-- 1 gclayton staff 10485893 Feb 9 10:45 llvm-gsymutil.gsym-0x1030b05d0
-rw-r--r-- 1 gclayton staff 10485802 Feb 9 10:45 llvm-gsymutil.gsym-0x1037caaac
-rw-r--r-- 1 gclayton staff 10485781 Feb 9 10:45 llvm-gsymutil.gsym-0x103e767a0
-rw-r--r-- 1 gclayton staff 10485832 Feb 9 10:45 llvm-gsymutil.gsym-0x10452d0d4
-rw-r--r-- 1 gclayton staff 10485782 Feb 9 10:45 llvm-gsymutil.gsym-0x104b93310
-rw-r--r-- 1 gclayton staff 6255785 Feb 9 10:45 llvm-gsymutil.gsym-0x10526bf34

Differential Revision: https://reviews.llvm.org/D145448

show more ...

Revert "Add the ability to segment GSYM files."

This reverts commit fe758254181a824d73ad960b651b42f671f8936b.

This change was causing several buildbot failures:
- https://lab.llvm.org/buildbot/#/bu

Revert "Add the ability to segment GSYM files."

This reverts commit fe758254181a824d73ad960b651b42f671f8936b.

This change was causing several buildbot failures:
- https://lab.llvm.org/buildbot/#/builders/38/builds/10105
- https://lab.llvm.org/buildbot/#/builders/192/builds/562
- https://lab.llvm.org/buildbot/#/builders/109/builds/58893
- https://lab.llvm.org/buildbot/#/builders/16/builds/44360
- https://lab.llvm.org/buildbot/#/builders/247/builds/2095
- https://lab.llvm.org/buildbot/#/builders/196/builds/27236
- https://lab.llvm.org/buildbot/#/builders/54/builds/3714

show more ...

Add the ability to segment GSYM files.

Some workflows can generate large GSYM files and sharding GSYM files into segments can help some performant workflows that can take advantage of smaller GSYM f

Add the ability to segment GSYM files.

Some workflows can generate large GSYM files and sharding GSYM files into segments can help some performant workflows that can take advantage of smaller GSYM files. This patch add a new --segment-size option to llvm-gsymutil. This option can specify a rough size in bytes of how large each segment should be.

Segmented GSYM files contain only the strings and files that are needed for the FunctionInfo objects that are added to each shard. The output file path gets the first address of the first contained function info appended as a suffix to the filename. If a base address of an image is set in the GsymCreator, then all segments will use this same base address which allows lookups for symbolication to happen correctly when the image has been slid in memory.

Code has been addeed to refactor and re-use methods within the GsymCreator to allow for segments to be created easily and tested.

Example of segmenting GSYM files:

$ llvm-gsymutil --convert llvm-gsymutil.dSYM -o llvm-gsymutil.gsym --segment-size 10485760
$ ls -l llvm-gsymutil.gsym-*
-rw-r--r-- 1 gclayton staff 10485839 Feb 9 10:45 llvm-gsymutil.gsym-0x1000030c0
-rw-r--r-- 1 gclayton staff 10485765 Feb 9 10:45 llvm-gsymutil.gsym-0x100668888
-rw-r--r-- 1 gclayton staff 10485881 Feb 9 10:45 llvm-gsymutil.gsym-0x100c948b8
-rw-r--r-- 1 gclayton staff 10485954 Feb 9 10:45 llvm-gsymutil.gsym-0x101659e70
-rw-r--r-- 1 gclayton staff 10485792 Feb 9 10:45 llvm-gsymutil.gsym-0x1022b1dc0
-rw-r--r-- 1 gclayton staff 10485889 Feb 9 10:45 llvm-gsymutil.gsym-0x102a18b10
-rw-r--r-- 1 gclayton staff 10485893 Feb 9 10:45 llvm-gsymutil.gsym-0x1030b05d0
-rw-r--r-- 1 gclayton staff 10485802 Feb 9 10:45 llvm-gsymutil.gsym-0x1037caaac
-rw-r--r-- 1 gclayton staff 10485781 Feb 9 10:45 llvm-gsymutil.gsym-0x103e767a0
-rw-r--r-- 1 gclayton staff 10485832 Feb 9 10:45 llvm-gsymutil.gsym-0x10452d0d4
-rw-r--r-- 1 gclayton staff 10485782 Feb 9 10:45 llvm-gsymutil.gsym-0x104b93310
-rw-r--r-- 1 gclayton staff 6255785 Feb 9 10:45 llvm-gsymutil.gsym-0x10526bf34

Differential Revision: https://reviews.llvm.org/D143793

show more ...

[llvm/unittests] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the am

[llvm/unittests] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[llvm] Use has_value instead of hasValue (NFC)

Revert "Don't use Optional::hasValue (NFC)"

This reverts commit aa8feeefd3ac6c78ee8f67bf033976fc7d68bc6d.

Don't use Optional::hasValue (NFC)

[llvm] Don't use Optional::{hasValue,getValue} (NFC)