[nfc][thinlto] remove unnecessary return from `renameModuleForThinLTO` (#121851)

Same goes for `FunctionImportGlobalProcessing::run`.

The return value was used, but it was always `false`.

[SystemZ][z/OS] Open text files in text mode (#109972)

This patch continues the work that was started here
https://reviews.llvm.org/D99426 to correctly open text files in text
mode.

[LTO] Reduce memory usage for import lists (#106772)

This patch reduces the memory usage for import lists by employing
memory-efficient data structures.

With this patch, an import list for a giv

[LTO] Reduce memory usage for import lists (#106772)

This patch reduces the memory usage for import lists by employing
memory-efficient data structures.

With this patch, an import list for a given destination module is
basically DenseSet<uint32_t> with each element indexing into the
deduplication table containing tuples of:

{SourceModule, GUID, Definition/Declaration}

In one of our large applications, the peak memory usage goes down by
9.2% from 6.120GB to 5.555GB during the LTO indexing step.

This patch addresses several sources of space inefficiency associated
with std::unordered_map:

- std::unordered_map<GUID, ImportKind> takes up 16 bytes because of
padding even though ImportKind only carries one bit of information.

- std::unordered_map uses pointers to elements, both in the hash table
proper and for collision chains.

- We allocate an instance of std::unordered_map for each
{Destination Module, Source Module} pair for which we have at least
one import. Most import lists have less than 10 imports, so the
metadata like the size of std::unordered_map and the pointer to the
hash table costs a lot relative to the actual contents.

show more ...

[LTO] Turn ImportMapTy into a proper class (NFC) (#105748)

This patch turns type alias ImportMapTy into a proper class to provide
a more intuitive interface like:

ImportList.addDefinition(...)

[LTO] Turn ImportMapTy into a proper class (NFC) (#105748)

This patch turns type alias ImportMapTy into a proper class to provide
a more intuitive interface like:

ImportList.addDefinition(...)

as opposed to:

FunctionImporter::addDefinition(ImportList, ...)

Also, this patch requires all non-const accesses to go through
addDefinition, maybeAddDeclaration, and addGUID while providing const
accesses via:

const ImportMapTyImpl &getImportMap() const { return ImportMap; }

I realize ImportMapTy may not be the best name as a class (maybe OK as
a type alias). I am not renaming ImportMapTy in this patch at least
because there are 47 mentions of ImportMapTy under llvm/.

show more ...

[LTO] Introduce helper functions to add GUIDs to ImportList (NFC) (#105555)

The new helper functions make the intent clearer while hiding
implementation details, including how we handle previously

[LTO] Introduce helper functions to add GUIDs to ImportList (NFC) (#105555)

The new helper functions make the intent clearer while hiding
implementation details, including how we handle previously added
entries. Note that:

- If we are adding a GUID as a GlobalValueSummary::Definition, then we
override a previously added GlobalValueSummary::Declaration entry
for the same GUID.

- If we are adding a GUID as a GlobalValueSummary::Declaration, then a
previously added GlobalValueSummary::Definition entry for the same
GUID takes precedence, and no change is made.

show more ...

[llvm] Remove redundant calls to std::unique_ptr<T>::get (NFC) (#97778)

Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#95482)

Make `FunctionsToImportTy` an `unordered_map` rather than `DenseMap`.
Cr

Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#95482)

Make `FunctionsToImportTy` an `unordered_map` rather than `DenseMap`.
Credit goes to jvoung@ for the 'DenseMap -> unordered_map' change. This
is a reland of https://github.com/llvm/llvm-project/pull/92718

* `DenseMap` allocates space for a large number of key/value pairs and
wastes space when the number of elements are small.
* While init bucket size is zero [1], it quickly allocates buckets for 64 elements [2]
when the number of elements is small (for example, 3 or 4 elements). The programmer
manual [3] also mentions it could waste space.
* Experiments show `FunctionsToImportTy.size()` is smaller than 4 for
multiple binaries with high indexing ram usage. `unordered_map` grows
factor is at most 2 in llvm libc [4] for insert operations.

With this change, `ComputeCrossModuleImport` ram increase is smaller
than 0.5G on a couple of binaries with high indexing ram usage. A wider
range of (pre-release) tests pass.

[1] https://github.com/llvm/llvm-project/blob/ad79a14c9e5ec4a369eed4adf567c22cc029863f/llvm/include/llvm/ADT/DenseMap.h#L431-L432
[2] https://github.com/llvm/llvm-project/blob/ad79a14c9e5ec4a369eed4adf567c22cc029863f/llvm/include/llvm/ADT/DenseMap.h#L849
[3] https://llvm.org/docs/ProgrammersManual.html#llvm-adt-densemap-h
[4] https://github.com/llvm/llvm-project/blob/ad79a14c9e5ec4a369eed4adf567c22cc029863f/libcxx/include/__hash_table#L1525-L1526

**Original commit message**
The goal is to populate `declaration` import status if a new flag
`-import-declaration` is on.

* For in-process ThinLTO, the `declaration` status is visible to backend
`function-import` pass, so `FunctionImporter::importFunctions` should
read the import status and be no-op for declaration summaries.
Basically, the postlink pipeline is updated to keep its current behavior
(import definitions), but not updated to handle `declaration` summaries.
Two use cases ([better call-graph
sort](https://discourse.llvm.org/t/rfc-for-better-call-graph-sort-build-a-more-complete-call-graph-by-adding-more-indirect-call-edges/74029#support-cross-module-function-declaration-import-5)
or [cross-module
auto-init](https://github.com/llvm/llvm-project/pull/87597#discussion_r1556067195))
would use this bit differently.

* For distributed ThinLTO, the `declaration` status is not serialized to
bitcode. As discussed, https://github.com/llvm/llvm-project/pull/87600
will do this.

show more ...

Reapply#4 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reapplies commit c5aeca73 (and its followup commit 21396be8), which were
reverted due to missing functionality in

Reapply#4 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reapplies commit c5aeca73 (and its followup commit 21396be8), which were
reverted due to missing functionality in MLIR and Flang regarding printing
debug records. This has now been added in commit 08aa511, along with support
for printing debug records in flang.

This reverts commit 2dc2290860355dd2bac3b655eea895fe30fde257.

show more ...

Revert new debug info format commits:

"[Flang] Update test to not check for tail calls on debug intrinsics" &
"Reapply#3 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Re

Revert new debug info format commits:

"[Flang] Update test to not check for tail calls on debug intrinsics" &
"Reapply#3 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Recent updates to flang have added debug info generation via MLIR, a path
which currently does not support debug records. The patch that enables
debug records by default (and a small followup patch) are thus being
reverted until the MLIR path has been fixed.

This reverts commits:
21396be865b4640abf6afa0b05de6708a1a996e0
c5aeca732d1ff6769b0659efebd1cfb5f60487e4

show more ...

Reapply#3 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reapplies commit 91446e2, which was reverted due to a downstream error,
discussed on the pull request. The error c

Reapply#3 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reapplies commit 91446e2, which was reverted due to a downstream error,
discussed on the pull request. The error could not be reproduced
upstream, and cannot be reproduced downstream as-of current main, so
until the error can be confirmed to still exist this patch should
return.

This reverts commit 23f8fac745bdde70ed4f9c585d19c4913734f1b8.

show more ...

Revert "Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#92718) (#94503)

This reverts commit e33db249b53fb70dce62db3ebd82d42239bd

Revert "Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#92718) (#94503)

This reverts commit e33db249b53fb70dce62db3ebd82d42239bd1d9d.

The change from *set to *map increases memory usage, and caused indexing
OOM in some applications. Need to profile offline to bring the memory
usage down.

show more ...

Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#92718)

The original PR is reviewed in
https://github.com/llvm/llvm-project/pull

Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#92718)

The original PR is reviewed in
https://github.com/llvm/llvm-project/pull/88024, and this PR adds one
line (https://github.com/llvm/llvm-project/pull/92718/commits/b9f04d199dec4f3c221d981dcb91e55298d0693f)
to fix test

Limit to one thread for in-process ThinLTO to test `LLVM_DEBUG` log.
- This should fix build bot failure like
https://lab.llvm.org/buildbot/#/builders/259/builds/4727 and
https://lab.llvm.org/buildbot/#/builders/9/builds/43876
- I could repro the failure and see interleaved log messages by using
`-thinlto-threads=all`

**Original Commit Message:**

The goal is to populate `declaration` import status if a new flag
`-import-declaration` is on.

* For in-process ThinLTO, the `declaration` status is visible to backend
`function-import` pass, so `FunctionImporter::importFunctions` should
read the import status and be no-op for declaration summaries.
Basically, the postlink pipeline is updated to keep its current behavior
(import definitions), but not updated to handle `declaration` summaries.
Two use cases ([better call-graph
sort](https://discourse.llvm.org/t/rfc-for-better-call-graph-sort-build-a-more-complete-call-graph-by-adding-more-indirect-call-edges/74029#support-cross-module-function-declaration-import-5)
or [cross-module
auto-init](https://github.com/llvm/llvm-project/pull/87597#discussion_r1556067195))
would use this bit differently.

* For distributed ThinLTO, the `declaration` status is not serialized to
bitcode. As discussed, https://github.com/llvm/llvm-project/pull/87600
will do this.

show more ...

Revert "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#92715)

Reverts llvm/llvm-project#88024

Build bot failures
(https://lab.llvm.

Revert "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#92715)

Reverts llvm/llvm-project#88024

Build bot failures
(https://lab.llvm.org/buildbot/#/builders/259/builds/4727 and
https://lab.llvm.org/buildbot/#/builders/9/builds/43876)

show more ...

[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option (#88024)

The goal is to populate `declaration` import status if a new flag`-import-declarat

[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option (#88024)

The goal is to populate `declaration` import status if a new flag`-import-declaration` is on.

* For in-process ThinLTO, the `declaration` status is visible to backend
`function-import` pass, so `FunctionImporter::importFunctions` should
read the import status and be no-op for declaration summaries.
Basically, the postlink pipeline is updated to keep its current behavior
(import definitions), but not updated to handle `declaration` summaries.
Two use cases (better call-graph sort and cross-module auto-init)
would use this bit differently.

* For distributed ThinLTO, the `declaration` status is not serialized to
bitcode. As discussed, https://github.com/llvm/llvm-project/pull/87600
will do this.

[1] https://discourse.llvm.org/t/rfc-for-better-call-graph-sort-build-a-more-complete-call-graph-by-adding-more-indirect-call-edges/74029#support-cross-module-function-declaration-import-5
[2] https://github.com/llvm/llvm-project/pull/87597#discussion_r1556067195

show more ...

Revert "Repply#2 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)""

This reverts commit 91446e2aa687ec57ad88dc0df793d0c6e694a7c9 and
a unittest followup 1530f319311908b06fe93

Revert "Repply#2 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)""

This reverts commit 91446e2aa687ec57ad88dc0df793d0c6e694a7c9 and
a unittest followup 1530f319311908b06fe935c89fca692d3e53184f (#90476).

In a stage-2 -flto=thin -gsplit-dwarf -g -fdebug-info-for-profiling
-fprofile-sample-use= build of clang, a ThinLTO backend compile has
assertion failures:

Global is external, but doesn't have external or weak linkage!
ptr @_ZN5clang12ast_matchers8internal18makeAllOfCompositeINS_8QualTypeEEENS1_15BindableMatcherIT_EEN4llvm8ArrayRefIPKNS1_7MatcherIS5_EEEE
function declaration may only have a unique !dbg attachment
ptr @_ZN5clang12ast_matchers8internal18makeAllOfCompositeINS_8QualTypeEEENS1_15BindableMatcherIT_EEN4llvm8ArrayRefIPKNS1_7MatcherIS5_EEEE

The failures somehow go away if -fprofile-sample-use= is removed.

show more ...

Repply#2 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reapplies the original commit:
2f01fd99eb8c8ab3db9aba72c4f00e31e9e60a05

The previous application of this patch f

Repply#2 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reapplies the original commit:
2f01fd99eb8c8ab3db9aba72c4f00e31e9e60a05

The previous application of this patch failed due to some missing
DbgVariableRecord support in clang, which has been added now by commit
8805465e.

This will probably break some downstream tools that don't already handle
debug records. If your downstream code breaks as a result of this
change, the simplest fix is to convert the module in question to the old
debug format before you process it, using
`Module::convertFromNewDbgValues()`. For more information about how to
handle debug records or about what has changed, see the migration
document:
https://llvm.org/docs/RemoveDIsDebugInfo.html

This reverts commit 4fd319ae273ed6c252f2067909c1abd9f6d97efa.

show more ...

Revert#2 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reverted following probably-causing failures on some clang buildbots:
https://lab.llvm.org/buildbot/#/builders/24

Revert#2 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Reverted following probably-causing failures on some clang buildbots:
https://lab.llvm.org/buildbot/#/builders/245/builds/24037

This reverts commit a12622543de15df45fb9ad64e8ab723289d55169.

show more ...

Reapply "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Fixes the broken tests in the original commit:
2f01fd99eb8c8ab3db9aba72c4f00e31e9e60a05

This will probably break

Reapply "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

Fixes the broken tests in the original commit:
2f01fd99eb8c8ab3db9aba72c4f00e31e9e60a05

This will probably break some downstream tools that don't already handle
debug records. If your downstream code breaks as a result of this
change, the simplest fix is to convert the module in question to the old
debug format before you process it, using
`Module::convertFromNewDbgValues()`. For more information about how to
handle debug records or about what has changed, see the migration
document:
https://llvm.org/docs/RemoveDIsDebugInfo.html

This reverts commit 00821fed09969305b0003d3313c44d1e761a7131.

show more ...

Revert "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

A unit test was broken by the above commit:
https://lab.llvm.org/buildbot/#/builders/139/builds/64627

This reverts

Revert "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"

A unit test was broken by the above commit:
https://lab.llvm.org/buildbot/#/builders/139/builds/64627

This reverts commit 2f01fd99eb8c8ab3db9aba72c4f00e31e9e60a05.

show more ...

[RemoveDIs] Load into new debug info format by default in LLVM (#89799)

This patch enables parsing and creating modules directly into the new
debug info format. Prior to this patch, all modules wer

[RemoveDIs] Load into new debug info format by default in LLVM (#89799)

This patch enables parsing and creating modules directly into the new
debug info format. Prior to this patch, all modules were constructed
with the old debug info format by default, and would be converted into
the new format just before running LLVM passes. This is an important
milestone, in that this means that every tool will now be exposed to
debug records, rather than those that run LLVM passes. As far as I've
tested, all LLVM tools/projects now either handle debug records, or
convert them to the old intrinsic format.

There are a few unit tests that need updating for this patch; these are
either cases of tests that previously needed to set the debug info
format to function, or tests that depend on the old debug info format in
some way. There should be no visible change in the output of any LLVM
tool as a result of this patch, although the likelihood of this patch
breaking downstream code means an NFC tag might be a little misleading,
if not technically incorrect:

This will probably break some downstream tools that don't already handle
debug records. If your downstream code breaks as a result of this
change, the simplest fix is to convert the module in question to the old
debug format before you process it, using
`Module::convertFromNewDbgValues()`. For more information about how to
handle debug records or about what has changed, see the migration
document:
https://llvm.org/docs/RemoveDIsDebugInfo.html

show more ...

[RemoveDIs] Make verify-uselistorder preserve the input debug info format (#87789)

Verify-uselistorder wants to take some input IR and verify that the
uselist order is stable after roundtripping to

[RemoveDIs] Make verify-uselistorder preserve the input debug info format (#87789)

Verify-uselistorder wants to take some input IR and verify that the
uselist order is stable after roundtripping to bitcode and assembly.
This is disrupted if the file is converted between the new and old debug
info formats after parsing - while there's no functional difference, the
change to the in-memory representation of the IR modifies the uselist.
This patch changes verify-uselistorder to not convert input files
between debug info formats by default, preventing changes from being
made to the file being checked. In addition, this patch makes it so that
when we _do_ print IR in the new debug info format to bitcode or
assembly, we delete any lingering debug intrinsic declarations, ensuring
that we don't write uselist entries for them.

show more ...

[RemoveDIs] Add flag to preserve the debug info format of input IR (#87379)

This patch adds a new flag: `--preserve-input-debuginfo-format`

This flag instructs the tool to not convert the debug i

[RemoveDIs] Add flag to preserve the debug info format of input IR (#87379)

This patch adds a new flag: `--preserve-input-debuginfo-format`

This flag instructs the tool to not convert the debug info format
(intrinsics/records) of input IR, but to instead determine the format of
the input IR and overwrite the other format-determining flags so that we
process and output the file in the same format that we received it in.
This flag is turned off by llvm-link, llvm-lto, and llvm-lto2, and
should be turned off by any other tool that expects to parse multiple IR
modules and have their debug info formats match.

The motivation for this flag is to allow tools to not convert the debug
info format - verify-uselistorder and llvm-reduce, and any downstream
tools that seek to test or mutate IR as-is, without applying extraneous
modifications to the input. This is a necessary step to using debug
records by default in all (other) LLVM tools.

show more ...

[RemoveDIs] Load into new debug info format by default in llvm-link (#86274)

Directly load all bitcode into the new debug info format in llvm-link.
This means that new-mode bitcode no longer round-

[RemoveDIs] Load into new debug info format by default in llvm-link (#86274)

Directly load all bitcode into the new debug info format in llvm-link.
This means that new-mode bitcode no longer round-trips back to old-mode
after parsing, and that old-mode bitcode gets auto-upgraded to new-mode
debug info (which is the current in-memory default in LLVM).

show more ...

[llvm-link] Improve missing file error message (#82514)

Add error messages showing the missing filenames.

Currently, we only get 'No such file or directory' without any(!)
further info. This pat

[llvm-link] Improve missing file error message (#82514)

Add error messages showing the missing filenames.

Currently, we only get 'No such file or directory' without any(!)
further info. This patch will (only upon ENOENT error) iterate over all
requested files and print which ones are actually missing.

show more ...

[NFC][RemoveDIs] Remove conditional compilation for RemoveDIs (#81149)

A colleague observes that switching the default value of
LLVM_EXPERIMENTAL_DEBUGINFO_ITERATORS to "On" hasn't flipped the valu

[NFC][RemoveDIs] Remove conditional compilation for RemoveDIs (#81149)

A colleague observes that switching the default value of
LLVM_EXPERIMENTAL_DEBUGINFO_ITERATORS to "On" hasn't flipped the value
in their CMakeCache.txt. This probably means that everyone with an
existing build tree is going to not have support built in, meaning
everyone in LLVM would need to clean+rebuild their worktree when we flip
the switch on... which doesn't sound good.

So instead, just delete the flag and everything it does, making everyone
build and run ~400 lit tests in RemoveDIs mode. None of the buildbots
have had trouble with this, so it Should Be Fine (TM).

(Sending for review as this is changing various comments, and touches
several different areas -- I don't want to get too punchy).

show more ...