Revert "[lldb] Switch debuginfod cache to use lldb index cache settings" (#122816)

This reverts commit 7b808e73aa0193c8a42eae8f2420a803f424bee1.

Previous commit which change default debuginfod ca

Revert "[lldb] Switch debuginfod cache to use lldb index cache settings" (#122816)

This reverts commit 7b808e73aa0193c8a42eae8f2420a803f424bee1.

Previous commit which change default debuginfod cache path and pruning
policy settings is problematic. It broke multiple tests across lldb and
llvm. Reverting for now.

Co-authored-by: George Hu <georgehuyubo@gmail.com>

show more ...

Debuginfod cache use index cache settings and include real file name (#120814)

This PR include two changes:
1. Change debuginfod cache file name to include origin file name, the
new file name woul

Debuginfod cache use index cache settings and include real file name (#120814)

This PR include two changes:
1. Change debuginfod cache file name to include origin file name, the
new file name would be something like:

llvmcache-13267c5f5d2e3df472c133c8efa45fb3331ef1ea-liblzma.so.5.2.2.debuginfo.dwp
So it will provide more information in image list instead of a plain
llvmcache-123
2. Switch debuginfod cache to use lldb index cache settings. Currently
we don't have proper settings for setting the cache path or the cache
expiration time for debuginfod cache. We want to use the lldb index
cache settings, as they make sense to be in the same place and have the
same TTL.

---------

Co-authored-by: George Hu <georgehuyubo@gmail.com>

show more ...

Split the llvm::ThreadPool into an abstract base class and an implementation (#82094)

This decouples the public API used to enqueue tasks and wait for
completion from the actual implementation, and

Split the llvm::ThreadPool into an abstract base class and an implementation (#82094)

This decouples the public API used to enqueue tasks and wait for
completion from the actual implementation, and opens up the possibility
for clients to set their own thread pool implementation for the pool.

https://discourse.llvm.org/t/construct-threadpool-from-vector-of-existing-threads/76883

show more ...

Rename `ThreadPool::getThreadCount()` to `getMaxConcurrency()` (NFC) (#82296)

This is addressing a long-time TODO to rename this misleading API. The
old one is preserved for now but marked deprecat

Rename `ThreadPool::getThreadCount()` to `getMaxConcurrency()` (NFC) (#82296)

This is addressing a long-time TODO to rename this misleading API. The
old one is preserved for now but marked deprecated.

show more ...

Added settings for DEBUGINFOD cache location and timeout (#78605)

I've been working on more/better configuration for improving DEBUGINFOD
support. This is the first (and easiest) slice of the work.

Added settings for DEBUGINFOD cache location and timeout (#78605)

I've been working on more/better configuration for improving DEBUGINFOD
support. This is the first (and easiest) slice of the work.

I've added `timeout` and `cache-path` settings that can override the
DEBUGINFOD library defaults (and environment variables.) I also renamed
the `plugin.symbol-locator.debuginfod.server_urls` setting to
`server-urls` to be more consistent with the rest of LLDB's settings
(the underscore switch is switched to a hyphen)

I've got a few tests that validate the cache-path setting (as a
side-effect), but they've exposed a few bugs that I'll be putting up a
separate PR for (which will include the tests).

---------

Co-authored-by: Kevin Frei <freik@meta.com>

show more ...

Switch from the std::shared_mutex to an LLVM RWMutex (#74383)

@nico pointed out that my usage of `std::shared_mutex` broke builds on
older macOS devices. Switching to `llvm::sys::RWMutex` is the so

Switch from the std::shared_mutex to an LLVM RWMutex (#74383)

@nico pointed out that my usage of `std::shared_mutex` broke builds on
older macOS devices. Switching to `llvm::sys::RWMutex` is the solution
that they provided.

Tracked in issue #74382

Co-authored-by: Kevin Frei <freik@meta.com>

show more ...

DEBUGINFOD based DWP acquisition for LLDB (#70996)

I've plumbed the LLVM DebugInfoD client into LLDB, and added automatic
downloading of DWP files to the SymbolFileDWARF.cpp plugin. If you have
DE

DEBUGINFOD based DWP acquisition for LLDB (#70996)

I've plumbed the LLVM DebugInfoD client into LLDB, and added automatic
downloading of DWP files to the SymbolFileDWARF.cpp plugin. If you have
DEBUGINFOD_URLS set to a space delimited set of web servers, LLDB will
try to use them as a last resort when searching for DWP files. If you do
*not* have that environment variable set, nothing should be changed.
There's also a setting, per @clayborg 's suggestion, that will override
the environment variable, or can be used instead of the environment
variable. The setting is why I also needed to add an API to the
llvm-debuginfod library

### Test Plan:

Suggestions are welcome here. I should probably have some positive and
negative tests, but I wanted to get the diff up for people who have a
clue what they're doing to rip it to pieces before spending too much
time validating the initial implementation.

---------

Co-authored-by: Kevin Frei <freik@meta.com>
Co-authored-by: Alex Langford <nirvashtzero@gmail.com>

show more ...

[llvm-debuginfod] Switch to xxh3_64bits

Many sources show that xxh3 is much better than xxh64. This particular
instance may or may not have noticeable difference, but this change
moves us toward rem

[llvm-debuginfod] Switch to xxh3_64bits

Many sources show that xxh3 is much better than xxh64. This particular
instance may or may not have noticeable difference, but this change
moves us toward removing xxHash64.

This change will invalid cache.

Reviewed By: mysterymath

Differential Revision: https://reviews.llvm.org/D155993

show more ...

[Object] Refactor build ID parsing into Object lib.

This makes parsing for build IDs in the markup filter slightly more
permissive, in line with fromHex.

It also removes the distinction between mis

[Object] Refactor build ID parsing into Object lib.

This makes parsing for build IDs in the markup filter slightly more
permissive, in line with fromHex.

It also removes the distinction between missing build ID and empty build
ID; empty build IDs aren't a useful concept, since their purpose is to
uniquely identify a binary. This removes a layer of indirection wherever
build IDs are obtained.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D147485

show more ...

[Debuginfod] Prune cache after fetch.

Previously, the size of the debuginfod cache would grow without bound.
This change prunes the cache after a successful debuginfod lookup, as is
done in libdebug

[Debuginfod] Prune cache after fetch.

Previously, the size of the debuginfod cache would grow without bound.
This change prunes the cache after a successful debuginfod lookup, as is
done in libdebuginfod.

The cache pruning behavior is configured by a new
DEBUGINFOD_CACHE_POLICY environment variable. The semantics of this are
the same as --thinlto_cache_policy.

Reviewed By: gulfem

Differential Revision: https://reviews.llvm.org/D145618

show more ...

Reland: [llvm-cov] Look up object files using debuginfod

Reviewed By: gulfem

Differential Revision: https://reviews.llvm.org/D136702

Revert "[llvm-cov] Look up object files using debuginfod"

This reverts commit efbc8bb18eda63007216ad0cb5a8de04963eddd5.

This change is causing failures when detecting curl on several build bots:
-

Revert "[llvm-cov] Look up object files using debuginfod"

This reverts commit efbc8bb18eda63007216ad0cb5a8de04963eddd5.

This change is causing failures when detecting curl on several build bots:
- https://lab.llvm.org/buildbot/#/builders/247/builds/884
- https://lab.llvm.org/buildbot/#/builders/231/builds/7688
- https://lab.llvm.org/buildbot/#/builders/121/builds/27389
- https://lab.llvm.org/buildbot/#/builders/230/builds/8464
- https://lab.llvm.org/buildbot/#/builders/57/builds/24209
- https://lab.llvm.org/buildbot/#/builders/127/builds/42722

show more ...

[llvm-cov] Look up object files using debuginfod

Reviewed By: gulfem

Differential Revision: https://reviews.llvm.org/D136702

[llvm] Use std::optional instead of Optional

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-ge

[llvm] Use std::optional instead of Optional

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[llvm] llvm::Optional::value => operator*/operator->

std::optional::value() has undesired exception checking semantics and is
unavailable in some older Xcode. The call sites block std::optional migr

[llvm] llvm::Optional::value => operator*/operator->

std::optional::value() has undesired exception checking semantics and is
unavailable in some older Xcode. The call sites block std::optional migration.

show more ...

std::optional::value => operator*/operator->

value() has undesired exception checking semantics and calls
__throw_bad_optional_access in libc++. Moreover, the API is unavailable without
_LIBCPP_NO_E

std::optional::value => operator*/operator->

value() has undesired exception checking semantics and calls
__throw_bad_optional_access in libc++. Moreover, the API is unavailable without
_LIBCPP_NO_EXCEPTIONS on older Mach-O platforms (see
_LIBCPP_AVAILABILITY_BAD_OPTIONAL_ACCESS).

This fixes check-llvm.

show more ...

[Object] llvm::Optional => std::optional

[llvm] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of ma

[llvm] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

Reland "[LTO][COFF] Use bitcode file names in lto native object file names."

This reverts commit 34108082947c964ae9bbfcd9808f2fd31c0d672f with fixes.

Revert "Reland "[LTO][COFF] Use bitcode file names in lto native object file names.""

Breaks build of LLVMgold here:
```
/repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no

Revert "Reland "[LTO][COFF] Use bitcode file names in lto native object file names.""

Breaks build of LLVMgold here:
```
/repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no matching function for call to 'localCache'
Cache = check(localCache("ThinLTO", "Thin", options::cache_dir, AddBuffer));
^~~~~~~~~~
/repositories/llvm-project/llvm/include/llvm/Support/Caching.h:72:21: note: candidate function not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1102:20)' to 'llvm::AddBufferFn' (aka 'function<void (unsigned int, const llvm::Twine &, std::unique_ptr<MemoryBuffer>)>') for 4th argument
Expected<FileCache> localCache(
^
/repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1110:18: error: no viable conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
check(Lto->run(AddStream, Cache));
^~~~~~~~~
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:375:7: note: candidate constructor not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' for 1st argument
function(nullptr_t) noexcept
^
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:386:7: note: candidate constructor not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream>> (unsigned int, const llvm::Twine &)> &' for 1st argument
function(const function& __x)
^
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:404:7: note: candidate constructor not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
function(function&& __x) noexcept
^
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:435:2: note: candidate template ignored: requirement '_Callable<(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &, (lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), std::__invoke_result<(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &, unsigned int, const llvm::Twine &>>::value' was not satisfied [with _Functor = (lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &]
function(_Functor&& __f)
^
/repositories/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
^

```

This reverts commit 387620aa8cea33174b6c1fb80c1af713fee732ac.

show more ...

Reland "[LTO][COFF] Use bitcode file names in lto native object file names."

This reverts commit eef5405f74ae208e3e2eb7daacecac923d7338f2.

Revert "[LTO][COFF] Use bitcode file names in lto native object file names."

This reverts commit 531ed6d5aa65f41c6dfe2e74905d5c6c88fc95a7.

[LTO][COFF] Use bitcode file names in lto native object file names.

Currently the lto native object files have names like main.exe.lto.1.obj. In
PDB, those names are used as names for each compiland

[LTO][COFF] Use bitcode file names in lto native object file names.

Currently the lto native object files have names like main.exe.lto.1.obj. In
PDB, those names are used as names for each compiland. Microsoft’s tool
SizeBench uses those names to present to users the size of each object files.
So, names like main.exe.lto.1.obj is not user friendly.

This patch makes the lto native object file names more readable by using
the bitcode file names as part of the file names. For example, if the input
bitcode file has path like "path/to/foo.obj", its corresponding lto native
object file path would be "path/to/main.exe.lto.foo.obj". Since the lto native
object file name only bothers PDB, this patch only changes the lld-linker's
behavior.

Reviewed By: tejohnson, MaskRay, #lld-macho

Differential Revision: https://reviews.llvm.org/D137217

show more ...

[llvm-debuginfod-find] Fix test/behavior on Windows.

[Debuginfod] DEBUGINFOD_HEADERS_FILE environment variable

This change adds a DEBUGINFOD_HEADERS_FILE environment variable provides
a file containing HTTP headers to attach to outgoing HTTP requests,

[Debuginfod] DEBUGINFOD_HEADERS_FILE environment variable

This change adds a DEBUGINFOD_HEADERS_FILE environment variable provides
a file containing HTTP headers to attach to outgoing HTTP requests, one
per line. This allows a file permissioned with OS access control
mechanisms to supply bearer credentials for Debuginfod requests.

This matches the mechanism recently added to elfutils' libdebuginfod.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D136303

show more ...