Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change f

Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change focuses on the
option's prefixed names and the array of prefixes. These are present in
every option and the dominant source of dynamic relocations for PIE or
PIC users of LLVM and Clang tooling. In some cases, 100s or 1000s of
them for the Clang driver which has a huge number of options.

This PR addresses this by building a string table and a prefixes table
that can be referenced with indices rather than pointers that require
dynamic relocations. This removes almost 7k dynmaic relocations from the
`clang` binary, roughly 8% of the remaining dynmaic relocations outside
of vtables. For busy-boxing use cases where many different option tables
are linked into the same binary, the savings add up a bit more.

The string table is a straightforward mechanism, but the prefixes
required some subtlety. They are encoded in a Pascal-string fashion with
a size followed by a sequence of offsets. This works relatively well for
the small realistic prefixes arrays in use.

Lots of code has to change in order to land this though: both all the
option library code has to be updated to use the string table and
prefixes table, and all the users of the options library have to be
updated to correctly instantiate the objects.

Some follow-up patches in the works to provide an abstraction for this
style of code, and to start using the same technique for some of the
other strings here now that the infrastructure is in place.

show more ...

Rename llvm::ThreadPool -> llvm::DefaultThreadPool (NFC) (#83702)

The base class llvm::ThreadPoolInterface will be renamed
llvm::ThreadPool in a subsequent commit.

This is a breaking change: cli

Rename llvm::ThreadPool -> llvm::DefaultThreadPool (NFC) (#83702)

The base class llvm::ThreadPoolInterface will be renamed
llvm::ThreadPool in a subsequent commit.

This is a breaking change: clients who use to create a ThreadPool must
now create a DefaultThreadPool instead.

show more ...

[llvm-driver] Fix usage of `InitLLVM` on Windows (#76306)

Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld

[llvm-driver] Fix usage of `InitLLVM` on Windows (#76306)

Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld
-flavor`, would not longer work on Windows, when these tools were linked
as part of `llvm-driver`. This was caused by `InitLLVM` which was part
of the `*_main()` function of these tools, which in turn calls
`windows::GetCommandLineArguments`. That function completly replaces
argc/argv by new UTF-8 contents, so any ajustements to argc/argv made by
`llvm-driver` prior to calling these tools was reset.

`InitLLVM` is now called by the `llvm-driver`. Any tool that
participates in (or is part of) the `llvm-driver` doesn't call
`InitLLVM` anymore.

show more ...

[llvm-{debuginfod,ml,objdump,symbolizer}, dsymutil] Enable multicall driver

Differential Revision: https://reviews.llvm.org/D157670

Revert "[llvm-{debuginfod,ml,objdump,symbolizer}, dsymutil] Enable multicall driver"

This reverts commit 2628fa3351b021d2ab82dcd833a14d7b52840a01 since
it broke the multicall driver build.

[llvm-{debuginfod,ml,objdump,symbolizer}, dsymutil] Enable multicall driver

Differential Revision: https://reviews.llvm.org/D157670

Reapply "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

This reverts commit 4e3b89483a6922d3f48670bb1c50a37f342918c6, with
fixes for places I'd missed updating in lld and lld

Reapply "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

This reverts commit 4e3b89483a6922d3f48670bb1c50a37f342918c6, with
fixes for places I'd missed updating in lld and lldb. I've also
renamed OptionVisibility::Default to "DefaultVis" to avoid ambiguity
since the undecorated name has to be available anywhere Options.inc is
included.

Original message follows:

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and adds
variants of the OptTable APIs that use Visibility mask instead of
Include/Exclude flags.

We need to do this to clean up a bunch of complexity in the clang
driver's option handling - there's a whole slew of flags like
CoreOption, NoDriverOption, and FlangOnlyOption there today to try to
handle all of the permutations of flags that the various drivers need,
but it really doesn't scale well, as can be seen by things like the
somewhat recently introduced CLDXCOption.

Instead, we'll provide an additive model for visibility that's
separate from the other flags. For things like "HelpHidden", which is
used as a "subtractive" modifier for option visibility, we leave that
in "Flags" and handle it as a special case.

Note that we don't actually update the users of the Include/Exclude
APIs here or change the flags that exist in clang at all - that will
come in a follow up that refactors clang's Options.td to use the
increased flexibility this change allows.

Differential Revision: https://reviews.llvm.org/D157149

show more ...

Revert "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

this is failing on bots, reverting to investigate.

This reverts commit a16104e6da6f36f3d72dbf53d10ba56495a0d65a.

[Option] Add "Visibility" field and clone the OptTable APIs to use it

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and a

[Option] Add "Visibility" field and clone the OptTable APIs to use it

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and adds
variants of the OptTable APIs that use Visibility mask instead of
Include/Exclude flags.

We need to do this to clean up a bunch of complexity in the clang
driver's option handling - there's a whole slew of flags like
CoreOption, NoDriverOption, and FlangOnlyOption there today to try to
handle all of the permutations of flags that the various drivers need,
but it really doesn't scale well, as can be seen by things like the
somewhat recently introduced CLDXCOption.

Instead, we'll provide an additive model for visibility that's
separate from the other flags. For things like "HelpHidden", which is
used as a "subtractive" modifier for option visibility, we leave that
in "Flags" and handle it as a special case.

Note that we don't actually update the users of the Include/Exclude
APIs here or change the flags that exist in clang at all - that will
come in a follow up that refactors clang's Options.td to use the
increased flexibility this change allows.

Differential Revision: https://reviews.llvm.org/D157149

show more ...

[llvm] Extract common `OptTable` bits into macros

All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OP

[llvm] Extract common `OptTable` bits into macros

All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OPT_##ID`), kind (`Option::KIND##Class`), group ID (`OPT_##GROUP`) and alias ID (`OPT_##ALIAS`). This patch extracts that common code into canonical macros. This results in fewer changes when tweaking the `OPTION` macros emitted by the TableGen backend.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D157028

show more ...

[llvm-debuginfod] Include llvm/Support/StringExtras.h after D155178

To fix undefined errors like to_float. This tool is often not built as
LLVM_ENABLE_HTTPLIB defaults to off (and the external depen

[llvm-debuginfod] Include llvm/Support/StringExtras.h after D155178

To fix undefined errors like to_float. This tool is often not built as
LLVM_ENABLE_HTTPLIB defaults to off (and the external dependency
cpp-httplib is difficult to set up due to a dependency on brotli) and
LLVM_TOOL_LLVM_DEBUGINFOD_BUILD disabling logic in D147185.

show more ...

[llvm-debuginfod][NFC] Switch to OptTable

Reviewed By: mysterymath

Differential Revision: https://reviews.llvm.org/D151273

Revert "[llvm-debuginfod][NFC] Switch to OptTable"

This reverts commit 1610627d2b42c0daf8cc20649c50ddad222f001b.

[llvm-debuginfod][NFC] Switch to OptTable

Reviewed By: mysterymath

Differential Revision: https://reviews.llvm.org/D151273

[llvm] Modernize bool literals (NFC)

Identified with modernize-use-bool-literals.

[llvm] [Debuginfod] LLVM debuginfod server.

This implements a debuginfod server in llvm using the `DebuginfodCollection` and `DebuginfodServer` classes. This is tested with lit tests against the deb

[llvm] [Debuginfod] LLVM debuginfod server.

This implements a debuginfod server in llvm using the `DebuginfodCollection` and `DebuginfodServer` classes. This is tested with lit tests against the debuginfod-find client.

The server scans 0 or more local directories for artifacts. It serves the debuginfod protocol over HTTP. Only the `executable` and `debuginfo` endpoints are supported (no `/source` endpoint).
The server also uses the debuginfod client as a fallback, so it can hit the local debuginfod cache or federate to other known debuginfod servers.
The client behavior is controllable through the standard environment variables (`DEBUGINFOD_URLS`, `DEBUGINFOD_CACHE_PATH`, `DEBUGINFOD_TIMEOUT`)

The server implements on-demand collection updates as follows:
If the build-id is not found by a local lookup, rescan immediately and look up the build-id again before returning 404. To protect against DoS attacks, do not rescan more frequently than once per N seconds (specified by `-m`).

Lit tests are provided which test the `llvm-debuginfod-find` client against the `llvm-debuginfod` server.

Reviewed By: mysterymath

Differential Revision: https://reviews.llvm.org/D114846

show more ...