Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change f

Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change focuses on the
option's prefixed names and the array of prefixes. These are present in
every option and the dominant source of dynamic relocations for PIE or
PIC users of LLVM and Clang tooling. In some cases, 100s or 1000s of
them for the Clang driver which has a huge number of options.

This PR addresses this by building a string table and a prefixes table
that can be referenced with indices rather than pointers that require
dynamic relocations. This removes almost 7k dynmaic relocations from the
`clang` binary, roughly 8% of the remaining dynmaic relocations outside
of vtables. For busy-boxing use cases where many different option tables
are linked into the same binary, the savings add up a bit more.

The string table is a straightforward mechanism, but the prefixes
required some subtlety. They are encoded in a Pascal-string fashion with
a size followed by a sequence of offsets. This works relatively well for
the small realistic prefixes arrays in use.

Lots of code has to change in order to land this though: both all the
option library code has to be updated to use the string table and
prefixes table, and all the users of the options library have to be
updated to correctly instantiate the objects.

Some follow-up patches in the works to provide an abstraction for this
style of code, and to start using the same technique for some of the
other strings here now that the infrastructure is in place.

show more ...

[MC] Remove unused bool arguments from createMCObjectStreamer callers

Give a warning when no dwo files are provided (#94336)

In some scenarios based on the split-dwarf build process, the dwo file
is not generated as expected(That is to say, no dwo file path is stored

Give a warning when no dwo files are provided (#94336)

In some scenarios based on the split-dwarf build process, the dwo file
is not generated as expected(That is to say, no dwo file path is stored
in the binary). When the llvm-dwp tool is called to generate the .dwp
file, it will exit without any warning.

So, the plan is to prompt a warning to tell the user that the dwo file
was not actually generated.
<img width="699" alt="image"
src="https://github.com/llvm/llvm-project/assets/150100070/5e5742f6-daad-450f-87e9-cb25449c3c7a">

show more ...

[llvm-dwp] Remove incorrect std::move()

DWOName is still used afterwards. The only reason this works out
right now is that SmallString does not actually have a constructor
that can take advantage of

[llvm-dwp] Remove incorrect std::move()

DWOName is still used afterwards. The only reason this works out
right now is that SmallString does not actually have a constructor
that can take advantage of the move.

show more ...

[llvm-driver] Fix usage of `InitLLVM` on Windows (#76306)

Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld

[llvm-driver] Fix usage of `InitLLVM` on Windows (#76306)

Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld
-flavor`, would not longer work on Windows, when these tools were linked
as part of `llvm-driver`. This was caused by `InitLLVM` which was part
of the `*_main()` function of these tools, which in turn calls
`windows::GetCommandLineArguments`. That function completly replaces
argc/argv by new UTF-8 contents, so any ajustements to argc/argv made by
`llvm-driver` prior to calling these tools was reset.

`InitLLVM` is now called by the `llvm-driver`. Any tool that
participates in (or is part of) the `llvm-driver` doesn't call
`InitLLVM` anymore.

show more ...

[DWP] Fix default for continue-on-cu-index-overflow (#75540)

This is follow up for https://github.com/llvm/llvm-project/pull/71902.
The
default option --continue-on-cu-index-overflow returned an e

[DWP] Fix default for continue-on-cu-index-overflow (#75540)

This is follow up for https://github.com/llvm/llvm-project/pull/71902.
The
default option --continue-on-cu-index-overflow returned an error
--continue-on-cu-index-overflow: missing argument. Changed it so that it
is the
same behavior as other flags like -gsplit-dwarf. Where
--continue-on-cu-index-overflow will default to continue, and user can
set mode
with --continue-on-cu-index-overflow=\<value>.

show more ...

Support soft failure on DWP section overflow, producing a truncated but valid DWP(#71902)

When 'ContinueOnCuIndexOverflow' enables without this modification, the
forcibly generated '.dwp' won't be

Support soft failure on DWP section overflow, producing a truncated but valid DWP(#71902)

When 'ContinueOnCuIndexOverflow' enables without this modification, the
forcibly generated '.dwp' won't be recognized by Debugger(gdb 10.1)
correctly.
<img width="657" alt="image"
src="https://github.com/llvm/llvm-project/assets/150100070/31732775-2548-453a-a47a-fa04c6d05fe9">
it looks like there's a problem with processing the dwarf header, which
makes debugging completely impossible. (unless the consumer walks the debug_info section to rebuild that column (if that's the only section that overflowed - if it's another section, there's no recovery))

**About patch:**
When llvm-dwp enables option '--continue-on-cu-index-overflow=soft-stop'
and recognizes the overflow problem , it will stop packing and generate
the '.dwp' file at once, discarding any DWO files that would not fit
within the 32 bit/4GB limits of the format.
<img width="625" alt="image"
src="https://github.com/llvm/llvm-project/assets/150100070/77d6be24-262b-4f4c-afc0-9a6c49e133c7">

show more ...

Reapply "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

This reverts commit 4e3b89483a6922d3f48670bb1c50a37f342918c6, with
fixes for places I'd missed updating in lld and lld

Reapply "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

This reverts commit 4e3b89483a6922d3f48670bb1c50a37f342918c6, with
fixes for places I'd missed updating in lld and lldb. I've also
renamed OptionVisibility::Default to "DefaultVis" to avoid ambiguity
since the undecorated name has to be available anywhere Options.inc is
included.

Original message follows:

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and adds
variants of the OptTable APIs that use Visibility mask instead of
Include/Exclude flags.

We need to do this to clean up a bunch of complexity in the clang
driver's option handling - there's a whole slew of flags like
CoreOption, NoDriverOption, and FlangOnlyOption there today to try to
handle all of the permutations of flags that the various drivers need,
but it really doesn't scale well, as can be seen by things like the
somewhat recently introduced CLDXCOption.

Instead, we'll provide an additive model for visibility that's
separate from the other flags. For things like "HelpHidden", which is
used as a "subtractive" modifier for option visibility, we leave that
in "Flags" and handle it as a special case.

Note that we don't actually update the users of the Include/Exclude
APIs here or change the flags that exist in clang at all - that will
come in a follow up that refactors clang's Options.td to use the
increased flexibility this change allows.

Differential Revision: https://reviews.llvm.org/D157149

show more ...

Revert "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

this is failing on bots, reverting to investigate.

This reverts commit a16104e6da6f36f3d72dbf53d10ba56495a0d65a.

[Option] Add "Visibility" field and clone the OptTable APIs to use it

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and a

[Option] Add "Visibility" field and clone the OptTable APIs to use it

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and adds
variants of the OptTable APIs that use Visibility mask instead of
Include/Exclude flags.

We need to do this to clean up a bunch of complexity in the clang
driver's option handling - there's a whole slew of flags like
CoreOption, NoDriverOption, and FlangOnlyOption there today to try to
handle all of the permutations of flags that the various drivers need,
but it really doesn't scale well, as can be seen by things like the
somewhat recently introduced CLDXCOption.

Instead, we'll provide an additive model for visibility that's
separate from the other flags. For things like "HelpHidden", which is
used as a "subtractive" modifier for option visibility, we leave that
in "Flags" and handle it as a special case.

Note that we don't actually update the users of the Include/Exclude
APIs here or change the flags that exist in clang at all - that will
come in a follow up that refactors clang's Options.td to use the
increased flexibility this change allows.

Differential Revision: https://reviews.llvm.org/D157149

show more ...

[llvm] Extract common `OptTable` bits into macros

All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OP

[llvm] Extract common `OptTable` bits into macros

All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OPT_##ID`), kind (`Option::KIND##Class`), group ID (`OPT_##GROUP`) and alias ID (`OPT_##ALIAS`). This patch extracts that common code into canonical macros. This results in fewer changes when tweaking the `OPTION` macros emitted by the TableGen backend.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D157028

show more ...

[dwp][libtool-darwin][sancov] Enable llvm-driver

Enable llvm-driver for:
llvm-dwp
llvm-libtoo-darwin
sancov

Differential Revision: https://reviews.llvm.org/D156758

[NFC][llvm-dwp] Switch from llvm::cl to OptTable

Switch the parse of command line options from llvm::cl to OptTable.

The motivation for this change is to continue adding llvm based tools
to the llv

[NFC][llvm-dwp] Switch from llvm::cl to OptTable

Switch the parse of command line options from llvm::cl to OptTable.

The motivation for this change is to continue adding llvm based tools
to the llvm driver multicall. For more information about the proposal
and motivation, please see https://discourse.llvm.org/t/rfc-llvm-busybox-proposal/58494

Reviewed By: abrachet

Differential Revision: https://reviews.llvm.org/D154642

show more ...

[DWP] add overflow check for llvm-dwp tools if offset overflow

Now, if the offset overflow happens, we just silently ignore it.
We will generate a bad dwp file, which will crash the gdb or make
it u

[DWP] add overflow check for llvm-dwp tools if offset overflow

Now, if the offset overflow happens, we just silently ignore it.
We will generate a bad dwp file, which will crash the gdb or make
it undefined behavior, and hard to address the root cause. So, we
need to produce some messages if overflow happens.

Reviewed By: ayermolo, dblaikie, steven.zhang

Differential Revision: https://reviews.llvm.org/D144565

show more ...

[llvm-dwp] Use std::optional in llvm-dwp.cpp (NFC)

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getva

[llvm-dwp] Use std::optional in llvm-dwp.cpp (NFC)

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

Don't use Optional::getPointer (NFC)

Since std::optional does not offer getPointer(), this patch replaces
X.getPointer() with &*X to make the migration from llvm::Optional to
std::optional easier.

Don't use Optional::getPointer (NFC)

Since std::optional does not offer getPointer(), this patch replaces
X.getPointer() with &*X to make the migration from llvm::Optional to
std::optional easier.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

Differential Revision: https://reviews.llvm.org/D138466

show more ...

[llvm-dwp] Report the filename if it cannot be found

For now, we report nothing if the execution/dwo file is missing, which is confusing.

Reviewed By: dblaikie

Differential Revision: https://revie

[llvm-dwp] Report the filename if it cannot be found

For now, we report nothing if the execution/dwo file is missing, which is confusing.

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D133549

show more ...

Revert "[llvm-dwp] Report the filename if it cannot be found"

This reverts commit 35028d417bb32bc05294e77c9ddcf50f24f277eb.
Breaks tests on Windows, see https://reviews.llvm.org/D133549#3785952

[llvm-dwp] Get the DWO file from relative path if the absolute path is not valid

Extend the llvm-dwp to support searching the DWOs that from relative path for the
case that build from remote buildin

[llvm-dwp] Get the DWO file from relative path if the absolute path is not valid

Extend the llvm-dwp to support searching the DWOs that from relative path for the
case that build from remote building system(different comp_dir).

Reviewd By: dblaikie

Differential Revision: https://reviews.llvm.org/D133480

show more ...

[llvm-dwp] Report the filename if it cannot be found

For now, we report nothing if the execution/dwo file is missing, which is confusing.

Reviewed By: dblaikie

Differential Revision: https://revie

[llvm-dwp] Report the filename if it cannot be found

For now, we report nothing if the execution/dwo file is missing, which is confusing.

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D133549

show more ...

[MC] De-capitalize MCStreamer functions

Follow-up to c031378ce01b8485ba0ef486654bc9393c4ac024 .
The class is mostly consistent now.

Remove unneeded cl::ZeroOrMore for cl::opt/cl::list options

Remove unneeded cl::ZeroOrMore for cl::list options

[NFC][MC] remove unused argument `MCRegisterInfo` in `MCCodeEmitter`

Reviewed By: skan

Differential Revision: https://reviews.llvm.org/D119846

Revert "[NFC][MC] remove unused argument `MCRegisterInfo` in `MCCodeEmitter`"

This reverts commit fe25c06cc5bdc2ef9427309f8ec1434aad69dc7a.