Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change f

Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change focuses on the
option's prefixed names and the array of prefixes. These are present in
every option and the dominant source of dynamic relocations for PIE or
PIC users of LLVM and Clang tooling. In some cases, 100s or 1000s of
them for the Clang driver which has a huge number of options.

This PR addresses this by building a string table and a prefixes table
that can be referenced with indices rather than pointers that require
dynamic relocations. This removes almost 7k dynmaic relocations from the
`clang` binary, roughly 8% of the remaining dynmaic relocations outside
of vtables. For busy-boxing use cases where many different option tables
are linked into the same binary, the savings add up a bit more.

The string table is a straightforward mechanism, but the prefixes
required some subtlety. They are encoded in a Pascal-string fashion with
a size followed by a sequence of offsets. This works relatively well for
the small realistic prefixes arrays in use.

Lots of code has to change in order to land this though: both all the
option library code has to be updated to use the string table and
prefixes table, and all the users of the options library have to be
updated to correctly instantiate the objects.

Some follow-up patches in the works to provide an abstraction for this
style of code, and to start using the same technique for some of the
other strings here now that the infrastructure is in place.

show more ...

[SystemZ][z/OS] Open text files in text mode (#109972)

This patch continues the work that was started here
https://reviews.llvm.org/D99426 to correctly open text files in text
mode.

[llvm-rc] Support ARM64EC resource generation (#78908)

This is already supported in llvm-cvtres, so only a small change is
needed.

[tools] Use SmallString::operator std::string (NFC)

[llvm-driver] Fix usage of `InitLLVM` on Windows (#76306)

Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld

[llvm-driver] Fix usage of `InitLLVM` on Windows (#76306)

Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld
-flavor`, would not longer work on Windows, when these tools were linked
as part of `llvm-driver`. This was caused by `InitLLVM` which was part
of the `*_main()` function of these tools, which in turn calls
`windows::GetCommandLineArguments`. That function completly replaces
argc/argv by new UTF-8 contents, so any ajustements to argc/argv made by
`llvm-driver` prior to calling these tools was reset.

`InitLLVM` is now called by the `llvm-driver`. Any tool that
participates in (or is part of) the `llvm-driver` doesn't call
`InitLLVM` anymore.

show more ...

[llvm-windres] Change the interpretation of --preprocessor to match Binutils 2.36 (#75391)

Binutils 2.36 had a somewhat controversial change in how the
--preprocessor option was handled in GNU wind

[llvm-windres] Change the interpretation of --preprocessor to match Binutils 2.36 (#75391)

Binutils 2.36 had a somewhat controversial change in how the
--preprocessor option was handled in GNU windres; previously, the option
was interpreted as a part of the command string, potentially containing
multiple arguments (which even was hinted at in the documentation).

In Binutils 2.36, this was changed to interpret the --preprocessor
argument as one argument (possibly containing spaces) pointing at the
preprocessor executable.

The existing behaviour where implicit arguments like -E -xc -DRC_INVOKED
are dropped if --preprocessor is specified, was kept.

This was a breaking change for some users of GNU windres, see
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=21c33bcbe36377abf01614fb1b9be439a3b6de20,
https://sourceware.org/bugzilla/show_bug.cgi?id=27594, and
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=5edb8e3f5ad8d74a83fc0df7f6e4514eed0aa77f.

As multiple years have passed since, the behaviour change seems to be
here to stay, and any users of the previous form of the option have been
forced to avoid this construct. Thus update llvm-windres to match the
new way Binutils of handling this option.

One construct for specifying the path to the preprocessor, which works
both before and after binutils 2.36 (and this change in llvm-windres) is
to specify options like this:

--preprocessor path/to/executable --preprocessor-arg -E
--preprocessor-arg -xc -DRC_INVOKED

show more ...

[llvm-windres] Resolve the --preprocessor executable in $PATH (#75390)

The llvm::sys::ExecuteAndWait function doesn't resolve the file to be
executed from $PATH - i.e. it is similar to execv(), not

[llvm-windres] Resolve the --preprocessor executable in $PATH (#75390)

The llvm::sys::ExecuteAndWait function doesn't resolve the file to be
executed from $PATH - i.e. it is similar to execv(), not execvp().

Due to this, specifying a --preprocessor argument to llvm-windres only
worked if it specified an absolute path to the preprocessor executable.
This was observed as one of the issues in
https://github.com/msys2/MINGW-packages/pull/19157.

Before d2fa6b694c2052cef1ddd507f6569bc84e3bbe35, this usage of
--preprocessor seemed to work, because the first argument of Args[] was
ignored and llvm-windres just executed the autodetected clang executable
regardless.

Also improve the error messages printed if preprocessing failed. (If the
preprocessor executable was started but itself returned an error, we
don't get any error string.)

show more ...

[llvm-windres] Pass user preprocessor arguments before the input filename (#75389)

If passing the windres option --preprocessor, the default arguments "-E
-xc -DRC_INVOKED" aren't passed. If these

[llvm-windres] Pass user preprocessor arguments before the input filename (#75389)

If passing the windres option --preprocessor, the default arguments "-E
-xc -DRC_INVOKED" aren't passed. If these are passed explicitly by the
user via --preprocessor-arg instead, we need to make sure that "-xc" is
passed before the input filename, as this compiler/preprocessor option
only has an effect on input files that follow it.

This fixes one of the issues with llvm-windres observed in
https://github.com/msys2/MINGW-packages/pull/19157.

show more ...

[llvm-windres] Implement the windres flag --use-temp-file

Whether a temp file or a pipe is used for preprocessing is an
internal detail, this flag has a notable effect on the preprocessing
in GNU wi

[llvm-windres] Implement the windres flag --use-temp-file

Whether a temp file or a pipe is used for preprocessing is an
internal detail, this flag has a notable effect on the preprocessing
in GNU windres. Without this flag, GNU windres passes command
arguments as-is to popen(), which means they get evaluated by a
shell without being re-escaped for this case. To mimic this,
llvm-windres has manually tried to unescape arguments.

When GNU windres is given the --use-temp-file flag, it uses a
different API for invoking the preprocessor, and this API takes care
of preserving special characters in the command line arguments.
For users of GNU windres, this means that by using --use-temp-file,
they don't need to do the (quite terrible) double escaping of
quotes/spaces etc.

The xz project uses the --use-temp-file flag when invoking
GNU windres, see
https://github.com/tukaani-project/xz/commit/6b117d3b1fe91eb26d533ab16a2e552f84148d47.
However as llvm-windres didn't implement this flag and just
assumed the GNU windres popen() behaviour, they had to use a
different codepath for llvm-windres.

That separate codepath for llvm-windres broke later when llvm-windres
got slightly more accurate unescaping of lone quotes in
0f4c6b120f21d582ab7c5c4f2b2a475086c34938 /
https://reviews.llvm.org/D146848 (fixing a discrepancy to GNU
windres as found in https://github.com/llvm/llvm-project/issues/57334),
and this was reported in
https://github.com/mstorsjo/llvm-mingw/issues/363.

Not touching the implementation of the --preprocessor option
with respect to the --use-temp-file flag; that option is doubly
tricky as GNU windres changed its behaviour in a backwards incompatible
way recently (and llvm-windres currently matches the old behaviour).
(See
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=21c33bcbe36377abf01614fb1b9be439a3b6de20,
https://sourceware.org/bugzilla/show_bug.cgi?id=27594 and
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=5edb8e3f5ad8d74a83fc0df7f6e4514eed0aa77f;hp=3abbafc2aacc6706fea3e3e326e2f08d107c3672
for the behaviour change.)

Differential Revision: https://reviews.llvm.org/D159223

show more ...

[llvm-rc] Continue to use Argv[0] to resolve executable path

In internal google builds, MainExecPath doesn't go to the directory with `clang`.
Fall back to using Argv0 if MainExecPath doesn't find a

[llvm-rc] Continue to use Argv[0] to resolve executable path

In internal google builds, MainExecPath doesn't go to the directory with `clang`.
Fall back to using Argv0 if MainExecPath doesn't find any clangs.

Differential Revision: https://reviews.llvm.org/D158901

show more ...

Reapply "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

This reverts commit 4e3b89483a6922d3f48670bb1c50a37f342918c6, with
fixes for places I'd missed updating in lld and lld

Reapply "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

This reverts commit 4e3b89483a6922d3f48670bb1c50a37f342918c6, with
fixes for places I'd missed updating in lld and lldb. I've also
renamed OptionVisibility::Default to "DefaultVis" to avoid ambiguity
since the undecorated name has to be available anywhere Options.inc is
included.

Original message follows:

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and adds
variants of the OptTable APIs that use Visibility mask instead of
Include/Exclude flags.

We need to do this to clean up a bunch of complexity in the clang
driver's option handling - there's a whole slew of flags like
CoreOption, NoDriverOption, and FlangOnlyOption there today to try to
handle all of the permutations of flags that the various drivers need,
but it really doesn't scale well, as can be seen by things like the
somewhat recently introduced CLDXCOption.

Instead, we'll provide an additive model for visibility that's
separate from the other flags. For things like "HelpHidden", which is
used as a "subtractive" modifier for option visibility, we leave that
in "Flags" and handle it as a special case.

Note that we don't actually update the users of the Include/Exclude
APIs here or change the flags that exist in clang at all - that will
come in a follow up that refactors clang's Options.td to use the
increased flexibility this change allows.

Differential Revision: https://reviews.llvm.org/D157149

show more ...

Revert "[Option] Add "Visibility" field and clone the OptTable APIs to use it"

this is failing on bots, reverting to investigate.

This reverts commit a16104e6da6f36f3d72dbf53d10ba56495a0d65a.

[Option] Add "Visibility" field and clone the OptTable APIs to use it

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and a

[Option] Add "Visibility" field and clone the OptTable APIs to use it

This splits OptTable's "Flags" field into "Flags" and "Visibility",
updates the places where we instantiate Option tables, and adds
variants of the OptTable APIs that use Visibility mask instead of
Include/Exclude flags.

We need to do this to clean up a bunch of complexity in the clang
driver's option handling - there's a whole slew of flags like
CoreOption, NoDriverOption, and FlangOnlyOption there today to try to
handle all of the permutations of flags that the various drivers need,
but it really doesn't scale well, as can be seen by things like the
somewhat recently introduced CLDXCOption.

Instead, we'll provide an additive model for visibility that's
separate from the other flags. For things like "HelpHidden", which is
used as a "subtractive" modifier for option visibility, we leave that
in "Flags" and handle it as a special case.

Note that we don't actually update the users of the Include/Exclude
APIs here or change the flags that exist in clang at all - that will
come in a follow up that refactors clang's Options.td to use the
increased flexibility this change allows.

Differential Revision: https://reviews.llvm.org/D157149

show more ...

[llvm-rc] Resolve the executable path if not present in Argv[0]

The llvm-rc tool tries to locate a suitable Clang executable to
use for preprocessing. For this purpose, it first checks within
the sa

[llvm-rc] Resolve the executable path if not present in Argv[0]

The llvm-rc tool tries to locate a suitable Clang executable to
use for preprocessing. For this purpose, it first checks within
the same directory as the llvm-rc tool, checking with a couple
different names, followed by checking all of $PATH for another
couple names.

On Windows, the InitLLVM() function always sets up Argv[0] with the
full path to the executable, while on Unix, Argv[0] is kept as is.

Therefore, call getMainExecutable to try to resolve the directory of
the executable before looking for colocated Clang executables.

This makes 282744a9ce18120dc0a6eceb02693b36980d9498 actually have
the desired effect.

Differential Revision: https://reviews.llvm.org/D157241

show more ...

[llvm] Extract common `OptTable` bits into macros

All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OP

[llvm] Extract common `OptTable` bits into macros

All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OPT_##ID`), kind (`Option::KIND##Class`), group ID (`OPT_##GROUP`) and alias ID (`OPT_##ALIAS`). This patch extracts that common code into canonical macros. This results in fewer changes when tweaking the `OPTION` macros emitted by the TableGen backend.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D157028

show more ...

[llvm-rc] Improve help printouts

This more clearly identifies the tool as llvm-rc.

This should hopefully allow Meson to check for parts of these strings
in the output of "$CMD /?" when detecting th

[llvm-rc] Improve help printouts

This more clearly identifies the tool as llvm-rc.

This should hopefully allow Meson to check for parts of these strings
in the output of "$CMD /?" when detecting the kind of resource compiler
tool, to allow Meson to recognize llvm-rc.

Differential Revision: https://reviews.llvm.org/D154545

show more ...

[lvm-windres] Try to match GNU windres regarding handling of unescaped quotes

Some background context: GNU windres invokes the preprocessor in
a subprocess. Some windres options are passed through t

[lvm-windres] Try to match GNU windres regarding handling of unescaped quotes

Some background context: GNU windres invokes the preprocessor in
a subprocess. Some windres options are passed through to the
preproocessor, e.g. -D options for predefining defines.
When GNU windres passes these options onwards, it takes the options
in exact the form they are received (in argv or similar) and
assembles them into a single preprocessor command string which gets
interpreted by a shell (IIRC via the popen() function, or similar).

When LLVM invokes subprocesses, it does so via APIs that take
properly split argument vectors, to avoid needing to worry about
shell quoting/escaping/unescaping. But in the case of LLVM windres,
we have to emulate the effect of the shell parsing done by popen().

Most of the relevant cases are already taken care of here, but this
patch fixes an uncommon case encountered in
https://github.com/llvm/llvm-project/issues/57334.
(This case is uncommon since it doesn't do what one would want to;
the quotes need to be escaped more to work as intended through
the popen() shell).

Differential Revision: https://reviews.llvm.org/D146848

show more ...

[llvm-rc] Remove transitional preprocessing fallback logic

When preprocessing was integrated to llvm-rc in 2021, this was a
new requirement (previously one could execute llvm-rc without a
suitable p

[llvm-rc] Remove transitional preprocessing fallback logic

When preprocessing was integrated to llvm-rc in 2021, this was a
new requirement (previously one could execute llvm-rc without a
suitable preprocessing tool to be available).

As a transitional helper, llvm-rc fell back on skipping preprocessing
if no suitable tool was found (with a warning printed), but users
could pass an llvm-rc specific option to silence the warning, if they
explicitly want to run the tool without preprocessing.

Now 2 years later, remove the transitional helper - error out if
preprocessing failed. The option for disabling preprocessing remains.

Differential Revision: https://reviews.llvm.org/D146797

show more ...

[llvm-rc] Fix the reference to the option for disabling preprocessing in a message

This was the original option name from the first iteration of the patch
that added the feature, but during review,

[llvm-rc] Fix the reference to the option for disabling preprocessing in a message

This was the original option name from the first iteration of the patch
that added the feature, but during review, a different name was suggested
and preferred - but the reference in the helpful message was missed.

Differential Revision: https://reviews.llvm.org/D146796

show more ...

[llvm-rc] Look for "clang-<major>" when locating a suitable preprocessor

In some cases, there's no adjacent executable named "clang" or
"clang-cl", but one name "clang-<major>". This logic doesn't
c

[llvm-rc] Look for "clang-<major>" when locating a suitable preprocessor

In some cases, there's no adjacent executable named "clang" or
"clang-cl", but one name "clang-<major>". This logic doesn't
cover every possible deployment setup of course, but should
cover more fairly common/reasonable cases.

See
https://github.com/curl/curl-for-win/commit/caaae171ac43af5b883403714dafd42030d8de61#commitcomment-105808524
for discussion about a case where this would have been helpful.

Differential Revision: https://reviews.llvm.org/D146794

show more ...

[llvm-rc] Respect the executable specified in the --preprocessor command

The arguments passed in this option were passed onto the child
process, but we still blindly used the clang binary that we ha

[llvm-rc] Respect the executable specified in the --preprocessor command

The arguments passed in this option were passed onto the child
process, but we still blindly used the clang binary that we had
found to sys::ExecuteAndWait as the intended executable to run.

If the user hasn't specified any custom --preprocessor command,
Args[0] is equal to the variable Clang.

This doesn't affect any tests, since the tests only print the
arguments it would try to execute (but not the first parameter to
sys::ExecuteAndWait), but there's no testes for executing it
(and validating that it did execute the right thing).

Differential Revision: https://reviews.llvm.org/D146793

show more ...

[llvm-driver] Pass extra arguments to tools

Differential Revision: https://reviews.llvm.org/D137799

[NFC][TargetParser] Replace uses of llvm/Support/Host.h

The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC w

[NFC][TargetParser] Replace uses of llvm/Support/Host.h

The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC warning about the fact it is deprecated, because it is used
in `isl` from where it is included by Polly.

show more ...

[NFC][TargetParser] Remove llvm/ADT/Triple.h

I also ran `git clang-format` to get the headers in the right order for
the new location, which has changed the order of other headers in two
files.

[OptTable] Precompute OptTable prefixes union table through tablegen

This avoid rediscovering this table when reading each options, providing
a sensible 2% speedup when processing and empty file, an

[OptTable] Precompute OptTable prefixes union table through tablegen

This avoid rediscovering this table when reading each options, providing
a sensible 2% speedup when processing and empty file, and a measurable
speedup on typical workloads, see:

This is optional, the legacy, on-the-fly, approach can still be used
through the GenericOptTable class, while the new one is used through
PrecomputedOptTable.

https://llvm-compile-time-tracker.com/compare.php?from=4da6cb3202817ee2897d6b690e4af950459caea4&to=19a492b704e8f5c1dea120b9c0d3859bd78796be&stat=instructions:u

Differential Revision: https://reviews.llvm.org/D140800

show more ...