Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change f

Rework the `Option` library to reduce dynamic relocations (#119198)

Apologies for the large change, I looked for ways to break this up and
all of the ones I saw added real complexity. This change focuses on the
option's prefixed names and the array of prefixes. These are present in
every option and the dominant source of dynamic relocations for PIE or
PIC users of LLVM and Clang tooling. In some cases, 100s or 1000s of
them for the Clang driver which has a huge number of options.

This PR addresses this by building a string table and a prefixes table
that can be referenced with indices rather than pointers that require
dynamic relocations. This removes almost 7k dynmaic relocations from the
`clang` binary, roughly 8% of the remaining dynmaic relocations outside
of vtables. For busy-boxing use cases where many different option tables
are linked into the same binary, the savings add up a bit more.

The string table is a straightforward mechanism, but the prefixes
required some subtlety. They are encoded in a Pascal-string fashion with
a size followed by a sequence of offsets. This works relatively well for
the small realistic prefixes arrays in use.

Lots of code has to change in order to land this though: both all the
option library code has to be updated to use the string table and
prefixes table, and all the users of the options library have to be
updated to correctly instantiate the objects.

Some follow-up patches in the works to provide an abstraction for this
style of code, and to start using the same technique for some of the
other strings here now that the infrastructure is in place.

show more ...

[NvlinkWrapper] Add support for `--undefined` (#113934)

Summary:
This flag is pretty canonical in ELF linkers, it allows us to force the
link job to extract a library if it defines a specific symb

[NvlinkWrapper] Add support for `--undefined` (#113934)

Summary:
This flag is pretty canonical in ELF linkers, it allows us to force the
link job to extract a library if it defines a specific symbol. This is
mostly useful for letting us forcibly extract things that don't fit the
normal model (i.e. kernels) from static libraries.

show more ...

[NvlinkWrapper] Use `-plugin-opt=mattr=` instead of a custom feature (#111712)

Summary:
We don't need a custom flag for this, LLVM had a way to get the features
which are forwarded via `plugin-opt

[NvlinkWrapper] Use `-plugin-opt=mattr=` instead of a custom feature (#111712)

Summary:
We don't need a custom flag for this, LLVM had a way to get the features
which are forwarded via `plugin-opt`.

show more ...

[nvlink-wrapper] Remove use of symlinks

Summary:
This was intended to be a neat optimization, but some objects come from
archives so this won't work. I could write some code to detect if it
came fro

[nvlink-wrapper] Remove use of symlinks

Summary:
This was intended to be a neat optimization, but some objects come from
archives so this won't work. I could write some code to detect if it
came from an archive but I don't think it's wroth the complexity when
this already doesn't work on Windows.

show more ...

[nvlink-wrapper] Use a symbolic link instead of copying the file (#110139)

Summary:
We need all inputs to `nvlink` to end in `.cubin` while the rest of the
compiler toolchain wants `.o`. Previously

[nvlink-wrapper] Use a symbolic link instead of copying the file (#110139)

Summary:
We need all inputs to `nvlink` to end in `.cubin` while the rest of the
compiler toolchain wants `.o`. Previously we copied `.o` file to
`.cubin` files, but this is wasteful. Instead, we can just create a link
against it. This saves some disk space during link time.

show more ...

[Clang] Simplify specifying passes via -Xoffload-linker (#102483)

Make it possible to do things like the following, regardless of whether
the offload target is nvptx or amdgpu:

```
$ clang -O1

[Clang] Simplify specifying passes via -Xoffload-linker (#102483)

Make it possible to do things like the following, regardless of whether
the offload target is nvptx or amdgpu:

```
$ clang -O1 -g -fopenmp --offload-arch=native test.c \
-Xoffload-linker -mllvm=-pass-remarks=inline \
-Xoffload-linker -mllvm=-force-remove-attribute=g.internalized:noinline\
-Xoffload-linker --lto-newpm-passes='forceattrs,default<O1>' \
-Xoffload-linker --lto-debug-pass-manager \
-foffload-lto
```

To accomplish that:

- In clang-linker-wrapper, do not forward options via `-Wl` if they
might have literal commas. Use `-Xlinker` instead.
- In clang-nvlink-wrapper, accept `--lto-debug-pass-manager` and
`--lto-newpm-passes`.
- In clang-nvlink-wrapper, drop `-passes` because it's inconsistent with
the interface of `lld`, which is used instead of clang-nvlink-wrapper
when the target is amdgpu. Without this patch, `-passes` is passed to
`nvlink`, producing an error anyway.

---------

Co-authored-by: Joseph Huber <huberjn@outlook.com>

show more ...

[Clang] Suppress missing architecture error when doing LTO (#100652)

Summary:
The `nvlink-wrapper` can do LTO now, which means we can still create
some LLVM-IR without needing an architecture. In th

[Clang] Suppress missing architecture error when doing LTO (#100652)

Summary:
The `nvlink-wrapper` can do LTO now, which means we can still create
some LLVM-IR without needing an architecture. In the case that we try to
invoke `nvlink` internally, that will still fail. This patch simply
defers the error until later so we can use `--lto-emit-llvm` to get the
IR without specifying an architecture.

show more ...

[NvlinkWrapper] Add relocatable link support to LTO pass

Summary:
We need `-r` to work so we can preserve symbols.

[Clang] Fix r-value binding that Windows doesn't like

[Clang] Fix 'clang-nvlink-wrapper' not working w/o CUDA

Summary:
This would try to find `nvlink` and then fail even in `dry-run` mode.
We now just let it continue and pretend like we found it. Also

[Clang] Fix 'clang-nvlink-wrapper' not working w/o CUDA

Summary:
This would try to find `nvlink` and then fail even in `dry-run` mode.
We now just let it continue and pretend like we found it. Also add it to
the depends.

show more ...

[Clang] Fix incorrect value assignment in nvlink wrapper

Summary:
Gah, forgot to push this before I merged.

[Clang] Introduce 'clang-nvlink-wrapper' to work around 'nvlink' (#96561)

Summary:
The `clang-nvlink-wrapper` is a utility that I removed awhile back
during the transition to the new driver. This pa

[Clang] Introduce 'clang-nvlink-wrapper' to work around 'nvlink' (#96561)

Summary:
The `clang-nvlink-wrapper` is a utility that I removed awhile back
during the transition to the new driver. This patch adds back in a new,
upgraded version that does LTO + archive linking. It's not an easy
choice to reintroduce something I happily deleted, but this is the only
way to move forward with improving GPU support in LLVM.

While NVIDIA provides a linker called 'nvlink', its main interface is
very difficult to work with. It does not provide LTO, or static linking,
requires all files to be named a non-standard `.cubin`, and rejects link
jobs that other linkers would be fine with (i.e empty). I have spent a
great deal of time hacking around this in the GPU `libc` implementation,
where I deliberately avoid LTO and static linking and have about 100
lines of hacky CMake dedicated to storing these files in a format that
the clang-linker-wrapper accepts to avoid this limitation.

The main reason I want to re-intorudce this tool is because I am
planning on creating a more standard C/C++ toolchain for GPUs to use.
This will install files like the following.
```
<install>/lib/nvptx64-nvidia-cuda/libc.a
<install>/lib/nvptx64-nvidia-cuda/libc++.a
<install>/lib/nvptx64-nvidia-cuda/libomp.a
<install>/lib/clang/19/lib/nvptx64-nvidia-cuda/libclang_rt.builtins.a
```
Linking in these libraries will then simply require passing `-lc` like
is already done for non-GPU toolchains. However, this doesn't work with
the currently deficient `nvlink` linker, so I consider this a blocking
issue to massively improving the state of building GPU libraries.

In the future we may be able to convince NVIDIA to port their linker to
`ld.lld`, but for now this is the only workable solution that allows us
to hack around the weird behavior of their closed-source software.
This also copies some amount of logic from the clang-linker-wrapper,
but not enough for it to be worthwhile to merge them I feel. In the
future it may be possible to delete that handling from there entirely.

show more ...