Reland [OffloadBundler] Compress bundles over 4GB (#122307)

Reland the patch after fixing the lit test.

Revert "[OffloadBundler] Compress bundles over 4GB (#122307)"

revert due to failure in buildbot

https://lab.llvm.org/buildbot/#/builders/144/builds/16114

This reverts commit 4e2efc3bd500836d0fa97

Revert "[OffloadBundler] Compress bundles over 4GB (#122307)"

revert due to failure in buildbot

https://lab.llvm.org/buildbot/#/builders/144/builds/16114

This reverts commit 4e2efc3bd500836d0fa977d6e257ffee2c92e178.

show more ...

[OffloadBundler] Compress bundles over 4GB (#122307)

Added initial support for version 3 of the compressed offload bundle
format, which uses 64-bit fields for Total File Size and Uncompressed
Bina

[OffloadBundler] Compress bundles over 4GB (#122307)

Added initial support for version 3 of the compressed offload bundle
format, which uses 64-bit fields for Total File Size and Uncompressed
Binary Size. This enables support for files larger than 4GB. The support
is currently experimental and can be enabled by setting the environment
variable `COMPRESSED_BUNDLE_FORMAT_VERSION=3`.

show more ...

[llvm][NFC] Rework Timer.cpp globals to ensure valid lifetimes (#121663)

This is intended to help with flang `-ftime-report` support:
- #107270.

With this change, I was able to cherry-pick #1072

[llvm][NFC] Rework Timer.cpp globals to ensure valid lifetimes (#121663)

This is intended to help with flang `-ftime-report` support:
- #107270.

With this change, I was able to cherry-pick #107270, uncomment
`llvm::TimePassesIsEnabled = true;` and compile with `-ftime-report`.

I also noticed that `clang/lib/Driver/OffloadBundler.cpp` was statically
constructing a `TimerGroup` and changed it to lazily construct via
ManagedStatic.

show more ...

[Driver] Remove unused includes (NFC) (#116316)

Identified with misc-include-cleaner.

[SystemZ][z/OS] Open text files in text mode (#109972)

This patch continues the work that was started here
https://reviews.llvm.org/D99426 to correctly open text files in text
mode.

[Driver] Avoid repeated hash lookups (NFC) (#108676)

[CUDA][NFC] CudaArch to OffloadArch rename (#97028)

Rename `CudaArch` to `OffloadArch` to better reflect its content and the
use.
Apply a similar rename to helpers handling the enum.

[ClangOffloadBundler] make hipv4 and hip compatible (#91637)

The distinction between the hip and hipv4 offload kinds is historically
based. Originally, these designations might have indicated diffe

[ClangOffloadBundler] make hipv4 and hip compatible (#91637)

The distinction between the hip and hipv4 offload kinds is historically
based. Originally, these designations might have indicated different
versions of the code object ABI (Application Binary Interface). However,
as the system has evolved, the ABI version is now embedded directly
within the code object itself, making these historical distinctions
irrelevant during the unbundling process. Consequently, hip and hipv4
are treated as compatible in current implementations, facilitating
interchangeable handling of code objects without differentiation based
on offload kind. This change streamlines code management within the
ecosystem.

show more ...

[ClangOffloadBundler] Add file size to header (#88827)

__hipRegisterFatBinary only accepts one pointer argument. It is expected
to get the fat binary size from the header.

This patch adds a file

[ClangOffloadBundler] Add file size to header (#88827)

__hipRegisterFatBinary only accepts one pointer argument. It is expected
to get the fat binary size from the header.

This patch adds a file size field to the header of the compressed
bundle.

show more ...

[HIP] add --offload-compression-level= option (#83605)

Added --offload-compression-level= option to clang and
-compression-level=
option to clang-offload-bundler for controlling compression level.

[HIP] add --offload-compression-level= option (#83605)

Added --offload-compression-level= option to clang and
-compression-level=
option to clang-offload-bundler for controlling compression level.

Added support of long distance matching (LDM) for llvm::zstd which is
off
by default. Enable it for clang-offload-bundler by default since it
improves compression rate in general.

Change default compression level to 3 for zstd for clang-offload-bundler
since it works well for bundle entry size from 1KB to 32MB, which should
cover most of the clang-offload-bundler usage. Users can still specify
compression level by -compression-level= option if necessary.

show more ...

[ClangOffloadBundler] fix unbundling archive (#84195)

When unbundling an archive, need to save the content of each object file
to a temporary file before passing it to llvm-objcopy, instead of
pas

[ClangOffloadBundler] fix unbundling archive (#84195)

When unbundling an archive, need to save the content of each object file
to a temporary file before passing it to llvm-objcopy, instead of
passing the original input archive file to llvm-objcopy.

Also allows extracting host bundles for archives.

Fixes: https://github.com/llvm/llvm-project/issues/83509

show more ...

[HIP] Allow partial linking for `-fgpu-rdc` (#81700)

`-fgpu-rdc` mode allows device functions call device functions in
different TU. However, currently all device objects have to be linked
togethe

[HIP] Allow partial linking for `-fgpu-rdc` (#81700)

`-fgpu-rdc` mode allows device functions call device functions in
different TU. However, currently all device objects have to be linked
together since only one fat binary is supported. This is time consuming
for AMDGPU backend since it only supports LTO.

There are use cases that objects can be divided into groups in which
device functions are self-contained but host functions are not. It is
desirable to link/optimize/codegen the device code and generate a fatbin
for each group, whereas partially link the host code with `ld -r` or
generate a static library by using the `--emit-static-lib` option of
clang. This avoids linking all device code together, therefore decreases
the linking time for `-fgpu-rdc`.

Previously, clang emits an external symbol `__hip_fatbin` for all
objects for `-fgpu-rdc`. With this patch, clang emits an unique external
symbol `__hip_fatbin_{cuid}` for the fat binary for each object. When a
group of objects are linked together to generate a fatbin, the symbols
are merged by alias and point to the same fat binary. Each group has its
own fat binary. One executable or shared library can have multiple fat
binaries. Device linking is done for undefined fab binary symbols only
to avoid repeated linking. `__hip_gpubin_handle` is also uniquefied and
merged to avoid repeated registering. Symbol `__hip_cuid_{cuid}` is
introduced to facilitate debugging and tooling.

Fixes: https://github.com/llvm/llvm-project/issues/77018

show more ...

[clang-offload-bundler] Add support for -check-input-archive (#73709)

In this patch, we add support for checking a heterogeneous archive. We
also significantly improve the clang-offload-bundler doc

[clang-offload-bundler] Add support for -check-input-archive (#73709)

In this patch, we add support for checking a heterogeneous archive. We
also significantly improve the clang-offload-bundler documentation.

show more ...

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an enum.
This patch replaces llvm:

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an enum.
This patch replaces llvm::support::{big,little,native} with
llvm::endianness::{big,little,native}.

show more ...

Reland "[HIP] Support compressing device binary"

Original PR: https://github.com/llvm/llvm-project/pull/67162

The commit was reverted due to UB detected by santizer:

https://lab.llvm.org/buildbot/

Reland "[HIP] Support compressing device binary"

Original PR: https://github.com/llvm/llvm-project/pull/67162

The commit was reverted due to UB detected by santizer:

https://lab.llvm.org/buildbot/#/builders/238/builds/5955

clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error:
load of misaligned address 0xaaaae2d90e7c for type
'const uint64_t' (aka 'const unsigned long'), which
requires 8 byte alignment

It was fixed by using memcpy instead of dereferencing int*
casted from unaligned char*.

show more ...

Revert "[HIP] Support compressing device binary (#67162)"

This reverts commit a1e81d2ead02e041471ec2299d7382f80c4dbba6.

Revert "Fix test hip-offload-compress-zlib.hip"

This reverts commit ba01ce60

Revert "[HIP] Support compressing device binary (#67162)"

This reverts commit a1e81d2ead02e041471ec2299d7382f80c4dbba6.

Revert "Fix test hip-offload-compress-zlib.hip"

This reverts commit ba01ce60665848478ba4e76190907153a8c26fe9.

Revert due to sanity fail at

https://lab.llvm.org/buildbot/#/builders/5/builds/37188

https://lab.llvm.org/buildbot/#/builders/238/builds/5955

/b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error: load of misaligned address 0xaaaae2d90e7c for type 'const uint64_t' (aka 'const unsigned long'), which requires 8 byte alignment
0xaaaae2d90e7c: note: pointer points here
bc 00 00 00 94 dc 29 9a 89 fb ca 2b 78 9c 8b 8f 77 f6 71 f4 73 8f f7 77 73 f3 f1 77 74 89 77 0a
^
#0 0xaaaaba125f70 in clang::CompressedOffloadBundle::decompress(llvm::MemoryBuffer const&, bool) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25
#1 0xaaaaba126150 in clang::OffloadBundler::ListBundleIDsInFile(llvm::StringRef, clang::OffloadBundlerConfig const&) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1089:7

Will reland after fixing it.

show more ...

[HIP] Support compressing device binary (#67162)

Add option -f[no-]offload-compress to clang to enable/disable
compression of device binary for HIP. By default it is disabled.

Add option -compre

[HIP] Support compressing device binary (#67162)

Add option -f[no-]offload-compress to clang to enable/disable
compression of device binary for HIP. By default it is disabled.

Add option -compress to clang-offload-bundler to enable compression of
offload bundle. By default it is disabled.

When enabled, zstd or zlib is used for compression when available.

When disabled, it is NFC compared to previous behavior. The same offload
bundle format is used as before.

Clang-offload-bundler automatically detects whether the input file to be
unbundled is compressed and the compression method and decompress if
necessary.

show more ...

[AIX] supporting -X options for llvm-ranlib in AIX OS

Summary:

llvm-ar is symlinked as llvm-ranlib and will act as ranlib when invoked in that mode. llvm-ar since [[ https://github.ibm.com/compiler

[AIX] supporting -X options for llvm-ranlib in AIX OS

Summary:

llvm-ar is symlinked as llvm-ranlib and will act as ranlib when invoked in that mode. llvm-ar since [[ https://github.ibm.com/compiler/llvm-project/commit/4f2cfbe5314b064625b2c87bde6ce5c8d04004c5 | compiler/llvm-project@4f2cfbe ]] supports the -X options, but doesn't seem to accept them when running as llvm-ranlib.

In AIX OS , according to https://www.ibm.com/docs/en/aix/7.2?topic=r-ranlib-command

-X mode Specifies the type of object file ranlib should examine. The mode must be one of the following:

32
Processes only 32-bit object files
64
Processes only 64-bit object files
32_64, any
Processes both 32-bit and 64-bit object files

The default is to process 32-bit object files (ignore 64-bit objects). The mode can also be set with the OBJECT_MODE environment variable. For example, OBJECT_MODE=64 causes ranlib to process any 64-bit objects and ignore 32-bit objects. The -X flag overrides the OBJECT_MODE variable.

Reviewers: James Henderson, MaskRay, Stephen Peckham
Differential Revision: https://reviews.llvm.org/D142660

show more ...

Migrate {starts,ends}with_insensitive to {starts,ends}_with_insensitive (NFC)

This patch migrates uses of StringRef::{starts,ends}with_insensitive
to StringRef::{starts,ends}_with_insensitive so tha

Migrate {starts,ends}with_insensitive to {starts,ends}_with_insensitive (NFC)

This patch migrates uses of StringRef::{starts,ends}with_insensitive
to StringRef::{starts,ends}_with_insensitive so that we can use names
similar to those used in std::string_view.

Note that the llvm/ directory has migrated in commit
6c3ea866e93003e16fc55d3b5cedd3bc371d1fde.

I'll post a separate patch to deprecate
StringRef::{starts,ends}with_insensitive.

Differential Revision: https://reviews.llvm.org/D150506

show more ...

[clang] Use *{Map,Set}::contains (NFC)

[clang] Use *{Set,Map}::contains (NFC)

[clang-offload-bundler] Standardize TargetID field for bundler

The bundler accepts both of the following for the --target option:
hip-amdgcn-amd-amdhsa-gfx900 (no env field)
hip-amdgcn-amd-am

[clang-offload-bundler] Standardize TargetID field for bundler

The bundler accepts both of the following for the --target option:
hip-amdgcn-amd-amdhsa-gfx900 (no env field)
hip-amdgcn-amd-amdhsa--gfx900 (blank env field)

The environment field is defined as optional for Triples
in Triple.h. However, in this patch we update the bundler to
internally standardize to include the env field. While users
aren't required to specify an env field when listing targets on
the commandline, bundles generated by the offload-bundler will
include the ABI field.

This standardization simplifies things for APIs that deal with
bundles generated by the clang-offload-bundler tool.

Differential Revision: https://reviews.llvm.org/D145770

show more ...

[NFC][TargetParser] Replace uses of llvm/Support/Host.h

The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC w

[NFC][TargetParser] Replace uses of llvm/Support/Host.h

The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC warning about the fact it is deprecated, because it is used
in `isl` from where it is included by Polly.

show more ...

[NFC][TargetParser] Remove llvm/ADT/Triple.h

I also ran `git clang-format` to get the headers in the right order for
the new location, which has changed the order of other headers in two
files.