[clang][Driver] Don't ignore -gmodules .gch files (#77711)

A previous commit (82f75ed) made clang ignore .gch files that were not
Clang AST files. This broke `-gmodules`, which embeds the Clang AST

[clang][Driver] Don't ignore -gmodules .gch files (#77711)

A previous commit (82f75ed) made clang ignore .gch files that were not
Clang AST files. This broke `-gmodules`, which embeds the Clang AST into
an object file containing debug info.

This changes the probing to detect any file format recognized by
`llvm::identify_magic()` as potentially containing a Clang AST.

Previous PR: https://github.com/llvm/llvm-project/pull/69204

show more ...

[ELF] Attempt to set the OS when using 'makeTriple()' (#76992)

Summary:
This patch fixes up the `makeTriple()` interface to emit append the
operating system information when it is readily avaialble

[ELF] Attempt to set the OS when using 'makeTriple()' (#76992)

Summary:
This patch fixes up the `makeTriple()` interface to emit append the
operating system information when it is readily avaialble from the ELF.
The main motivation for this is so the GPU architectures can be easily
identified correctly when given and ELF. E.g. we want
`amdgpu-amd-amdhsa` as the output and not `amdgpu--`.

This required adding support for the CUDA OS/ABI, which is easily found
to be `0x33` when using `readelf`.

show more ...

[LLVM] Add file magic detection for SPIR-V files. (#75363)

Summary:
More SPIR-V related patches are being upstreamed. We should add support
to detect when a binary file is SPIR-V. This will be used

[LLVM] Add file magic detection for SPIR-V files. (#75363)

Summary:
More SPIR-V related patches are being upstreamed. We should add support
to detect when a binary file is SPIR-V. This will be used in the future
when support for SPIR-V is added to the offloading runtime or more
support for bundling.

The magic number is described in the official documentation:
https://registry.khronos.org/SPIR-V/specs/1.0/SPIRV.html#Magic. Notably,
SPIR-V files are streams of 32-bit words. This means that the magic
numbers differ depending on the endianness. Here we simply check the
strandard and byte-reversed versions.

show more ...

Reland "[HIP] Support compressing device binary"

Original PR: https://github.com/llvm/llvm-project/pull/67162

The commit was reverted due to UB detected by santizer:

https://lab.llvm.org/buildbot/

Reland "[HIP] Support compressing device binary"

Original PR: https://github.com/llvm/llvm-project/pull/67162

The commit was reverted due to UB detected by santizer:

https://lab.llvm.org/buildbot/#/builders/238/builds/5955

clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error:
load of misaligned address 0xaaaae2d90e7c for type
'const uint64_t' (aka 'const unsigned long'), which
requires 8 byte alignment

It was fixed by using memcpy instead of dereferencing int*
casted from unaligned char*.

show more ...

Revert "[HIP] Support compressing device binary (#67162)"

This reverts commit a1e81d2ead02e041471ec2299d7382f80c4dbba6.

Revert "Fix test hip-offload-compress-zlib.hip"

This reverts commit ba01ce60

Revert "[HIP] Support compressing device binary (#67162)"

This reverts commit a1e81d2ead02e041471ec2299d7382f80c4dbba6.

Revert "Fix test hip-offload-compress-zlib.hip"

This reverts commit ba01ce60665848478ba4e76190907153a8c26fe9.

Revert due to sanity fail at

https://lab.llvm.org/buildbot/#/builders/5/builds/37188

https://lab.llvm.org/buildbot/#/builders/238/builds/5955

/b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error: load of misaligned address 0xaaaae2d90e7c for type 'const uint64_t' (aka 'const unsigned long'), which requires 8 byte alignment
0xaaaae2d90e7c: note: pointer points here
bc 00 00 00 94 dc 29 9a 89 fb ca 2b 78 9c 8b 8f 77 f6 71 f4 73 8f f7 77 73 f3 f1 77 74 89 77 0a
^
#0 0xaaaaba125f70 in clang::CompressedOffloadBundle::decompress(llvm::MemoryBuffer const&, bool) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25
#1 0xaaaaba126150 in clang::OffloadBundler::ListBundleIDsInFile(llvm::StringRef, clang::OffloadBundlerConfig const&) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1089:7

Will reland after fixing it.

show more ...

[HIP] Support compressing device binary (#67162)

Add option -f[no-]offload-compress to clang to enable/disable
compression of device binary for HIP. By default it is disabled.

Add option -compre

[HIP] Support compressing device binary (#67162)

Add option -f[no-]offload-compress to clang to enable/disable
compression of device binary for HIP. By default it is disabled.

Add option -compress to clang-offload-bundler to enable compression of
offload bundle. By default it is disabled.

When enabled, zstd or zlib is used for compression when available.

When disabled, it is NFC compared to previous behavior. The same offload
bundle format is used as before.

Clang-offload-bundler automatically detects whether the input file to be
unbundled is compressed and the compression method and decompress if
necessary.

show more ...

Reland "[gold] Add preliminary FatLTO support to the Gold plugin""

This changes the definition if isSectionBitcode to only be valid for the
.llvm.lto section, since this API is only called from LTO,

Reland "[gold] Add preliminary FatLTO support to the Gold plugin""

This changes the definition if isSectionBitcode to only be valid for the
.llvm.lto section, since this API is only called from LTO, and the
.llvmbc section was not intended to be used for LTO. This allows the
gold plugin to keep its existing behavior without introducing any
significant changes.

Reviewed By: MaskRay, nikic

Differential Revision: https://reviews.llvm.org/D152973

show more ...

Revert "[gold] Add preliminary FatLTO support to the Gold plugin"

This reverts commit 421e4026111315d002879b1e7a0cf3aacd00f488.

One of the test needs a requires line, but we've also seen some issue

Revert "[gold] Add preliminary FatLTO support to the Gold plugin"

This reverts commit 421e4026111315d002879b1e7a0cf3aacd00f488.

One of the test needs a requires line, but we've also seen some issues
for downstream projects that may need coordination, so I'm reverting
this for until we can address those issues. see
https://reviews.llvm.org/D152973#4520240 for context.

show more ...

[gold] Add preliminary FatLTO support to the Gold plugin

This changes the definition if `isSectionBitcode` to only be valid for the
`.llvm.lto` section, since this API is only called from LTO, and t

[gold] Add preliminary FatLTO support to the Gold plugin

This changes the definition if `isSectionBitcode` to only be valid for the
`.llvm.lto` section, since this API is only called from LTO, and the
`.llvmbc` section was not intended to be used for LTO. This allows the
gold plugin to keep its existing behavior without introducing any
significant changes.

Depends on D146778

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D152973

show more ...

Reland [llvm] Preliminary fat-lto-objects support

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not

Reland [llvm] Preliminary fat-lto-objects support

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not to link-time. This is a feature available in GCC for some time,
and makes the existing -ffat-lto-objects flag functional in the same
way as GCC's.

Within LLVM, we add a new EmbedBitcodePass that serializes the module to
the object file, and expose a new pass pipeline for compiling fat
objects. The new pipeline initially clones the module and runs the
selected (Thin)LTOPrelink pipeline, after which it will serialize the
module into a `.llvm.lto` section of an ELF file. When compiling for
(Thin)LTO, this normally the point at which the compiler would emit a
object file containing the bitcode and metadata.

After that point we compile the original module using the
PerModuleDefaultPipeline used for non-LTO compilation. We generate
standard object files at the end of this pipeline, which contain machine
code and the new `.llvm.lto` section containing bitcode.

Since the two pipelines operate on different copies of the module, we
can be sure that the bitcode in the `.llvm.lto` section and object code
in `.text` are congruent with the existing output produced by the
default and LTO pipelines.

Original RFC: https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977

Earlier versions of this patch were missing REQUIRES lines for llc
related tests in Transforms/EmbedBitcode. Those tests are now under
CodeGen/X86, which should avoid running the check on unsupported
platforms.

The EmbedbBitcodePass also returned PreservedAnalyses::all when adding a
metadata section, which failed expensive checks, since it modified the
module. This is now corrected.

Reviewed By: tejohnson, MaskRay, nikic

Differential Revision: https://reviews.llvm.org/D146776

show more ...

Revert "Reland [llvm] Preliminary fat-lto-objects support"

This reverts commit 44265dc3554ef40920b587eeb787a400663af6c7.

Reland [llvm] Preliminary fat-lto-objects support

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not

Reland [llvm] Preliminary fat-lto-objects support

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not to link-time. This is a feature available in GCC for some time,
and makes the existing -ffat-lto-objects flag functional in the same
way as GCC's.

Within LLVM, we add a new EmbedBitcodePass that serializes the module to
the object file, and expose a new pass pipeline for compiling fat
objects. The new pipeline initially clones the module and runs the
selected (Thin)LTOPrelink pipeline, after which it will serialize the
module into a `.llvm.lto` section of an ELF file. When compiling for
(Thin)LTO, this normally the point at which the compiler would emit a
object file containing the bitcode and metadata.

After that point we compile the original module using the
PerModuleDefaultPipeline used for non-LTO compilation. We generate
standard object files at the end of this pipeline, which contain machine
code and the new `.llvm.lto` section containing bitcode.

Since the two pipelines operate on different copies of the module, we
can be sure that the bitcode in the `.llvm.lto` section and object code
in `.text` are congruent with the existing output produced by the
default and LTO pipelines.

Original RFC: https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977

Earlier versions of this patch were missing REQUIRES lines for llc
related tests in Transforms/EmbedBitcode. Those tests are now under
CodeGen/X86, which should avoid running the check on unsupported
platforms.

Reviewed By: tejohnson, MaskRay, nikic

Differential Revision: https://reviews.llvm.org/D146776

show more ...

Revert "[llvm] Preliminary fat-lto-objects support"

There seems to be a problem on arm buildbots. Reverting until I can
investigate. https://lab.llvm.org/buildbot#builders/245/builds/10184

This re

Revert "[llvm] Preliminary fat-lto-objects support"

There seems to be a problem on arm buildbots. Reverting until I can
investigate. https://lab.llvm.org/buildbot#builders/245/builds/10184

This reverts commit a67208e1c697649ce432e6497f56a93675273dd8
and dependent commit e54a3112cee5ae0a9117359ecbea878e1388f51e.

show more ...

[llvm] Preliminary fat-lto-objects support

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not to lin

[llvm] Preliminary fat-lto-objects support

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not to link-time. This is a feature available in GCC for some time,
and makes the existing -ffat-lto-objects flag functional in the same
way as GCC's.

Within LLVM, we add a new EmbedBitcodePass that serializes the module to
the object file, and expose a new pass pipeline for compiling fat
objects. The new pipeline initially clones the module and runs the
selected (Thin)LTOPrelink pipeline, after which it will serialize the
module into a `.llvm.lto` section of an ELF file. When compiling for
(Thin)LTO, this normally the point at which the compiler would emit a
object file containing the bitcode and metadata.

After that point we compile the original module using the
PerModuleDefaultPipeline used for non-LTO compilation. We generate
standard object files at the end of this pipeline, which contain machine
code and the new `.llvm.lto` section containing bitcode.

Since the two pipelines operate on different copies of the module, we
can be sure that the bitcode in the `.llvm.lto` section and object code
in `.text` are congruent with the existing output produced by the
default and LTO pipelines.

Original RFC: https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977

Reviewed By: tejohnson, MaskRay, nikic

Differential Revision: https://reviews.llvm.org/D146776

show more ...

[SystemZ][z/OS] Add GOFFObjectFile class support for HDR, ESD and END records

This patch details the GOFF file format and implements the GOFFObjectfile class
with support for only the HDR, ESD and E

[SystemZ][z/OS] Add GOFFObjectFile class support for HDR, ESD and END records

This patch details the GOFF file format and implements the GOFFObjectfile class
with support for only the HDR, ESD and END GOFF records.

Reviewed By: jhenderson, kpn

Differential Revision: https://reviews.llvm.org/D98437

show more ...

Revert "[SystemZ][z/OS] Add GOFFObjectFile class support for HDR, ESD and END records"

This reverts commit 5b2423183cb35703723ec91ce5b93b99cec36fb2.

[SystemZ][z/OS] Add GOFFObjectFile class support for HDR, ESD and END records

This patch details the GOFF file format and implements the GOFFObjectfile class
with support for only the HDR, ESD and E

[SystemZ][z/OS] Add GOFFObjectFile class support for HDR, ESD and END records

This patch details the GOFF file format and implements the GOFFObjectfile class
with support for only the HDR, ESD and END GOFF records.

Reviewed By: jhenderson, kpn

Differential Revision: https://reviews.llvm.org/D98437

show more ...

[NFC] [Object] Create library to fetch debug info by build ID.

This creates a library for fetching debug info by build ID, whether
locally or remotely via debuginfod. The functionality was refactore

[NFC] [Object] Create library to fetch debug info by build ID.

This creates a library for fetching debug info by build ID, whether
locally or remotely via debuginfod. The functionality was refactored
out of existing code in the Symboliize library. Existing utilities
were refactored to use this library.

Reviewed By: phosek

Differential Revision: https://reviews.llvm.org/D132504

show more ...

Add MachO MH_FILESET support to objdump

https://reviews.llvm.org/D131909

[Binary] Promote OffloadBinary to inherit from Binary

We use the `OffloadBinary` to create binary images of offloading files
and their corresonding metadata. This patch changes this to inherit from

[Binary] Promote OffloadBinary to inherit from Binary

We use the `OffloadBinary` to create binary images of offloading files
and their corresonding metadata. This patch changes this to inherit from
the base `Binary` class. This allows us to create and insepect these
more generically. This patch includes all the necessary glue to
implement this as a new binary format, along with added the magic bytes
we use to distinguish the offloading binary to the `file_magic`
implementation.

Reviewed By: tra

Differential Revision: https://reviews.llvm.org/D126812

show more ...

[NFC] Add missing switch cases

This silences warnings encountered on some bots.

[dsymutil] Apply relocations present in Swift reflection sections

The strippable Swift reflection sections contain subtractor relocations
that need to be applied. There are two situations we need to

[dsymutil] Apply relocations present in Swift reflection sections

The strippable Swift reflection sections contain subtractor relocations
that need to be applied. There are two situations we need to support.
1) Both symbols used in the relocation come from the .o file (for
example, one symbol lives in __swift5_fieldmd and the second in
__swift5_reflstr).
2) One symbol comes from th .o file and the second from the main
binary (for example, __swift5_fieldmd and __swift5_typeref).

Differential Revision: https://reviews.llvm.org/D120574

show more ...

[CUDA] Add CUDA fatbinary magic

Nvidia uses fatbinaries to bundle all of their device code. This patch
adds the magic number "0x50ed55ba" used in their propeitary format to
the list of magic identif

[CUDA] Add CUDA fatbinary magic

Nvidia uses fatbinaries to bundle all of their device code. This patch
adds the magic number "0x50ed55ba" used in their propeitary format to
the list of magic identifies. This is technically undocumented and could
unlikely be changed by Nvidia in the future.

Reviewed By: tra

Differential Revision: https://reviews.llvm.org/D120932

show more ...

Cleanup includes: DebugInfo & CodeGen

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121332

Cleanup LLVMObject headers

Most notably,

llvm/Object/Binary.h no longer includes llvm/Support/MemoryBuffer.h
llvm/Object/MachOUniversal*.h no longer include llvm/Object/Archive.h
llvm/Object/TapiUn

Cleanup LLVMObject headers

Most notably,

llvm/Object/Binary.h no longer includes llvm/Support/MemoryBuffer.h
llvm/Object/MachOUniversal*.h no longer include llvm/Object/Archive.h
llvm/Object/TapiUniversal.h no longer includes llvm/Object/TapiFile.h

llvm-project preprocessed size:
before: 1068185081
after: 1068324320

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119457

show more ...