[clang][modules] Partially revert 48d0eb518 to fix -gmodules output (#124003)

With the changes in 48d0eb518, the CodeGenOptions used to emit .pcm
files with -fmodule-format=obj (-gmodules) were the

[clang][modules] Partially revert 48d0eb518 to fix -gmodules output (#124003)

With the changes in 48d0eb518, the CodeGenOptions used to emit .pcm
files with -fmodule-format=obj (-gmodules) were the ones from the
original invocation, rather than the ones specifically crafted for
outputting the pcm. This was causing the pcm to be written with only the
debug info and without the __clangast section in some cases (e.g. -O2).
This unforunately was not covered by existing tests, because compiling
and loading a module within a single compilation load the ast content
from the in-memory module cache rather than reading it from the pcm file
that was written. This broke bootstrapping a build of clang with modules
enabled on Darwin.

rdar://143418834

show more ...

-ftime-report: reorganize timers

The code generation time is unclear in the -ftime-report output:

* The two clang timers "Code Generation Time" and "LLVM IR Generation
Time" are in the default gr

-ftime-report: reorganize timers

The code generation time is unclear in the -ftime-report output:

* The two clang timers "Code Generation Time" and "LLVM IR Generation
Time" are in the default group "Miscellaneous Ungrouped Timers".
* There is also a "Clang front-end time" group, which actually includes
code generation time.

```
===-------------------------------------------------------------------------===
Miscellaneous Ungrouped Timers
===-------------------------------------------------------------------------===

---User Time--- --System Time-- --User+System-- ---Wall Time--- --- Name ---
0.0611 ( 1.7%) 0.0099 ( 4.4%) 0.0710 ( 1.9%) 0.0713 ( 1.9%) LLVM IR Generation Time
3.5140 ( 98.3%) 0.2165 ( 95.6%) 3.7306 ( 98.1%) 3.7342 ( 98.1%) Code Generation Time
3.5751 (100.0%) 0.2265 (100.0%) 3.8016 (100.0%) 3.8055 (100.0%) Total
...
===-------------------------------------------------------------------------===
Clang front-end time report
===-------------------------------------------------------------------------===
Total Execution Time: 3.9108 seconds (3.9146 wall clock)

---User Time--- --System Time-- --User+System-- ---Wall Time--- --- Name ---
3.6802 (100.0%) 0.2306 (100.0%) 3.9108 (100.0%) 3.9146 (100.0%) Clang front-end timer
3.6802 (100.0%) 0.2306 (100.0%) 3.9108 (100.0%) 3.9146 (100.0%) Total
```

This patch

* renames "Clang front-end time report" (FrontendAction time) to "Clang
time report",
* renames "Clang front-end" to "Front end",
* moves "LLVM IR Generation" into the group,
* replaces "Code Generation time" with "Optimizer" (middle end) and
"Machine code generation" (back end).

```
% clang -c sqlite3.i -w -ftime-report -mllvm -sort-timers=0
...
===-------------------------------------------------------------------------===
Clang time report
===-------------------------------------------------------------------------===
Total Execution Time: 1.5922 seconds (1.5972 wall clock)

---User Time--- --System Time-- --User+System-- ---Wall Time--- --- Name ---
0.5107 ( 35.9%) 0.0105 ( 6.2%) 0.5211 ( 32.7%) 0.5222 ( 32.7%) Front end
0.2464 ( 17.3%) 0.0340 ( 20.0%) 0.2804 ( 17.6%) 0.2814 ( 17.6%) LLVM IR generation
0.6240 ( 43.9%) 0.1235 ( 72.7%) 0.7475 ( 47.0%) 0.7503 ( 47.0%) Machine code generation
0.0413 ( 2.9%) 0.0018 ( 1.0%) 0.0431 ( 2.7%) 0.0433 ( 2.7%) Optimizer
1.4224 (100.0%) 0.1698 (100.0%) 1.5922 (100.0%) 1.5972 (100.0%) Total
```

Pull Request: https://github.com/llvm/llvm-project/pull/122225

show more ...

[CodeGen] Simplify EmitAssemblyHelper and emitBackendOutput

Prepare for -ftime-report change (#122225).

[clang] Simplify BackendConsumer after #69371

[clang] Simplify BackendConsumer. NFC

[CodeGen] Remove unused includes (NFC) (#116459)

Identified with misc-include-cleaner.

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are doc

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.

show more ...

[clang][CodeGen] Strip unneeded calls to raw_string_ostream::str() (NFC)

Try to avoid excess layer of indirection when possible.

p.s. Remove a call to raw_string_ostream::flush() which is a no-op.

Revert "[C++20] [Modules] Embed all source files for C++20 Modules (#102444)"

This reverts commit 2eeeff842f993a694159183a2834b4d305549cad.

See the post commit discussion in
https://github.com/llvm

Revert "[C++20] [Modules] Embed all source files for C++20 Modules (#102444)"

This reverts commit 2eeeff842f993a694159183a2834b4d305549cad.

See the post commit discussion in
https://github.com/llvm/llvm-project/commit/2eeeff842f993a694159183a2834b4d305549cad

show more ...

[C++20] [Modules] Embed all source files for C++20 Modules (#102444)

Close https://github.com/llvm/llvm-project/issues/72383

The implementation rationale is, I don't want to pass
`-fmodules-embe

[C++20] [Modules] Embed all source files for C++20 Modules (#102444)

Close https://github.com/llvm/llvm-project/issues/72383

The implementation rationale is, I don't want to pass
`-fmodules-embed-all-files` all the time since we can't test it in lit
tests (we're using `clang_cc1`). So I tried to set it in FrontendActions
for modules.

show more ...

[BPF] Fix linking issues in static map initializers (#91310)

When BPF object files are linked with bpftool, every symbol must be
accompanied by BTF info. Ensure that extern functions referenced by

[BPF] Fix linking issues in static map initializers (#91310)

When BPF object files are linked with bpftool, every symbol must be
accompanied by BTF info. Ensure that extern functions referenced by
global variable initializers are included in BTF.

The primary motivation is "static" initialization of PROG maps:

```c
extern int elsewhere(struct xdp_md *);

struct {
__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
__uint(max_entries, 1);
__type(key, int);
__type(value, int);
__array(values, int (struct xdp_md *));
} prog_map SEC(".maps") = { .values = { elsewhere } };
```

BPF backend needs debug info to produce BTF. Debug info is not
normally generated for external variables and functions. Previously, it
was solved differently for variables (collecting variable declarations
in ExternalDeclarations vector) and functions (logic invoked during
codegen in CGExpr.cpp).

This patch generalises ExternalDefclarations to include both function
and variable declarations. This change ensures that function references
are not missed no matter the context. Previously external functions
referenced in constant expressions lacked debug info.

show more ...

[clang][CodeGen] Remove unnecessary ShouldLinkFiles conditional (#96951)

We have reworked the bitcode linking option to no longer link twice if
post-optimization linking is requested. As such, we n

[clang][CodeGen] Remove unnecessary ShouldLinkFiles conditional (#96951)

We have reworked the bitcode linking option to no longer link twice if
post-optimization linking is requested. As such, we no longer need to
conditionally link bitcodes supplied via -mlink-bitcode-file, as there
is no danger of linking them twice

show more ...

CodeGen, IR: Add target-{cpu,features} attributes to functions created via createWithDefaultAttr().

Functions created with createWithDefaultAttr() need to have the
correct target-{cpu,features} attr

CodeGen, IR: Add target-{cpu,features} attributes to functions created via createWithDefaultAttr().

Functions created with createWithDefaultAttr() need to have the
correct target-{cpu,features} attributes to avoid miscompilations
such as using the wrong relocation type to access globals (missing
tagged-globals feature), clobbering registers specified via -ffixed-*
(missing reserve-* feature), and so on.

There's already a number of attributes copied from the module flags
onto functions created by createWithDefaultAttr(). I don't think
module flags are the right choice for the target attributes because
we don't need the conflict resolution logic between modules with
different target attributes, nor does it seem sensible to add it:
there's no unambiguously "correct" set of target attributes when
merging two modules with different attributes, and nor should there
be; it's perfectly valid for two modules to be compiled with different
target attributes, that's the whole reason why they are per-function.

This also implies that it's unnecessary to serialize the attributes in
bitcode, which implies that they shouldn't be stored on the module. We
can also observe that for the most part, createWithDefaultAttr()
is called from compiler passes such as sanitizers, coverage and
profiling passes that are part of the compile time pipeline, not
the LTO pipeline. This hints at a solution: we need to store the
attributes in a non-serialized location associated with the ambient
compilation context. Therefore in this patch I elected to store the
attributes on the LLVMContext.

There are calls to createWithDefaultAttr() in the NVPTX and AMDGPU
backends, and those calls would happen at LTO time. For those callers,
the bug still potentially exists and it would be necessary to refactor
them to create the functions at compile time if this issue is relevant
on those platforms.

Fixes #93633.

Reviewers: fmayer, MaskRay, eugenis

Reviewed By: MaskRay

Pull Request: https://github.com/llvm/llvm-project/pull/96721

show more ...

Fix unused private field warning (#91500)

After 11a6799740f8 "[clang][CodeGen] Omit pre-opt link when post-opt is
link requested (#85672)" I'm seeing a new warning:

> BackendConsumer.h:37:22: er

Fix unused private field warning (#91500)

After 11a6799740f8 "[clang][CodeGen] Omit pre-opt link when post-opt is
link requested (#85672)" I'm seeing a new warning:

> BackendConsumer.h:37:22: error: private field 'FileMgr' is not used
[-Werror,-Wunused-private-field]

Remove the field since it's no longer used.

show more ...

[clang][CodeGen] Omit pre-opt link when post-opt is link requested (#85672)

Currently, when the -relink-builtin-bitcodes-postop option is used we
link builtin bitcodes twice: once before optimizati

[clang][CodeGen] Omit pre-opt link when post-opt is link requested (#85672)

Currently, when the -relink-builtin-bitcodes-postop option is used we
link builtin bitcodes twice: once before optimization, and again after
optimization.

With this change, we omit the pre-opt linking when the option is set,
and we rename the option to the following:

-Xclang -mlink-builtin-bitcodes-postopt
(-Xclang -mno-link-builtin-bitcodes-postopt)

The goal of this change is to reduce compile time. We do lose the
theoretical benefits of pre-opt linking, but in practice these are small
than the overhead of linking twice. However we may be able to address
this in a future patch by adjusting the position of the builtin-bitcode
linking pass.

Compilations not setting the option are unaffected

show more ...

[C++20] [Modules] Introduce -fexperimental-modules-reduced-bmi (#85050)

This is the driver part of
https://github.com/llvm/llvm-project/pull/75894.

This patch introduces '-fexperimental-modules-

[C++20] [Modules] Introduce -fexperimental-modules-reduced-bmi (#85050)

This is the driver part of
https://github.com/llvm/llvm-project/pull/75894.

This patch introduces '-fexperimental-modules-reduced-bmi' to enable
generating the reduced BMI.

This patch did:
- When `-fexperimental-modules-reduced-bmi` is specified but
`--precompile` is not specified for a module unit, we'll skip the
precompile phase to avoid unnecessary two-phase compilation phases. Then
if `-c` is specified, we will generate the reduced BMI in CodeGenAction
as a by-product.
- When `-fexperimental-modules-reduced-bmi` is specified and
`--precompile` is specified, we will generate the reduced BMI in
GenerateModuleInterfaceAction as a by-product.
- When `-fexperimental-modules-reduced-bmi` is specified for a
non-module unit. We don't do anything nor try to give a warn. This is
more user friendly so that the end users can try to test and experiment
with the feature without asking help from the build systems.

The core design idea is that users should be able to enable this easily
with the existing cmake mechanisms.

The future plan for the flag is:
- Add this to clang19 and make it opt-in for 1~2 releases. It depends on
the testing feedback to decide how long we like to make it opt-in.
- Then we can announce the existing BMI generating may be deprecated and
suggesting people (end users or build systems) to enable this for 1~2
releases.
- Finally we will enable this by default. When that time comes, the term
`BMI` will refer to the reduced BMI today and the existing BMI will only
be meaningful to build systems which loves to support two phase
compilations.

I'll send release notes and document in seperate commits after this get
landed.

show more ...

[clang][CodeGen] Add missing error check (#81777)

Add missing error check. This resolves "error: variable 'Err' set but
not used" warnings

[clang][CodeGen] Shift relink option implementation away from module cloning (#81693)

We recently implemented a new option allowing relinking of bitcode
modules via the "-mllvm -relink-builtin-bitc

[clang][CodeGen] Shift relink option implementation away from module cloning (#81693)

We recently implemented a new option allowing relinking of bitcode
modules via the "-mllvm -relink-builtin-bitcode-postop"
option.

This implementation relied on llvm::CloneModule() in order to pass
copies to modules and preserve the original modules for later relinking.
However, cloning modules has been found to be prohibitively expensive,
significantly increasing compilation time for large bitcode libraries.

In this patch, we shift the relink option implementation to instead link
the original modules initially, and reload modules from the file system
if relinking is requested. This approach results in significantly
reduced overhead.

We accomplish this by creating a new ReloadModules() routine that can be
called from a BackendConsumer class, to mimic the behavior of
ASTConsumer's loadLinkModules(), but without access to the
CompilerInstance.

Because loading the bitcodes from the filesystem requires access to the
FileManager class, we also forward a reference to the CompilerInstance
class to the BackendConsumer. This mirrors what is already done for
several CompilerInstance members, such as TargetOptions and
CodeGenOptions.

Finally, we needed to add a const specifier to the
FileManager::getBufferForFile() routine to allow it to be called using
the const reference returned from CompilerInstance::getFileManager()

show more ...

[clang] Use StringRef::consume_front (NFC)

DiagnosticHandler: refactor error checking (#75889)

In LLVMContext::diagnose, set `HasErrors` for `DS_Error` so that all
derived `DiagnosticHandler` have correct `HasErrors` information.

An alte

DiagnosticHandler: refactor error checking (#75889)

In LLVMContext::diagnose, set `HasErrors` for `DS_Error` so that all
derived `DiagnosticHandler` have correct `HasErrors` information.

An alternative is to set `HasErrors` in
`DiagnosticHandler::handleDiagnostics`, but all derived
`handleDiagnostics` would have to call the base function.

show more ...

[LTO] Improve diagnostics handling when parsing module-level inline assembly (#75726)

Non-LTO compiles set the buffer name to "<inline asm>"
(`AsmPrinter::addInlineAsmDiagBuffer`) and pass diagnost

[LTO] Improve diagnostics handling when parsing module-level inline assembly (#75726)

Non-LTO compiles set the buffer name to "<inline asm>"
(`AsmPrinter::addInlineAsmDiagBuffer`) and pass diagnostics to
`ClangDiagnosticHandler` (through the `MCContext` handler in
`MachineModuleInfoWrapperPass::doInitialization`) to ensure that
the exit code is 1 in the presence of errors. In contrast, LTO compiles
spuriously succeed even if error messages are printed.

```
% cat a.c
void _start() {}
asm("unknown instruction");
% clang -c a.c
<inline asm>:1:1: error: invalid instruction mnemonic 'unknown'
1 | unknown instruction
| ^
1 error generated.
% clang -c -flto a.c; echo $? # -flto=thin is the same
error: invalid instruction mnemonic 'unknown'
unknown instruction
^~~~~~~
error: invalid instruction mnemonic 'unknown'
unknown instruction
^~~~~~~
0
```

`CollectAsmSymbols` parses inline assembly and is transitively called by
both `ModuleSummaryIndexAnalysis::run` and `WriteBitcodeToFile`, leading
to duplicate diagnostics.

This patch updates `CollectAsmSymbols` to be similar to non-LTO
compiles.
```
% clang -c -flto=thin a.c; echo $?
<inline asm>:1:1: error: invalid instruction mnemonic 'unknown'
1 | unknown instruction
| ^
1 errors generated.
1
```

The `HasErrors` check does not prevent duplicate warnings but assembler
warnings are very uncommon.

show more ...

[clang] Use StringRef::{starts,ends}_with (NFC) (#75149)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}:

[clang] Use StringRef::{starts,ends}_with (NFC) (#75149)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.

I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.

show more ...

[CodeGen] Add conditional to module cloning in bitcode linking (#72478)

Now that we have a commandline option dictating a second link step,
(-relink-builtin-bitcode-postop), we can condition the mo

[CodeGen] Add conditional to module cloning in bitcode linking (#72478)

Now that we have a commandline option dictating a second link step,
(-relink-builtin-bitcode-postop), we can condition the module creation
when linking in bitcode modules. This aims to improve performance by
avoiding unnecessary linking

show more ...

[CodeGen] Implement post-opt linking option for builtin bitocdes (#69371)

In this patch, we create a new ModulePass that mimics the LinkInModules
API from CodeGenAction.cpp, and a new command line

[CodeGen] Implement post-opt linking option for builtin bitocdes (#69371)

In this patch, we create a new ModulePass that mimics the LinkInModules
API from CodeGenAction.cpp, and a new command line option to enable the
pass. As part of the implementation, we needed to refactor the
BackendConsumer class definition into a new separate header (instead of
embedded in CodeGenAction.cpp). With this new pass, we can now re-link
bitcodes supplied via the -mlink-built-in bitcodes as part of the
RunOptimizationPipeline.

With the re-linking pass, we now handle cases where new device library
functions are introduced as part of the optimization pipeline.
Previously, these newly introduced functions (for example a fused sincos
call) would result in a linking error due to a missing function
definition. This new pass can be initiated via:

-mllvm -relink-builtin-bitcode-postop

Also note we intentionally exclude bitcodes supplied via the
-mlink-bitcode-file option from the second linking step

show more ...

[CodeGen] Support bitcode input containing multiple modules

When using -fsplit-lto-unit (explicitly specified or due to using
-fsanitize=cfi/-fwhole-program-vtables), the emitted LLVM IR contains a

[CodeGen] Support bitcode input containing multiple modules

When using -fsplit-lto-unit (explicitly specified or due to using
-fsanitize=cfi/-fwhole-program-vtables), the emitted LLVM IR contains a module
flag metadata `"EnableSplitLTOUnit"`. If a module contains both type metadata
and `"EnableSplitLTOUnit"`, `ThinLTOBitcodeWriter.cpp` will write two modules
into the bitcode file. Compiling the bitcode (not ThinLTO backend compilation)
will lead to an error due to `parseIR` requiring a single module.

```
% clang -flto=thin a.cc -c -o a.bc
% clang -c a.bc
% clang -fsplit-lto-unit -flto=thin a.cc -c -o a.bc
% clang -c a.bc
error: Expected a single module
1 error generated.
```

There are multiple ways to have just one module in a bitcode file
output: `-Xclang -fno-lto-unit`, not using features like `-fsanitize=cfi`,
using `-fsanitize=cfi` with `-fno-split-lto-unit`. I think whether a
bitcode input file contains 2 modules (internal implementation strategy)
should not be a criterion to require an additional driver option when
the user seek for a non-LTO compile action.

Let's place the extra module (if present) into CodeGenOptions::LinkBitcodeFiles
(originally for -cc1 -mlink-bitcode-file). Linker::linkModules will link the two
modules together. This patch makes the following commands work:

```
clang -S -emit-llvm a.bc
clang -S a.bc
clang -c a.bc
```

Reviewed By: ormris

Differential Revision: https://reviews.llvm.org/D154923

show more ...