[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used

[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used by
`ASTReader` now only holds `StringRef`s that point into the PCM file
buffer, and the full input file paths get resolved on demand.

The dependency scanner makes use of this in a bit of a roundabout way:
`ModuleDeps` now only holds (an owning copy of) the short unresolved
input file paths, which get resolved lazily. This can be a big win, I'm
seeing up to a 5% speedup.

show more ...

[clang][deps] Collect discovered module dependencies' Link Libraries (#93588)

This will allow scanner clients to be able to compute e.g. auto-linking
dependencies of the scanned translation unit.

[clang/DependencyScanning/ModuleDepCollector] Refactor part of `makeCommonInvocationForModuleBuild` into its own function (#88447)

The new function is about clearing out benign codegen options and c

[clang/DependencyScanning/ModuleDepCollector] Refactor part of `makeCommonInvocationForModuleBuild` into its own function (#88447)

The new function is about clearing out benign codegen options and can be
applied for PCH invocations as well.

show more ...

[clang][deps] Remove pgo profile flags from modules (#87724)

These are not necessary when not performing codegen.

[clang][deps] Remove -coverage-data-file and -coverage-notes-file from modules

When not performing codegen, we can strip the coverage-data-file and
coverage-notes-file options to improve canonicaliz

[clang][deps] Remove -coverage-data-file and -coverage-notes-file from modules

When not performing codegen, we can strip the coverage-data-file and
coverage-notes-file options to improve canonicalization.

rdar://107443796

Differential Revision: https://reviews.llvm.org/D147282

show more ...

[clang][deps] Remove more codegen options

Codegen options are typically unused by modules. Reset some of them to increase sharing between TUs with different flags.

Reviewed By: Bigcheese

Different

[clang][deps] Remove more codegen options

Codegen options are typically unused by modules. Reset some of them to increase sharing between TUs with different flags.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D135720

show more ...

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.

Original commit message follows.

---

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc1

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc17.

show more ...

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands,

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines,

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D131420

show more ...

[clang][deps] Override dependency and serialized diag files for modules

When building modules, override secondary outputs (dependency file,
dependency targets, serialized diagnostic file) in additio

[clang][deps] Override dependency and serialized diag files for modules

When building modules, override secondary outputs (dependency file,
dependency targets, serialized diagnostic file) in addition to the pcm
file path. This avoids inheriting per-TU command-line options that
cause non-determinism in the results (non-deterministic command-line for
the module build, non-determinism in which TU's .diag and .d files will
contain the module outputs). In clang-scan-deps we infer whether to
generate dependency or serialized diagnostic files based on an original
command-line. In a real build system this should be modeled explicitly.

Differential Revision: https://reviews.llvm.org/D129389

show more ...

[clang][deps] Further canonicalize implicit modules options in dep scan

Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for t

[clang][deps] Further canonicalize implicit modules options in dep scan

Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for the modules
cache path. This reduces uninteresting differences between
command-lines, which is particularly useful if there is a tool that can
cache the compilations.

Differential Revision: https://reviews.llvm.org/D127883

show more ...

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `File

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `FileCheck`'s `-D` option.

Note that we use `%/t` instead of `%t` in order to normalize to forward slashes on Windows, which matches what we do with `sed 's:\\\\\?:/:g'`.

Differential Revision: https://reviews.llvm.org/D121516

show more ...

[clang][deps] Reset some benign codegen options

Some command-line codegen arguments are likely to differ between identical modules discovered from different translation units. This patch removes the

[clang][deps] Reset some benign codegen options

Some command-line codegen arguments are likely to differ between identical modules discovered from different translation units. This patch removes them to make builds deterministic and/or reduce the number of built modules.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D112923

show more ...

[clang][deps] Reset non-modular language and preprocessor options

There are a number of language and preprocessor options that are reset in the `CompilerInvocation` that describes the build of an im

[clang][deps] Reset non-modular language and preprocessor options

There are a number of language and preprocessor options that are reset in the `CompilerInvocation` that describes the build of an implicit module. This patch uses the logic for explicit modules as well.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108710

show more ...