[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used

[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used by
`ASTReader` now only holds `StringRef`s that point into the PCM file
buffer, and the full input file paths get resolved on demand.

The dependency scanner makes use of this in a bit of a roundabout way:
`ModuleDeps` now only holds (an owning copy of) the short unresolved
input file paths, which get resolved lazily. This can be a big win, I'm
seeing up to a 5% speedup.

show more ...

[clang][deps] Collect discovered module dependencies' Link Libraries (#93588)

This will allow scanner clients to be able to compute e.g. auto-linking
dependencies of the scanned translation unit.

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.

Original commit message follows.

---

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc1

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc17.

show more ...

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands,

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

[clang][deps] Allow switching between lazily/eagerly loaded PCMs

This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load

[clang][deps] Allow switching between lazily/eagerly loaded PCMs

This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.

To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D132066

show more ...

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines,

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D131420

show more ...

[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special

[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`.
This patch only includes NFC renaming changes to make reviewing of the functionality changing parts easier.

Differential Revision: https://reviews.llvm.org/D125484

show more ...

Revert "[clang][deps] NFC: De-duplicate clang-cl tests"

This reverts commit 30cb49b44e4e2af2e0627b2aa33df0ba57ab9e55.
It caused test failures on Fuchsia Toolchain Mac builds:
https://luci-milo.appsp

Revert "[clang][deps] NFC: De-duplicate clang-cl tests"

This reverts commit 30cb49b44e4e2af2e0627b2aa33df0ba57ab9e55.
It caused test failures on Fuchsia Toolchain Mac builds:
https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-mac-x64/b8818888168677577537/overview

show more ...

[clang][deps] NFC: De-duplicate clang-cl tests

In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.

We don't need to duplicate all test cases, th

[clang][deps] NFC: De-duplicate clang-cl tests

In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.

We don't need to duplicate all test cases, though. Testing the few special cases we have in `clang-scan-deps` for `clang-cl` should be good enough:

1. Deducing output path (and therefore target name in our make output).
2. Ignoring `-Xclang` arguments in step 1.
3. Deducing resource directory by invoking the compiler executuable.

This test de-duplicates the extra clang-cl test cases.

Reviewed By: dexonsmith, saudi

Differential Revision: https://reviews.llvm.org/D121812

show more ...

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `File

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `FileCheck`'s `-D` option.

Note that we use `%/t` instead of `%t` in order to normalize to forward slashes on Windows, which matches what we do with `sed 's:\\\\\?:/:g'`.

Differential Revision: https://reviews.llvm.org/D121516

show more ...

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation uni

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation unit command lines, the scanner only generates additional driver arguments the build system is expected to append to the original command line.

While this works most of the time, there are situations where that's not the case. For example with `-Wunused-command-line-argument`, Clang will complain about the `-fmodules-cache-path=` argument that's not being used in explicit modular builds. Combine that with `-Werror` and the build outright fails.

To prevent such failures, this patch changes the dependency scanner to return the full driver command line to compile the original translation unit. This gives us more opportunities to massage the arguments into something reasonable.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118986

show more ...

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fn

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fno-implicit-modules`, `-fno-implicit-module-maps`), we don't need to load the original module map files at all. This patch stops emitting the `-fmodule-map-file=` arguments we don't need, saving some compilation time due to avoiding parsing such module maps and making the command line shorter.

Reviewed By: bnbarham

Differential Revision: https://reviews.llvm.org/D113473

show more ...

[clang-scan-deps] ignore top-level module dependencies that aren't actually imported

Whenever -fmodule-name=top_level_module name is parsed, and clang actually tries to
import top_level_module, the

[clang-scan-deps] ignore top-level module dependencies that aren't actually imported

Whenever -fmodule-name=top_level_module name is parsed, and clang actually tries to
import top_level_module, the headers are imported textually and the module isn't actually
built. However, the dependency scanner could still record it as a potential dependency
if the module was reimported and thus recorded by the preprocessor callbacks.
This change avoids collecting this kind of module as a dependency by verifying that we don't
collect top level modules without actual PCM files.

Differential Revision: https://reviews.llvm.org/D106100

show more ...