[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used

[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used by
`ASTReader` now only holds `StringRef`s that point into the PCM file
buffer, and the full input file paths get resolved on demand.

The dependency scanner makes use of this in a bit of a roundabout way:
`ModuleDeps` now only holds (an owning copy of) the short unresolved
input file paths, which get resolved lazily. This can be a big win, I'm
seeing up to a 5% speedup.

show more ...

[clang][deps] Collect discovered module dependencies' Link Libraries (#93588)

This will allow scanner clients to be able to compute e.g. auto-linking
dependencies of the scanned translation unit.

[Driver] -include: deprecate probing .gch (#67084)

`-include a.h` probes `a.h.pch` and `a.h.gch`, if not found, falls back to
`a.h`. `.pch` is the preferred extension name. Probing .gch is supposed

[Driver] -include: deprecate probing .gch (#67084)

`-include a.h` probes `a.h.pch` and `a.h.gch`, if not found, falls back to
`a.h`. `.pch` is the preferred extension name. Probing .gch is supposed to
provide compatibility with build systems that do
```
clang -x c-header a.h -o out/a.h.gch
clang -include out/a.h -c a.c # out/a.h.gch is present while out/a.h is absent
```
(not sure what projects actually do this with Clang)

But it can often get in the way [^0][^1][^2] when GCC and Clang are mixed as the file
format is incompatible with GCC's. Let's deprecate .gch probing.
Some tests using `-include` are switched to `.pch`.
`test/PCH/pch-dir.c` shows the -Wdeprecated warning.

[^0]: https://discourse.llvm.org/t/how-to-have-clang-ignore-gch-directories/51835
[^1]: https://bugreports.qt.io/browse/QTCREATORBUG-22427
[^2]: https://gitlab.kitware.com/cmake/cmake/-/issues/22081

show more ...

[lit][AIX] Convert clang tests to use 'target={{.*}}-aix{{.*}}'

Part of the project to eliminate special handling for triples in lit
expressions.

Differential Revision: https://reviews.llvm.org/D13

[lit][AIX] Convert clang tests to use 'target={{.*}}-aix{{.*}}'

Part of the project to eliminate special handling for triples in lit
expressions.

Differential Revision: https://reviews.llvm.org/D137437

show more ...

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.

Original commit message follows.

---

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc1

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc17.

show more ...

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands,

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

[clang][deps] Allow switching between lazily/eagerly loaded PCMs

This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load

[clang][deps] Allow switching between lazily/eagerly loaded PCMs

This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.

To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D132066

show more ...

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines,

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D131420

show more ...

[clang][deps] Create lit substitution for deps-to-rsp

This patch gets rid of the ridiculous relative path we use to invoke the `module-deps-to-rsp.py` script and creates proper lit substitution, cle

[clang][deps] Create lit substitution for deps-to-rsp

This patch gets rid of the ridiculous relative path we use to invoke the `module-deps-to-rsp.py` script and creates proper lit substitution, cleaning up the tests.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D121525

show more ...

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `File

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `FileCheck`'s `-D` option.

Note that we use `%/t` instead of `%t` in order to normalize to forward slashes on Windows, which matches what we do with `sed 's:\\\\\?:/:g'`.

Differential Revision: https://reviews.llvm.org/D121516

show more ...

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation uni

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation unit command lines, the scanner only generates additional driver arguments the build system is expected to append to the original command line.

While this works most of the time, there are situations where that's not the case. For example with `-Wunused-command-line-argument`, Clang will complain about the `-fmodules-cache-path=` argument that's not being used in explicit modular builds. Combine that with `-Werror` and the build outright fails.

To prevent such failures, this patch changes the dependency scanner to return the full driver command line to compile the original translation unit. This gives us more opportunities to massage the arguments into something reasonable.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118986

show more ...

[NFC][AIX]Disable precompiled module file tests on AIX

The PCH reader looks for `__clangast` section in the precompiled module file, which is not present in the file on AIX, and we don't support wri

[NFC][AIX]Disable precompiled module file tests on AIX

The PCH reader looks for `__clangast` section in the precompiled module file, which is not present in the file on AIX, and we don't support writing this custom section in XCOFF yet.

Reviewed By: daltenty

Differential Revision: https://reviews.llvm.org/D114481

show more ...

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fn

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fno-implicit-modules`, `-fno-implicit-module-maps`), we don't need to load the original module map files at all. This patch stops emitting the `-fmodule-map-file=` arguments we don't need, saving some compilation time due to avoiding parsing such module maps and making the command line shorter.

Reviewed By: bnbarham

Differential Revision: https://reviews.llvm.org/D113473

show more ...

[clang][deps] Collect precompiled deps from submodules too

In this patch, the dependency scanner starts collecting precompiled dependencies from all encountered submodules, not only from top-level m

[clang][deps] Collect precompiled deps from submodules too

In this patch, the dependency scanner starts collecting precompiled dependencies from all encountered submodules, not only from top-level modules.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108540

show more ...