[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used

[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used by
`ASTReader` now only holds `StringRef`s that point into the PCM file
buffer, and the full input file paths get resolved on demand.

The dependency scanner makes use of this in a bit of a roundabout way:
`ModuleDeps` now only holds (an owning copy of) the short unresolved
input file paths, which get resolved lazily. This can be a big win, I'm
seeing up to a 5% speedup.

show more ...

[clang][deps] Collect discovered module dependencies' Link Libraries (#93588)

This will allow scanner clients to be able to compute e.g. auto-linking
dependencies of the scanned translation unit.

[clang][deps] Fix module context hash for constant strings

We were not hashing constant strings in the command-line, only ones that
required allocations. This was causing us to get the same hash acr

[clang][deps] Fix module context hash for constant strings

We were not hashing constant strings in the command-line, only ones that
required allocations. This was causing us to get the same hash across
different flag options.

rdar://101053855

Differential Revision: https://reviews.llvm.org/D143027

show more ...

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds

Reapply "[clang][deps] Split translation units into individual -cc1 or other commands"

Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.

Original commit message follows.

---

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc1

Revert "[clang][deps] Split translation units into individual -cc1 or other commands"

Failing on some bots, reverting until I can fix it.

This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc17.

show more ...

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands,

[clang][deps] Split translation units into individual -cc1 or other commands

Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.

This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.

In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.

Differential Revision: https://reviews.llvm.org/D132405

show more ...

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines,

[clang][deps] Always generate module paths

Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D131420

show more ...

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `File

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `FileCheck`'s `-D` option.

Note that we use `%/t` instead of `%t` in order to normalize to forward slashes on Windows, which matches what we do with `sed 's:\\\\\?:/:g'`.

Differential Revision: https://reviews.llvm.org/D121516

show more ...

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation uni

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation unit command lines, the scanner only generates additional driver arguments the build system is expected to append to the original command line.

While this works most of the time, there are situations where that's not the case. For example with `-Wunused-command-line-argument`, Clang will complain about the `-fmodules-cache-path=` argument that's not being used in explicit modular builds. Combine that with `-Werror` and the build outright fails.

To prevent such failures, this patch changes the dependency scanner to return the full driver command line to compile the original translation unit. This gives us more opportunities to massage the arguments into something reasonable.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118986

show more ...

[clang][deps] Fix fragile test

This test was failing on GreenDragon.

[clang][deps] Ensure reported context hash is strict

One of main goals of the dependency scanner is to be strict about module compatibility. This is achieved through strict context hash. This patch

[clang][deps] Ensure reported context hash is strict

One of main goals of the dependency scanner is to be strict about module compatibility. This is achieved through strict context hash. This patch ensures that strict context hash is enabled not only during the scan itself (and its minimized implicit build), but also when actually reporting the dependency.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111720

show more ...