[clang][deps] Print tracing VFS data (#108056)

Clang's `-cc1 -print-stats` shows lots of useful internal data including
basic `FileManager` stats. Since this layer caches some results, it is
uncle

[clang][deps] Print tracing VFS data (#108056)

Clang's `-cc1 -print-stats` shows lots of useful internal data including
basic `FileManager` stats. Since this layer caches some results, it is
unclear how that information translates to actual filesystem accesses.
This PR uses `llvm::vfs::TracingFileSystem` to provide that missing
information.

Similar mechanism is implemented for `clang-scan-deps`'s verbose mode
(`-v`). IO contention proved to be a real bottleneck a couple of times
already and this new feature should make those easier to detect in the
future. The tracing VFS is inserted below the caching FS and above the
real FS.

show more ...

[clang][DepScan] Make OptimizeArgs a bit mask enum and enable by default (#71588)

Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's current

[clang][DepScan] Make OptimizeArgs a bit mask enum and enable by default (#71588)

Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's currently only one such
optimization, but more will be added in followup commits.

show more ...

[clang][deps] Abolish FileManager sharing

This patch removes the ability of a dependency scanning worker to share a `FileManager` instance between individual scans. It's not sound and doesn't provid

[clang][deps] Abolish FileManager sharing

This patch removes the ability of a dependency scanning worker to share a `FileManager` instance between individual scans. It's not sound and doesn't provide performance benefits (due to the underlying caching VFS).

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D134976

show more ...

[clang][deps] Allow switching between lazily/eagerly loaded PCMs

This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load

[clang][deps] Allow switching between lazily/eagerly loaded PCMs

This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.

To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D132066

show more ...

[Tooling/DependencyScanning] Make skipping excluded PP ranges during dependency scanning the default

This is to improve maintenance a bit and remove need to maintain the additional option and relate

[Tooling/DependencyScanning] Make skipping excluded PP ranges during dependency scanning the default

This is to improve maintenance a bit and remove need to maintain the additional option and related code-paths.

Differential Revision: https://reviews.llvm.org/D124558

show more ...

[clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and c

[clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.

In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.

We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.

There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.

Depends on D102923.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D102488

show more ...

[clang][deps] Support object files

When a project uses PCH with explicit modules, the build will look like this:

1. scan PCH dependencies
2. explicitly build PCH
3. scan TU dependencies
4. explicit

[clang][deps] Support object files

When a project uses PCH with explicit modules, the build will look like this:

1. scan PCH dependencies
2. explicitly build PCH
3. scan TU dependencies
4. explicitly build TU

Step 2 produces an object file for the PCH, which the dependency scanner needs to read in step 3. This patch adds support for this.

The `clang-scan-deps` invocation in the attached test would fail without this change.

Depends on D103516.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D103519

show more ...

[clang][clang-scan-deps] Add support for extracting full module dependencies.

This is a recommit of d8a4ef0e685c with the nondeterminism fixed.

This adds experimental support for extracting a Clang

[clang][clang-scan-deps] Add support for extracting full module dependencies.

This is a recommit of d8a4ef0e685c with the nondeterminism fixed.

This adds experimental support for extracting a Clang module dependency graph
from a compilation database. The output format is experimental and will change.
It is currently a concatenation of JSON outputs for each compilation. Future
patches will change this to deduplicate modules between compilations.

Differential Revision: https://reviews.llvm.org/D69420

show more ...

Revert "[clang][clang-scan-deps] Add support for extracting full module dependencies."

This reverts commit d8a4ef0e685cec1fc73d4953b48220b649d05b40.

This commit broke some of the bots. I believe it

Revert "[clang][clang-scan-deps] Add support for extracting full module dependencies."

This reverts commit d8a4ef0e685cec1fc73d4953b48220b649d05b40.

This commit broke some of the bots. I believe it's due to nondeterminism. Will fix and recommit.

show more ...

[clang][clang-scan-deps] Add support for extracting full module dependencies.

This adds experimental support for extracting a Clang module dependency graph
from a compilation database. The output fo

[clang][clang-scan-deps] Add support for extracting full module dependencies.

This adds experimental support for extracting a Clang module dependency graph
from a compilation database. The output format is experimental and will change.
It is currently a concatenation of JSON outputs for each compilation. Future
patches will change this to deduplicate modules between compilations.

Differential Revision: https://reviews.llvm.org/D69420

show more ...

[clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization

This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preproc

[clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization

This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor
blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive.
The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens.

On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on
the compilation database for a recent LLVM and Clang (3511 files).

Differential Revision: https://reviews.llvm.org/D67127

llvm-svn: 371656

show more ...

[clang-scan-deps] reuse the file manager across invocations of
the dependency scanner on a single worker thread

This behavior can be controlled using the new `-reuse-filemanager` clang-scan-deps
opt

[clang-scan-deps] reuse the file manager across invocations of
the dependency scanner on a single worker thread

This behavior can be controlled using the new `-reuse-filemanager` clang-scan-deps
option. By default the file manager is reused.

The added test/ClangScanDeps/symlink.cpp is able to pass with
the reused filemanager after the related FileEntryRef changes
landed earlier. The test test/ClangScanDeps/subframework_header_dir_symlink.m
still fails when the file manager is reused (I run the FileCheck with not to
make it PASS). I will address this in a follow-up patch that improves
the DirectoryEntry name modelling in the FileManager.

llvm-svn: 370420

show more ...

[clang-scan-deps] Implementation of dependency scanner over minimized sources

This commit implements the fast dependency scanning mode in clang-scan-deps: the
preprocessing is done on files that are

[clang-scan-deps] Implementation of dependency scanner over minimized sources

This commit implements the fast dependency scanning mode in clang-scan-deps: the
preprocessing is done on files that are minimized using the dependency directives source minimizer.

A shared file system cache is used to ensure that the file system requests and source minimization
is performed only once. The cache assumes that the underlying filesystem won't change during the course
of the scan (or if it will, it will not affect the output), and it can't be evicted. This means that the
service and workers can be used for a single run of a dependency scanner, and can't be reused across multiple,
incremental runs. This is something that we'll most likely support in the future though.
Note that the driver still utilizes the underlying real filesystem.

This commit is also still missing the fast skipped PP block skipping optimization that I mentioned at EuroLLVM talk.
Additionally, the file manager is still not reused by the threads as well.

Differential Revision: https://reviews.llvm.org/D63907

llvm-svn: 368086

show more ...