[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used

[clang][deps][modules] Allocate input file paths lazily (#114457)

This PR builds on top of #113984 and attempts to avoid allocating input
file paths eagerly. Instead, the `InputFileInfo` type used by
`ASTReader` now only holds `StringRef`s that point into the PCM file
buffer, and the full input file paths get resolved on demand.

The dependency scanner makes use of this in a bit of a roundabout way:
`ModuleDeps` now only holds (an owning copy of) the short unresolved
input file paths, which get resolved lazily. This can be a big win, I'm
seeing up to a 5% speedup.

show more ...

[clang][modules] Preserve the module map that allowed inferring (#113389)

With inferred modules, the dependency scanner takes care to replace the
fake "__inferred_module.map" path with the file tha

[clang][modules] Preserve the module map that allowed inferring (#113389)

With inferred modules, the dependency scanner takes care to replace the
fake "__inferred_module.map" path with the file that allowed the module
to be inferred. However, this only worked when such a module was
imported directly in the TU. Whenever such module got loaded
transitively, the scanner would fail to perform the replacement. This is
caused by the fact that PCM files are lossy and drop this information.

This patch makes sure that PCMs include this file for each submodule (in
the `SUBMODULE_DEFINITION` record), fixes one existing test with an
incorrect assertion, and does a little drive-by refactoring of
`ModuleMap`.

show more ...

[DependencyScanning] Avoid repeated hash lookups (NFC) (#111088)

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are doc

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.

show more ...

[clang][scan] Report module dependencies in topological order (#107474)

[clang][deps] Collect discovered module dependencies' Link Libraries (#93588)

This will allow scanner clients to be able to compute e.g. auto-linking
dependencies of the scanned translation unit.

[clang][modules] Only avoid pruning module maps when asked to (#89428)

Pruning non-affecting module maps is useful even when passing module
maps explicitly via `-fmodule-map-file=<path>`. For this

[clang][modules] Only avoid pruning module maps when asked to (#89428)

Pruning non-affecting module maps is useful even when passing module
maps explicitly via `-fmodule-map-file=<path>`. For this situation, this
patch reinstates the behavior we had prior to #87849. For the situation
where the explicit module map file arguments were generated by the
dependency scanner (which already pruned the non-affecting ones), this
patch introduces new `-cc1` flag
`-fno-modules-prune-non-affecting-module-map-files` that avoids the
extra work.

show more ...

[clang/DependencyScanning/ModuleDepCollector] Refactor part of `makeCommonInvocationForModuleBuild` into its own function (#88447)

The new function is about clearing out benign codegen options and c

[clang/DependencyScanning/ModuleDepCollector] Refactor part of `makeCommonInvocationForModuleBuild` into its own function (#88447)

The new function is about clearing out benign codegen options and can be
applied for PCH invocations as well.

show more ...

[clang][deps] Remove pgo profile flags from modules (#87724)

These are not necessary when not performing codegen.

[clang][ScanDeps] Allow PCHs to have different VFS overlays (#82294)

It turns out it's not that uncommon for real code to pass a different
set of VFSs while building a PCH than while using the PCH.

[clang][ScanDeps] Allow PCHs to have different VFS overlays (#82294)

It turns out it's not that uncommon for real code to pass a different
set of VFSs while building a PCH than while using the PCH. This can
cause problems as seen in `test/ClangScanDeps/optimize-vfs-pch.m`. If
you scan `compile-commands-tu-no-vfs-error.json` without -Werror and run
the resulting commands, Clang will emit a fatal error while trying to
emit a note saying that it can't find a remapped header.

This also adds textual tracking of VFSs for prebuilt modules that are
part of an included PCH, as the same issue can occur in a module we are
building if we drop VFSs. This has to be textual because we have no
guarantee the PCH had the same list of VFSs as the current TU.

This uses the `PrebuiltModuleListener` to collect `VFSOverlayFiles`
instead of trying to extract it out of a `serialization::ModuleFile`
each time it's needed. There's not a great way to just store a pointer
to the list of strings in the serialized AST.

show more ...

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
s

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
suggested module, regardless of whether it was actually imported or not.
The extra `bool ModuleImported` parameter then denotes whether the
header `#include` will be automatically translated into import the the
module.

The main change is in `clang/lib/Lex/PPDirectives.cpp`, where we take
care to not modify `SuggestedModule` after it's been populated by
`LookupHeaderIncludeOrImport()`. We now exclusively use the `SM`
(`ModuleToImport`) variable instead, which has been equivalent to
`SuggestedModule` until now. This allows us to use the original
non-modified `SuggestedModule` for the callback itself.

(This patch turns out to be necessary for
https://github.com/apple/llvm-project/pull/8011).

show more ...

[clang][DependencyScanner] Remove unused -fmodule-map-file arguments (#80090)

Since we already add a `-fmodule-map-file=` argument for every used
modulemap, we can remove all `ModuleMapFiles` entri

[clang][DependencyScanner] Remove unused -fmodule-map-file arguments (#80090)

Since we already add a `-fmodule-map-file=` argument for every used
modulemap, we can remove all `ModuleMapFiles` entries before adding
them.

This reduces the number of module variants when `-fmodule-map-file=`
appears on the original command line.

show more ...

[clang][DependencyScanner] Remove unused -ivfsoverlay files (#73734)

`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tre

[clang][DependencyScanner] Remove unused -ivfsoverlay files (#73734)

`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.

Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.

show more ...

[clang][modules] Strip LLVM options (#75405)

Currently, the dep scanner does not remove LLVM options from the
argument list.
Since LLVM options shouldn't affect the AST, it is safe to remove them

[clang][modules] Strip LLVM options (#75405)

Currently, the dep scanner does not remove LLVM options from the
argument list.
Since LLVM options shouldn't affect the AST, it is safe to remove them
all.

show more ...

[clang] Use StringRef::{starts,ends}_with (NFC) (#75149)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}:

[clang] Use StringRef::{starts,ends}_with (NFC) (#75149)

This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.

I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.

show more ...

[clang] NFC: Remove `OptionalFileEntryRefDegradesToFileEntryPtr` (#74899)

[clang][DependencyScanner] Include the working directory in the context hash (#73719)

The working directory is included in the PCM, but is not currently part
of the context hash. This causes proble

[clang][DependencyScanner] Include the working directory in the context hash (#73719)

The working directory is included in the PCM, but is not currently part
of the context hash. This causes problems because different builds of a
PCM with exactly the same command line can end up with different binary
content for a PCM. If a build system tracks tasks by both working
directory and command line, it may build a given PCM multiple times,
causing a "module file out of date" error when loading the PCM due to
different sizes.

show more ...

[clang][DependencyScanner] Remove all warning flags when suppressing warnings (#71612)

Since system modules don't emit most warnings, remove the warning flags
to increase module reuse.

[clang][DepScan] Make OptimizeArgs a bit mask enum and enable by default (#71588)

Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's current

[clang][DepScan] Make OptimizeArgs a bit mask enum and enable by default (#71588)

Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's currently only one such
optimization, but more will be added in followup commits.

show more ...

[clang-scan-deps] [P1689] Keep consistent behavior for make dependencies with clang (#69551)

Close https://github.com/llvm/llvm-project/issues/69439.

This patch tries to reuse the codes to genera

[clang-scan-deps] [P1689] Keep consistent behavior for make dependencies with clang (#69551)

Close https://github.com/llvm/llvm-project/issues/69439.

This patch tries to reuse the codes to generate make style dependencies
information with P1689 format directly.

show more ...

Use llvm::endianness{,::little,::native} (NFC)

Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
llvm::support::endianness wi

Use llvm::endianness{,::little,::native} (NFC)

Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
llvm::support::endianness with llvm::endianness.

show more ...

[clang][deps] NFCI: Use `FileEntryRef` in `ModuleDepCollectorPP`

[clang][deps] Generate command lines lazily (#65691)

This patch makes the generation of command lines for modular
dependencies lazy/on-demand. That operation is somewhat expensive and
prior to thi

[clang][deps] Generate command lines lazily (#65691)

This patch makes the generation of command lines for modular
dependencies lazy/on-demand. That operation is somewhat expensive and
prior to this patch used to be performed multiple times for the
identical `ModuleDeps` (i.e. when they were imported from multiple
different TUs).

show more ...

[clang][deps] Store common, partially-formed invocation (#65677)

We create one `CompilerInvocation` for each modular dependency we
discover. This means we create a lot of copies, even though most o

[clang][deps] Store common, partially-formed invocation (#65677)

We create one `CompilerInvocation` for each modular dependency we
discover. This means we create a lot of copies, even though most of the
invocation is the same between modules. This patch makes use of the
copy-on-write flavor of `CompilerInvocation` to share the common parts,
reducing memory usage and speeding up the scan.

show more ...

[clang] NFCI: Change returned LanguageOptions pointer to reference