[clang][modules] Timestamp PCM files when writing (#112452)

Clang uses timestamp files to track the last time an implicitly-built
PCM file was verified to be up-to-date with regard to its inputs. W

[clang][modules] Timestamp PCM files when writing (#112452)

Clang uses timestamp files to track the last time an implicitly-built
PCM file was verified to be up-to-date with regard to its inputs. With
`-fbuild-session-{file,timestamp}=` and
`-fmodules-validate-once-per-build-session` this reduces the number of
times a PCM file is checked per "build session".

The behavior I'm seeing with the current scheme is that when lots of
Clang instances wait for the same PCM to be built, they race to validate
it as soon as the file lock gets released, causing lots of concurrent
IO.

This patch makes it so that the timestamp is written by the same Clang
instance responsible for building the PCM while still holding the lock.
This makes it so that whenever a PCM file gets compiled, it's never
re-validated in the same build session.

I believe this is as sound as the current scheme. One thing to be aware
of is that there might be a time interval between accessing input file N
and writing the timestamp file, where changes to input files 0..<N would
not result in a rebuild. Since this is the case current scheme too, I'm
not too concerned about that.

I've seen this speed up `clang-scan-deps` by ~27%.

show more ...

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are doc

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.

show more ...

[clang] NFC: Remove `OptionalFileEntryRefDegradesToFileEntryPtr` (#74899)

[clang] NFCI: Make `ModuleFile::File` non-optional (#74892)

AFAICT, `ModuleFile::File` can be `std::nullopt` only for PCM files
loaded from the standard input. This patch starts setting that variab

[clang] NFCI: Make `ModuleFile::File` non-optional (#74892)

AFAICT, `ModuleFile::File` can be `std::nullopt` only for PCM files
loaded from the standard input. This patch starts setting that variable
to `FileManager::getSTDIN()` in that case, which makes it possible to
remove the optionality, and also simplifies code that actually reads the
file.

This is part of an effort to get rid of
`Optional{File,Directory}EntryRefDegradesTo{File,Directory}EntryPtr`.

show more ...

[clang] NFCI: Use `FileEntryRef` in `FileManager::getBufferForFile()`

[clang] NFCI: Use `FileEntryRef` in `ModuleManager`

[clang] Use DenseMapBase::lookup (NFC)

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.

I tested this with GCC 7.5, the oldest supported GCC I had around.

Differential Revision: https://reviews.llvm.org/D140332

show more ...

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.

On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).

show more ...

[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional

[clang] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of m

[clang] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[modules] Fix error "malformed or corrupted AST file: 'SourceLocation remap refers to unknown module...'".

When a framework can be found at a new location, all references to it in
the module cache b

[modules] Fix error "malformed or corrupted AST file: 'SourceLocation remap refers to unknown module...'".

When a framework can be found at a new location, all references to it in
the module cache become outdated. When we try to load such outdated .pcm
file, we shouldn't change any already loaded and processed modules.

If `Module` has `ASTFile`, it means we've read its AST block already and
it is too late to undo that. If `ASTFile` is `None`, there is no value
in setting it to `None` again. So we don't reset `ASTFile` in
`ModuleManager::removeModules` at all.

rdar://97216258

Differential Revision: https://reviews.llvm.org/D134249

show more ...

unique_ptrify the ModuleManager's VisitState linked list

[clang] Use range-based for loops with llvm::reverse (NFC)

[clang] Use llvm::erase_if (NFC)

Change Module::ASTFile and ModuleFile::File => Optional<FileEntryRef>, NFC

Change `Module::ASTFile` and `ModuleFile::File` to use
`Optional<FileEntryRef>` instead of `const FileEntry *`. One of many

Change Module::ASTFile and ModuleFile::File => Optional<FileEntryRef>, NFC

Change `Module::ASTFile` and `ModuleFile::File` to use
`Optional<FileEntryRef>` instead of `const FileEntry *`. One of many
steps toward removing `FileEntry::getName`.

Differential Revision: https://reviews.llvm.org/D89836

show more ...

ModuleManager: Simplify lookupModuleFile by only setting the out parameter once, NFC

Differential Revision: https://reviews.llvm.org/D89835

Perform an extra consistency check when searching ModuleManager's
cache for implicit modules.

The ModuleManager's use of FileEntry nodes as the keys for its map of
loaded modules is less than ideal.

Perform an extra consistency check when searching ModuleManager's
cache for implicit modules.

The ModuleManager's use of FileEntry nodes as the keys for its map of
loaded modules is less than ideal. Uniqueness for FileEntry nodes is
maintained by FileManager, which in turn uses inode numbers on hosts
that support that. When coupled with the module cache's proclivity for
turning over and deleting stale PCMs, this means entries for different
module files can wind up reusing the same underlying inode. When this
happens, subsequent accesses to the Modules map will disagree on the
ModuleFile associated with a given file.

In general, it is not sufficient to resolve this conundrum with a type
like FileEntryRef that stores the name of the FileEntry node on first
access because of path canonicalization issues. However, the paths
constructed for implicit module builds are fully under Clang's
control. We *can*, therefore, rely on their structure being consistent
across operating systems and across subsequent accesses to the Modules
map.

To mitigate the effects of inode reuse, perform an extra name check when
implicit modules are returned from the cache. This has the effect of
forcing reused FileEntry nodes to stomp over existing-but-stale entries
in the cache, which simulates a miss - exactly the desired behavior.

rdar://48443680

Patch by Robert Widmann!

Differential Revision: https://reviews.llvm.org/D86823

show more ...

[clang] Use IsVolatile=true and RequiresNullTerminator=false for PCMs

This change got missed while upstreaming
https://reviews.llvm.org/D77772. This is the part of that change that
actually passes t

[clang] Use IsVolatile=true and RequiresNullTerminator=false for PCMs

This change got missed while upstreaming
https://reviews.llvm.org/D77772. This is the part of that change that
actually passes the correct arguments when opening a PCM.

The test didn't catch this because it starts at the
`MemoryBuffer::getOpenFile` level. It's not really possible to test
`ModuleManager::addModule` itself to verify how the file was opened.

show more ...

Revert "[modules] Do not cache invalid state for modules that we attempted to load."
As per comment on https://reviews.llvm.org/D72860, it is suggested to
revert this change in the meantime, since it

Revert "[modules] Do not cache invalid state for modules that we attempted to load."
As per comment on https://reviews.llvm.org/D72860, it is suggested to
revert this change in the meantime, since it has introduced regression.

This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.

show more ...

[NFC] Remove trailing space

sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h,td}

[modules] Do not cache invalid state for modules that we attempted to load.

Partially reverts 0a2be46cfdb698fefcc860a56b47dde0884d5335 as it turned
out to cause redundant module rebuilds in multi-pr

[modules] Do not cache invalid state for modules that we attempted to load.

Partially reverts 0a2be46cfdb698fefcc860a56b47dde0884d5335 as it turned
out to cause redundant module rebuilds in multi-process incremental builds.
When a module was getting out of date, all compilation processes started at the
same time were marking it as `ToBuild`. So each process was building the same
module instead of checking if it was built by someone else and using that
result. In addition to the work duplication, contention on the same .pcm file
wasn't making builds faster.

Note that for a single-process build this change would cause redundant module
reads and validations. But reading a module is faster than building it and
multi-process builds are more common than single-process. So I'm willing to
make such a trade-off.

rdar://problem/54395127

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D72860

show more ...

clang/Modules: Move Serialization/Module.{h,cpp} to ModuleFile, NFC

Remove some cognitive load by renaming clang/Serialization/Module.h to
clang/Serialization/ModuleFile.h, since it declares the Mod

clang/Modules: Move Serialization/Module.{h,cpp} to ModuleFile, NFC

Remove some cognitive load by renaming clang/Serialization/Module.h to
clang/Serialization/ModuleFile.h, since it declares the ModuleFile
class. This also makes editing a bit easier, since the basename of the
file no long conflicts with clang/Basic/Module.h, which declares the
Module class. Also move lib/Serialization/Module.cpp to
lib/Serialization/ModuleFile.cpp.

show more ...

clang/Modules: Remove unused parameter from ModuleManager::removeModules

The other paremeters appear to be sufficient to determine which modules
have just been loaded and need to be removed, so stop

clang/Modules: Remove unused parameter from ModuleManager::removeModules

The other paremeters appear to be sufficient to determine which modules
have just been loaded and need to be removed, so stop collecting and
sending in that set explicitly.

show more ...

FileManager: Remove ShouldCloseOpenFile argument from getBufferForFile, NFC

Remove this dead code. We always close it.

llvm-svn: 370488