Revert "[modules] Do not cache invalid state for modules that we attempted to load."
As per comment on https://reviews.llvm.org/D72860, it is suggested to
revert this change in the meantime, since it

Revert "[modules] Do not cache invalid state for modules that we attempted to load."
As per comment on https://reviews.llvm.org/D72860, it is suggested to
revert this change in the meantime, since it has introduced regression.

This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.

show more ...

[modules] Do not cache invalid state for modules that we attempted to load.

Partially reverts 0a2be46cfdb698fefcc860a56b47dde0884d5335 as it turned
out to cause redundant module rebuilds in multi-pr

[modules] Do not cache invalid state for modules that we attempted to load.

Partially reverts 0a2be46cfdb698fefcc860a56b47dde0884d5335 as it turned
out to cause redundant module rebuilds in multi-process incremental builds.
When a module was getting out of date, all compilation processes started at the
same time were marking it as `ToBuild`. So each process was building the same
module instead of checking if it was built by someone else and using that
result. In addition to the work duplication, contention on the same .pcm file
wasn't making builds faster.

Note that for a single-process build this change would cause redundant module
reads and validations. But reading a module is faster than building it and
multi-process builds are more common than single-process. So I'm willing to
make such a trade-off.

rdar://problem/54395127

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D72860

show more ...

Stop relying on allocator behaviour in modules unit test

Another fixup for r355778 for Windows bots, this time to stop
accidentally relying on allocator behaviour for the test to pass.

llvm-svn: 35

Stop relying on allocator behaviour in modules unit test

Another fixup for r355778 for Windows bots, this time to stop
accidentally relying on allocator behaviour for the test to pass.

llvm-svn: 355780

show more ...

Modules: Invalidate out-of-date PCMs as they're discovered

Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
im

Modules: Invalidate out-of-date PCMs as they're discovered

Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.

Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.

Now the cache explicitly tracks module state, where each module must be
one of:

- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.

Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.

The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)

The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.

For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.

Example
=======

To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:

TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'

If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:

TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'

After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:

TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'

The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.

rdar://problem/48545366

llvm-svn: 355778

show more ...

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module b

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.

Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.

llvm-svn: 355777

show more ...