Modules: Cache PCMs in memory and avoid a use-after-free

Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back

Modules: Cache PCMs in memory and avoid a use-after-free

Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands). Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly. Furthermore, the LockFileManager is not
robust in every environment. Other -cc1 commands can stall until
timeout (after about eight minutes).

This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack. The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module. Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout. Perhaps we should
reconsider blocking at all.

This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.

The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename. Its ownership is shared by the
CompilerInstance and ModuleManager.

- The ModuleManager stores PCMs there that it loads from disk, never
touching the disk if the cache is hot.

- When modules fail to validate, they're removed from the cache.

- When a CompilerInstance is spawned to build a new module, each
already-loaded PCM is assumed to be valid, and is frozen to avoid
the use-after-free.

- Any newly-built module is written directly to the cache to avoid the
round-trip to the filesystem, making lock files unnecessary for
correctness.

Original patch by Manman Ren; most testcases by Adrian Prantl!

llvm-svn: 298165

show more ...

shared_ptrify (from InclusiveRefCntPtr) HeaderSearchOptions

llvm-svn: 291202

Move PreprocessorOptions to std::shared_ptr from IntrusiveRefCntPtr

llvm-svn: 291160

Make TargetInfo store an actual DataLayout instead of a string.

Use it to calculate UserLabelPrefix, instead of specifying it (often
incorrectly).

Note that the *actual* user label prefix has alway

Make TargetInfo store an actual DataLayout instead of a string.

Use it to calculate UserLabelPrefix, instead of specifying it (often
incorrectly).

Note that the *actual* user label prefix has always come from the
DataLayout, and is handled within LLVM. The main thing clang's
TargetInfo::UserLabelPrefix did was to set the #define value. Having
these be different from each-other is just silly.

Differential Revision: http://reviews.llvm.org/D17183

llvm-svn: 262737

show more ...

Fix -Wextra-semi warnings.

Patch by Eugene Zelenko!

Differential Revision: http://reviews.llvm.org/D11401

llvm-svn: 242931

Revert r240270 ("Fixed/added namespace ending comments using clang-tidy").

llvm-svn: 240353

Fixed/added namespace ending comments using clang-tidy. NFC

The patch is generated using this command:

$ tools/extra/clang-tidy/tool/run-clang-tidy.py -fix \
-checks=-*,llvm-namespace-comme

Fixed/added namespace ending comments using clang-tidy. NFC

The patch is generated using this command:

$ tools/extra/clang-tidy/tool/run-clang-tidy.py -fix \
-checks=-*,llvm-namespace-comment -header-filter='llvm/.*|clang/.*' \
work/llvm/tools/clang

To reduce churn, not touching namespaces spanning less than 10 lines.

llvm-svn: 240270

show more ...

Switch PPCallbacks to take the new MacroDefinition instead of MacroDirective*, in order to preserve full information on module macro expansion.

llvm-svn: 236404

[modules] Start moving the module visibility information off the Module itself.

It has no place there; it's not a property of the Module, and it makes
restoring the visibility set when we leave a su

[modules] Start moving the module visibility information off the Module itself.

It has no place there; it's not a property of the Module, and it makes
restoring the visibility set when we leave a submodule more difficult.

llvm-svn: 236300

show more ...

Use 'override/final' instead of 'virtual' for overridden methods

Summary:
The patch is generated using clang-tidy misc-use-override check.

This command was used:

tools/clang/tools/extra/clang-ti

Use 'override/final' instead of 'virtual' for overridden methods

Summary:
The patch is generated using clang-tidy misc-use-override check.

This command was used:

tools/clang/tools/extra/clang-tidy/tool/run-clang-tidy.py \
-checks='-*,misc-use-override' -header-filter='llvm|clang' -j=32 -fix

Reviewers: dblaikie

Reviewed By: dblaikie

Subscribers: klimek, cfe-commits

Differential Revision: http://reviews.llvm.org/D8926

llvm-svn: 234678

show more ...

Unique_ptrify PPCallbacks ownership.

Unique_ptr creation stil needs to be moved earlier at some of the call sites.

llvm-svn: 217474

unique_ptrify SourceManager::createFileID

llvm-svn: 216715

Overload SourceManager::overrideFileContents so that unconditionally passing ownership is explicitly done using unique_ptr.

Only those callers who are dynamically passing ownership should need the
3

Overload SourceManager::overrideFileContents so that unconditionally passing ownership is explicitly done using unique_ptr.

Only those callers who are dynamically passing ownership should need the
3 argument form. Those accepting the default ("do pass ownership")
should do so explicitly with a unique_ptr now.

llvm-svn: 216614

show more ...

Update for LLVM api change.

llvm-svn: 216585

Use non-intrusive refcounting for TargetOptions

llvm-svn: 212388

[C++11] Use 'nullptr'. Unittests edition.

llvm-svn: 210423

Remove the last remaining llvm/Config/config.h includes

This corrects long-standing misuses of LLVM's internal config.h.

In most cases the public llvm-config.h header was intended and we can now
re

Remove the last remaining llvm/Config/config.h includes

This corrects long-standing misuses of LLVM's internal config.h.

In most cases the public llvm-config.h header was intended and we can now
remove the old hacks thanks to LLVM r210144.

The config.h header is private, won't be installed and should no longer be
included by clang or other modules.

llvm-svn: 210145

show more ...

SourceManager: Use setMainFileID() consistently

Eliminate createMainFileID() / createMainFileIDForMemBuffer() utility
functions. These didn't add much convenience and conflated two distinct
operatio

SourceManager: Use setMainFileID() consistently

Eliminate createMainFileID() / createMainFileIDForMemBuffer() utility
functions. These didn't add much convenience and conflated two distinct
operations.

This change makes things easier to follow by providing a consistent interface
and getting rid of a bunch of cast-to-voids.

llvm-svn: 209266

show more ...

Reformat code following Preprocessor constructor updates

Landing this separately to make the previous commits easy to follow at home.

llvm-svn: 207826

Factor TargetInfo pointer/DelayInitialization bool pair out of Preprocessor ctor

The Preprocessor::Initialize() function already offers a clear interface to
achieve this, further reducing the confus

Factor TargetInfo pointer/DelayInitialization bool pair out of Preprocessor ctor

The Preprocessor::Initialize() function already offers a clear interface to
achieve this, further reducing the confusing number of states a newly
constructed preprocessor can have.

llvm-svn: 207825

show more ...

Quick fix for layering that broke shared library build.

llvm-svn: 207011

Initial implementation of -modules-earch-all option, for searching for symbols in non-imported modules.

llvm-svn: 206977

Use the same SourceManager for ModuleMaps and compilations.

This allows using virtual file mappings on the original SourceManager to
map in virtual module.map files. Without this patch, the ModuleMa

Use the same SourceManager for ModuleMaps and compilations.

This allows using virtual file mappings on the original SourceManager to
map in virtual module.map files. Without this patch, the ModuleMap
search will find a module.map file (as the FileEntry exists in the
FileManager), but will be unable to get the content from the
SourceManager (as ModuleMap previously created its own SourceManager).

Two problems needed to be fixed which this patch exposed:

1. Storing the inferred module map
When writing out a module, the ASTWriter stores the names of the files
in the main source manager; when loading the AST again, the ASTReader
errs out if such a file is found missing, unless it is overridden.
Previously CompilerInstance's compileModule method would store the
inferred module map to a temporary file; the problem with this approach
is that now that the module map is handled by the main source manager,
the ASTWriter stores the name of the temporary module map as source to
the compilation; later, when the module is loaded, the temporary file
has already been deleted, which leads to a compilation error. This patch
changes the inferred module map to instead inject a virtual file into
the source manager. This both saves some disk IO, and works with how the
ASTWriter/ASTReader handle overridden source files.

2. Changing test input in test/Modules/Inputs/*
Now that the module map file is handled by the main source manager, the
VerifyDiagnosticConsumer will not ignore diagnostics created while
parsing the module map file. The module test test/Modules/renamed.m uses
-I test/Modules/Inputs and triggers recursive loading of all module maps
in test/Modules/Inputs, some of which had conflicting names, thus
leading errors while parsing the module maps. Those diagnostics already
occur on trunk, but before this patch they would not break the test, as
they were ignored by the VerifyDiagnosticConsumer. This patch thus
changes the module maps that have been recently introduced which broke
the invariant of compatible modules maps in test/Modules/Inputs.

llvm-svn: 193314

show more ...

[Preprocessor] For the MacroExpands preprocessor callback, also pass the MacroArgs object that provides information about
the argument tokens for a function macro.

llvm-svn: 181065

<rdar://problem/12368093> Extend module maps with a 'conflict' declaration, and warn when a newly-imported module conflicts with an already-imported module.

llvm-svn: 177577