clang/Modules: Refactor CompilerInstance::loadModule, NFC

Refactor the logic on CompilerInstance::loadModule and a couple of
surrounding methods in order to clarify what's going on.

- Rename Module

clang/Modules: Refactor CompilerInstance::loadModule, NFC

Refactor the logic on CompilerInstance::loadModule and a couple of
surrounding methods in order to clarify what's going on.

- Rename ModuleLoader::loadModuleFromSource to compileModuleFromSource
and fix its documentation, since it never loads a module. It just
creates/compiles one.
- Rename one of the overloads of compileModuleImpl to compileModule,
making it more obvious which one calls the other.
- Rename compileAndLoadModule to compileModuleAndReadAST. This
clarifies the relationship between this helper and its caller,
CompilerInstance::loadModule (the old name implied the opposite
relationship). It also (correctly) indicates that more needs to be
done to load the module than this function is responsible for.
- Split findOrCompileModuleAndReadAST out of loadModule. Besides
reducing nesting for this code thanks to early returns and the like,
this refactor clarifies the logic in loadModule, particularly around
calls to ModuleMap::cacheModuleLoad and
ModuleMap::getCachedModuleLoad. findOrCompileModuleAndReadAST also
breaks early if the initial ReadAST call returns Missing or OutOfDate,
allowing the last ditch call to compileModuleAndReadAST to come at the
end of the function body.
- Additionally split out selectModuleSource, clarifying the logic
due to early returns.
- Add ModuleLoadResult::isNormal and OtherUncachedFailure, so that
loadModule knows whether to cache the result.
OtherUncachedFailure was added to keep this patch NFC, but there's
a chance that these cases were uncached by accident, through
copy/paste/modify failures. These should be audited as a
follow-up (maybe we can eliminate this case).
- Do *not* lift the setting of `ModuleLoadFailed = true` to
loadModule because there isn't a clear pattern for when it's set.
This should be reconsidered in a follow-up, in case it would be
correct to set `ModuleLoadFailed` whenever no module is returned
and the result is either Normal or OtherUncachedFailure.
- Add some header documentation where it was missing, and fix it where
it was wrong.

This should have no functionality change.

https://reviews.llvm.org/D70556

show more ...

clang/Modules: Early return in CompilerInstance::createModuleManager, NFC

Reduce nesting with an early `return`.

clang/Modules: Bring back optimization lost in 31e14f41a21f

31e14f41a21f9016050a20f07d5da03db2e8c13e accidentally dropped caching of
failed module loads. This brings it back by making
ModuleMap::ge

clang/Modules: Bring back optimization lost in 31e14f41a21f

31e14f41a21f9016050a20f07d5da03db2e8c13e accidentally dropped caching of
failed module loads. This brings it back by making
ModuleMap::getCachedModuleLoad return an Optional.

show more ...

clang/Modules: Sink CompilerInstance::KnownModules into ModuleMap

Avoid use-after-frees when FrontendAction::BeginSourceFile is called
twice on the same CompilerInstance by sinking
CompilerInstance:

clang/Modules: Sink CompilerInstance::KnownModules into ModuleMap

Avoid use-after-frees when FrontendAction::BeginSourceFile is called
twice on the same CompilerInstance by sinking
CompilerInstance::KnownModules into ModuleMap. On the way, rename the
map to CachedModuleLoads. I considered (but rejected) merging this with
ModuleMap::Modules, since that only has top-level modules and this map
includes submodules.

This is an alternative to https://reviews.llvm.org/D58497. Thanks to
nemanjai for the detailed analysis of the problem!

show more ...

[Clang] Add VerboseOutputStream to CompilerInstance

Remove one instance of a hardcoded output stream in
CompilerInstance::ExecuteAction. There are still other cases of output
being hard-coded to sta

[Clang] Add VerboseOutputStream to CompilerInstance

Remove one instance of a hardcoded output stream in
CompilerInstance::ExecuteAction. There are still other cases of output
being hard-coded to standard streams in ExecuteCompilerInvocation, but
this patch covers the case when no flags like -version or -help are
passed, namely the "X warnings and Y errors generated." diagnostic.

Differential Revision: https://reviews.llvm.org/D53768

llvm-svn: 375442

show more ...

Replace platform-dependent `stat` with `llvm::sys::fs::status`. NFC intended.

Reviewers: bruno, sammccall

Reviewed By: sammccall

Subscribers: jkorous, dexonsmith, arphaman, ributzka, cfe-commits

Replace platform-dependent `stat` with `llvm::sys::fs::status`. NFC intended.

Reviewers: bruno, sammccall

Reviewed By: sammccall

Subscribers: jkorous, dexonsmith, arphaman, ributzka, cfe-commits

Differential Revision: https://reviews.llvm.org/D69011

llvm-svn: 375031

show more ...

Reapply: [Modules][PCH] Hash input files content

Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the

Reapply: [Modules][PCH] Hash input files content

Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:

- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.

This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.

I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.

Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".

rdar://problem/29320105

Reviewers: dexonsmith, arphaman, rsmith, aprantl

Subscribers: jkorous, cfe-commits, ributzka

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67249

> llvm-svn: 374841

llvm-svn: 374895

show more ...

Temporarily Revert [Modules][PCH] Hash input files content
as it's breaking a few bots.

This reverts r374841 (git commit 2a1386c81de504b5bda44fbecf3f7b4cdfd748fc)

llvm-svn: 374842

[Modules][PCH] Hash input files content

Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content

[Modules][PCH] Hash input files content

Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:

- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.

This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.

I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.

Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".

rdar://problem/29320105

Reviewers: dexonsmith, arphaman, rsmith, aprantl

Subscribers: jkorous, cfe-commits, ributzka

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67249

llvm-svn: 374841

show more ...

FileManager: Use llvm::Expected in new getFileRef API

`FileManager::getFileRef` is a modern API which we expect to convert to
over time. We should modernize the error handling as well, using
`llvm:

FileManager: Use llvm::Expected in new getFileRef API

`FileManager::getFileRef` is a modern API which we expect to convert to
over time. We should modernize the error handling as well, using
`llvm::Expected` instead of `llvm::ErrorOr`, to help clients that care
about errors to ensure nothing is missed.

However, not all clients care. I've also added another path for those
that don't:

- `FileEntryRef` is now copy- and move-assignable (using a pointer
instead of a reference).
- `FileManager::getOptionalFileRef` returns an `llvm::Optional` instead
of `llvm::Expected`.
- Added an `llvm::expectedToOptional` utility in case this is useful
elsewhere.

https://reviews.llvm.org/D66705

llvm-svn: 369943

show more ...

Improve behavior in the case of stack exhaustion.

Summary:
Clang performs various recursive operations (such as template instantiation),
and may use non-trivial amounts of stack space in each recurs

Improve behavior in the case of stack exhaustion.

Summary:
Clang performs various recursive operations (such as template instantiation),
and may use non-trivial amounts of stack space in each recursive step (for
instance, due to recursive AST walks). While we try to keep the stack space
used by such steps to a minimum and we have explicit limits on the number of
such steps we perform, it's impractical to guarantee that we won't blow out the
stack on deeply recursive template instantiations on complex ASTs, even with
only a moderately high instantiation depth limit.

The user experience in these cases is generally terrible: we crash with
no hint of what went wrong. Under this patch, we attempt to do better:

* Detect when the stack is nearly exhausted, and produce a warning with a
nice template instantiation backtrace, telling the user that we might
run slowly or crash.

* For cases where we're forced to trigger recursive template
instantiation in arbitrarily-deeply-nested contexts, check whether
we're nearly out of stack space and allocate a new stack (by spawning
a new thread) after producing the warning.

Reviewers: rnk, aaron.ballman

Subscribers: mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66361

llvm-svn: 369940

show more ...

Introduce FileEntryRef and use it when handling includes to report correct dependencies
when the FileManager is reused across invocations

This commit introduces a parallel API to FileManager's getFi

Introduce FileEntryRef and use it when handling includes to report correct dependencies
when the FileManager is reused across invocations

This commit introduces a parallel API to FileManager's getFile: getFileEntryRef, which returns
a reference to the FileEntry, and the name that was used to access the file. In the case of
a VFS with 'use-external-names', the FileEntyRef contains the external name of the file,
not the filename that was used to access it.

The new API is adopted only in the HeaderSearch and Preprocessor for include file lookup, so that the
accessed path can be propagated to SourceManager's FileInfo. SourceManager's FileInfo now can report this accessed path, using
the new getName method. This API is then adopted in the dependency collector, which now correctly reports dependencies when a file
is included both using a symlink and a real path in the case when the FileManager is reused across multiple Preprocessor invocations.

Note that this patch does not fix all dependency collector issues, as the same problem is still present in other cases when dependencies
are obtained using FileSkipped, InclusionDirective, and HasInclude. This will be fixed in follow-up commits.

Differential Revision: https://reviews.llvm.org/D65907

llvm-svn: 369680

show more ...

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
o

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942

show more ...

Move LangStandard*, InputKind::Language to Basic

This patch is a prerequisite for using LangStandard from Driver in
https://reviews.llvm.org/D64793.

It moves LangStandard* and InputKind::Language t

Move LangStandard*, InputKind::Language to Basic

This patch is a prerequisite for using LangStandard from Driver in
https://reviews.llvm.org/D64793.

It moves LangStandard* and InputKind::Language to Basic. It is mostly
mechanical, with only a few changes of note:

- enum Language has been changed into enum class Language : uint8_t to
avoid a clash between OpenCL in enum Language and OpenCL in enum
LangFeatures and not to increase the size of class InputKind.

- Now that getLangStandardForName, which is currently unused, also checks
both canonical and alias names, I've introduced a helper getLangKind
which factors out a code pattern already used 3 times.

The patch has been tested on x86_64-pc-solaris2.11, sparcv9-sun-solaris2.11,
and x86_64-pc-linux-gnu.

There's a companion patch for lldb which uses LangStandard.h
(https://reviews.llvm.org/D65717).

While polly includes isl which in turn uses InputKind::C, that part of the
code isn't even built inside the llvm tree. I've posted a patch to allow
for both InputKind::C and Language::C upstream
(https://groups.google.com/forum/#!topic/isl-development/6oEvNWOSQFE).

Differential Revision: https://reviews.llvm.org/D65562

llvm-svn: 367864

show more ...

Rename F_{None,Text,Append} to OF_{None,Text,Append}. NFC

F_{None,Text,Append} are kept for compatibility since r334221.

llvm-svn: 367800

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616

show more ...

BitStream reader: propagate errors

The bitstream reader handles errors poorly. This has two effects:

* Bugs in file handling (especially modules) manifest as an "unexpected end of
file" crash

BitStream reader: propagate errors

The bitstream reader handles errors poorly. This has two effects:

* Bugs in file handling (especially modules) manifest as an "unexpected end of
file" crash
* Users of clang as a library end up aborting because the code unconditionally
calls `report_fatal_error`

The bitstream reader should be more resilient and return Expected / Error as
soon as an error is encountered, not way late like it does now. This patch
starts doing so and adopting the error handling where I think it makes sense.
There's plenty more to do: this patch propagates errors to be minimally useful,
and follow-ups will propagate them further and improve diagnostics.

https://bugs.llvm.org/show_bug.cgi?id=42311
<rdar://problem/33159405>

Differential Revision: https://reviews.llvm.org/D63518

llvm-svn: 364464

show more ...

Unify DependencyFileGenerator class and DependencyCollector interface (NFCI)

Make DependencyFileGenerator a DependencyCollector as it was intended when
DependencyCollector was introduced. The missin

Unify DependencyFileGenerator class and DependencyCollector interface (NFCI)

Make DependencyFileGenerator a DependencyCollector as it was intended when
DependencyCollector was introduced. The missing PPCallbacks overrides are added to
the DependencyCollector as well.

This change will allow clang-scan-deps to access the produced dependencies without
writing them out to .d files to disk, so that it will be able collate them and
report them to the user.

Differential Revision: https://reviews.llvm.org/D63290

llvm-svn: 363840

show more ...

[Frontend] SetUpDiagnosticLog should handle unowned diagnostic consumer
in the compiler

The function SetUpDiagnosticLog that was called from createDiagnostics didn't
handle the case where the diagno

[Frontend] SetUpDiagnosticLog should handle unowned diagnostic consumer
in the compiler

The function SetUpDiagnosticLog that was called from createDiagnostics didn't
handle the case where the diagnostics engine didn't own the diagnostics consumer.
This is a potential problem for a clang tool, in particular some of the follow-up
patches for clang-scan-deps will need this fix.

Differential Revision: https://reviews.llvm.org/D63101

llvm-svn: 363009

show more ...

[CodeComplete] Remove obsolete isOutputBinary().

Summary:
It's never set to true. Its only effect would be to set stdout to binary mode.
Hopefully we have better ways of doing this by now :-)

Revie

[CodeComplete] Remove obsolete isOutputBinary().

Summary:
It's never set to true. Its only effect would be to set stdout to binary mode.
Hopefully we have better ways of doing this by now :-)

Reviewers: hokein

Subscribers: jkorous, arphaman, kadircet, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D60871

llvm-svn: 358696

show more ...

Adds `-ftime-trace` option to clang that produces Chrome `chrome://tracing` compatible JSON profiling output dumps.

This change adds hierarchical "time trace" profiling blocks that can be visualized

Adds `-ftime-trace` option to clang that produces Chrome `chrome://tracing` compatible JSON profiling output dumps.

This change adds hierarchical "time trace" profiling blocks that can be visualized in Chrome, in a "flame chart" style. Each profiling block can have a "detail" string that for example indicates the file being processed, template name being instantiated, function being optimized etc.

This is taken from GitHub PR: https://github.com/aras-p/llvm-project-20170507/pull/2

Patch by Aras Pranckevičius.

Differential Revision: https://reviews.llvm.org/D58675

llvm-svn: 357340

show more ...

Basic: Return a reference from FileManager::getVirtualFileSystem, NFC

FileManager constructs a VFS in its constructor if it isn't passed one,
and there's no way to reset it. Make that contract clea

Basic: Return a reference from FileManager::getVirtualFileSystem, NFC

FileManager constructs a VFS in its constructor if it isn't passed one,
and there's no way to reset it. Make that contract clear by returning a
reference from its accessor.

https://reviews.llvm.org/D59388

llvm-svn: 357038

show more ...

Frontend: Remove CompilerInstance::VirtualFileSystem, NFC

Remove CompilerInstance::VirtualFileSystem and
CompilerInstance::setVirtualFileSystem, instead relying on the VFS in
the FileManager. Compi

Frontend: Remove CompilerInstance::VirtualFileSystem, NFC

Remove CompilerInstance::VirtualFileSystem and
CompilerInstance::setVirtualFileSystem, instead relying on the VFS in
the FileManager. CompilerInstance and its clients already went to some
trouble to make these match. Now they are guaranteed to match.

As part of this, I added a VFS parameter (defaults to nullptr) to
CompilerInstance::createFileManager, to avoid repeating construction
logic in clients that just wanted to customize the VFS.

https://reviews.llvm.org/D59377

llvm-svn: 357037

show more ...

Modules: Invalidate out-of-date PCMs as they're discovered

Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
im

Modules: Invalidate out-of-date PCMs as they're discovered

Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.

Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.

Now the cache explicitly tracks module state, where each module must be
one of:

- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.

Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.

The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)

The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.

For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.

Example
=======

To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:

TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'

If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:

TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'

After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:

TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'

The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.

rdar://problem/48545366

llvm-svn: 355778

show more ...

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module b

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.

Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.

llvm-svn: 355777

show more ...