[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
s

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
suggested module, regardless of whether it was actually imported or not.
The extra `bool ModuleImported` parameter then denotes whether the
header `#include` will be automatically translated into import the the
module.

The main change is in `clang/lib/Lex/PPDirectives.cpp`, where we take
care to not modify `SuggestedModule` after it's been populated by
`LookupHeaderIncludeOrImport()`. We now exclusively use the `SM`
(`ModuleToImport`) variable instead, which has been equivalent to
`SuggestedModule` until now. This allows us to use the original
non-modified `SuggestedModule` for the callback itself.

(This patch turns out to be necessary for
https://github.com/apple/llvm-project/pull/8011).

show more ...

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to av

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to avoid quite some code duplication.

Differential Revision: https://reviews.llvm.org/D158413

show more ...

[clang][Lex] Add back PPCallbacks::FileNotFound

This callback was removed with commit 7a124f4859, but we use it
downstream in ROOT/Cling to implement handling of a special include
syntax. Add back a

[clang][Lex] Add back PPCallbacks::FileNotFound

This callback was removed with commit 7a124f4859, but we use it
downstream in ROOT/Cling to implement handling of a special include
syntax. Add back a "safe" version of the callback that only takes
the file name and return a bool to silently skip the file.

Differential Revision: https://reviews.llvm.org/D142196

show more ...

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.

I tested this with GCC 7.5, the oldest supported GCC I had around.

Differential Revision: https://reviews.llvm.org/D140332

show more ...

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.

On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).

show more ...

[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<FileEntryRef>`.

With the API change in place, this patch then removes some uses of the deprecated `FileEntry::getName()` (e.g. in `DependencyGraph.cpp` and `ModuleDependencyCollector.cpp`).

Reviewed By: dexonsmith, bnbarham

Differential Revision: https://reviews.llvm.org/D123574

show more ...

Reland "[clang-repl] Implement partial translation units and error recovery."

Original commit message:

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org

Reland "[clang-repl] Implement partial translation units and error recovery."

Original commit message:

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:

Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.

The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.

The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.

Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.

It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.

We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.

We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.

This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:

```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```

Differential revision: https://reviews.llvm.org/D104918

show more ...

Revert "[clang-repl] Implement partial translation units and error recovery."

This reverts commit 6775fc6ffa3ca1c36b20c25fa4e7f48f81213cf2.

It also reverts "[lldb] Fix compilation by adjusting to t

Revert "[clang-repl] Implement partial translation units and error recovery."

This reverts commit 6775fc6ffa3ca1c36b20c25fa4e7f48f81213cf2.

It also reverts "[lldb] Fix compilation by adjusting to the new ASTContext signature."

This reverts commit 03a3f86071c10a1f6cbbf7375aa6fe9d94168972.

We see some failures on the lldb infrastructure, these changes might play a role
in it. Let's revert it now and see if the bots will become green.

Ref: https://reviews.llvm.org/D104918

show more ...

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:

Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.

The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.

The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.

Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.

It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.

We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.

We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.

This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:

```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```

Differential revision: https://reviews.llvm.org/D104918

show more ...

Lex: add a callback for `#pragma mark`

Allow a preprocessor observer to be notified of mark pragmas. Although
this does not impact code generation in any way, it is useful for other
clients, such a

Lex: add a callback for `#pragma mark`

Allow a preprocessor observer to be notified of mark pragmas. Although
this does not impact code generation in any way, it is useful for other
clients, such as clangd, to be able to identify any marked regions.

Reviewed By: dgoldman

Differential Revision: https://reviews.llvm.org/D105368

show more ...

Introduce a DirectoryEntryRef that stores both a reference and an
accessed name to the directory entry

This commit introduces a parallel API that returns a DirectoryEntryRef
to the FileManager, simi

Introduce a DirectoryEntryRef that stores both a reference and an
accessed name to the directory entry

This commit introduces a parallel API that returns a DirectoryEntryRef
to the FileManager, similar to the parallel FileEntryRef API. All
uses will have to be update in follow-up patches. The immediate use of the new API in this
patch fixes the issue where a file manager was reused in clang-scan-deps,
but reported an different file path whenever a framework lookup was done through a symlink.

Differential Revision: https://reviews.llvm.org/D67026

llvm-svn: 370562

show more ...

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616

show more ...

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module b

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.

Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.

llvm-svn: 355777

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Fix -Wsign-compare in new tests

llvm-svn: 351478

Revert r351209 (which was a revert of r350891) with a fix.

The test case had a parse error that was causing the condition string to be misreported. We now have better fallback code for error cases.

Revert r351209 (which was a revert of r350891) with a fix.

The test case had a parse error that was causing the condition string to be misreported. We now have better fallback code for error cases.

llvm-svn: 351470

show more ...

Revert "Correct the source range returned from preprocessor callbacks."

This reverts commit r350891. Also add a test case that would return an
empty string with r350891.

llvm-svn: 351209

Silence -Wsign-compare in unittests

llvm-svn: 350933

Correct the source range returned from preprocessor callbacks.

This adjusts the source range passed in to the preprocessor callbacks to only include the condition range itself, rather than all of th

Correct the source range returned from preprocessor callbacks.

This adjusts the source range passed in to the preprocessor callbacks to only include the condition range itself, rather than all of the conditionally skipped tokens.

llvm-svn: 350891

show more ...

Lift VFS from clang to llvm (NFC)

This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang

Lift VFS from clang to llvm (NFC)

This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.

RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html

Differential revision: https://reviews.llvm.org/D52783

llvm-svn: 344140

show more ...

Reland '[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective'

This commit relands r331904.

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, an

Reland '[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective'

This commit relands r331904.

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This will be useful
in https://reviews.llvm.org/D43778 to determine which includes are
system
headers.

Differential Revision: https://reviews.llvm.org/D46614

llvm-svn: 332021

show more ...

Revert "[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective"

This reverts commit r331904 because of a memory leak.

llvm-svn: 331932

[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This

[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This will be useful
in https://reviews.llvm.org/D43778 to determine which includes are system
headers.

Differential Revision: https://reviews.llvm.org/D46614

llvm-svn: 331904

show more ...

Add #pragma clang module build/endbuild pragmas for performing a module build
as part of a compilation.

This is intended for two purposes:

1) Writing self-contained test cases for modules: we can n

Add #pragma clang module build/endbuild pragmas for performing a module build
as part of a compilation.

This is intended for two purposes:

1) Writing self-contained test cases for modules: we can now write a single
source file test that builds some number of module files on the side and
imports them.

2) Debugging / test case reduction. A single-source testcase is much more
amenable to reduction, compared to a VFS tarball or .pcm files.

llvm-svn: 305101

show more ...