[C++20][Modules] Quote header unit name in preprocessor output (-E) (#112883)

Summary:
Before this change clang produced output with header unit names that may
conaint path separators, dots and ot

[C++20][Modules] Quote header unit name in preprocessor output (-E) (#112883)

Summary:
Before this change clang produced output with header unit names that may
conaint path separators, dots and other non-identifier characters. This
diff prints header unit name in quotes and -E output can be compiled
again. Also remove unnecessary space between header unit name and semi.

Test Plan: check-clang

show more ...

[clang-repl] Fix printing preprocessed tokens and macros (#104964)

[NFC][clang] Avoid unnecessary assignment (#100728)

`IsStartOfLine` is set to false at the end of the loop.

Revert "[Clang] Implement P3034R1 Module Declarations Shouldn’t be Macros" (#99838)

Reverts llvm/llvm-project#90574

[Clang] Implement P3034R1 Module Declarations Shouldn’t be Macros (#90574)

This PR implement [P3034R1 Module Declarations Shouldn’t be
Macros](https://wg21.link/P3034R1), and refactor the convolute

[Clang] Implement P3034R1 Module Declarations Shouldn’t be Macros (#90574)

This PR implement [P3034R1 Module Declarations Shouldn’t be
Macros](https://wg21.link/P3034R1), and refactor the convoluted state
machines in module name lexical analysis.

---------

Signed-off-by: yronglin <yronglin777@gmail.com>
Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>

show more ...

Reland [clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (#95802)

This commit implements the entirety of the now-accepted [N3017
-Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg1

Reland [clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (#95802)

This commit implements the entirety of the now-accepted [N3017
-Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm) and
its sister C++ paper [p1967](https://wg21.link/p1967). It implements
everything in the specification, and includes an implementation that
drastically improves the time it takes to embed data in specific
scenarios (the initialization of character type arrays). The mechanisms
used to do this are used under the "as-if" rule, and in general when the
system cannot detect it is initializing an array object in a variable
declaration, will generate EmbedExpr AST node which will be expanded by
AST consumers (CodeGen or constant expression evaluators) or expand
embed directive as a comma expression.

This reverts commit
https://github.com/llvm/llvm-project/commit/682d461d5a231cee54d65910e6341769419a67d7.

---------

Co-authored-by: The Phantom Derpstorm <phdofthehouse@gmail.com>
Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
Co-authored-by: H. Vetinari <h.vetinari@gmx.com>

show more ...

Revert "✨ [Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy)" (#95299)

Reverts llvm/llvm-project#68620

Introduce or expose a memory leak and UB, see llvm/llvm-pro

Revert "✨ [Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy)" (#95299)

Reverts llvm/llvm-project#68620

Introduce or expose a memory leak and UB, see llvm/llvm-project#68620

show more ...

[clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy) (#68620)

This commit implements the entirety of the now-accepted [N3017 -
Preprocessor
Embed](https://www

[clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy) (#68620)

This commit implements the entirety of the now-accepted [N3017 -
Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm) and
its sister C++ paper [p1967](https://wg21.link/p1967). It implements
everything in the specification, and includes an implementation that
drastically improves the time it takes to embed data in specific
scenarios (the initialization of character type arrays). The mechanisms
used to do this are used under the "as-if" rule, and in general when the
system cannot detect it is initializing an array object in a variable
declaration, will generate EmbedExpr AST node which will be expanded
by AST consumers (CodeGen or constant expression evaluators) or
expand embed directive as a comma expression.

---------

Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
Co-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: Podchishchaeva, Mariya <mariya.podchishchaeva@intel.com>

show more ...

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
s

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
suggested module, regardless of whether it was actually imported or not.
The extra `bool ModuleImported` parameter then denotes whether the
header `#include` will be automatically translated into import the the
module.

The main change is in `clang/lib/Lex/PPDirectives.cpp`, where we take
care to not modify `SuggestedModule` after it's been populated by
`LookupHeaderIncludeOrImport()`. We now exclusively use the `SM`
(`ModuleToImport`) variable instead, which has been equivalent to
`SuggestedModule` until now. This allows us to use the original
non-modified `SuggestedModule` for the callback itself.

(This patch turns out to be necessary for
https://github.com/apple/llvm-project/pull/8011).

show more ...

Add -fkeep-system-includes modifier for -E

This option will cause -E to preserve the #include directives
for system headers, rather than expanding them into the output.
This can greatly reduce the v

Add -fkeep-system-includes modifier for -E

This option will cause -E to preserve the #include directives
for system headers, rather than expanding them into the output.
This can greatly reduce the volume of preprocessed source text
in a test case, making test case reduction simpler.

Note that -fkeep-system-includes is not always appropriate. For
example, if the problem you want to reproduce is induced by a
system header file, it's better to expand those headers fully.
If your source defines symbols that influence the content of a
system header (e.g., _POSIX_SOURCE) then -E will eliminate the
definition, potentially changing the meaning of the preprocessed
source. If you use -isystem to point to non-system headers, for
example to suppress warnings in third-party software, those will
not be expanded and might make the preprocessed source less useful
as a test case.

show more ...

[NFC] Change a reference member to pointer

This will allow the raw_ostream to be redirected in a subsequent commit.

[clang] Add a new annotation token: annot_repl_input_end

This patch is the first part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

It adds an anno

[clang] Add a new annotation token: annot_repl_input_end

This patch is the first part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

It adds an annotation token which will replace the original EOF token
when we are in the incremental C++ mode. In addition, when we're
parsing an ExprStmt and there's a missing semicolon after the
expression, we set a marker in the annotation token and continue
parsing.

Eventually, we propogate this info in ParseTopLevelStmtDecl and are able
to mark this Decl as something we want to do value printing. Below is a
example:

clang-repl> int x = 42;
clang-repl> x
// `x` is a TopLevelStmtDecl and without a semicolon, we should set
// it's IsSemiMissing bit so we can do something interesting in
// ASTConsumer::HandleTopLevelDecl.

The idea about annotation toke is proposed by Richard Smith, thanks!

Signed-off-by: Jun Zhang <jun@junz.org>

Differential Revision: https://reviews.llvm.org/D148997

show more ...

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.

I tested this with GCC 7.5, the oldest supported GCC I had around.

Differential Revision: https://reviews.llvm.org/D140332

show more ...

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.

On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).

show more ...

[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional

[clang] Use std::size instead of llvm::array_lengthof

LLVM contains a helpful function for getting the size of a C-style
array: `llvm::array_lengthof`. This is useful prior to C++17, but not as
help

[clang] Use std::size instead of llvm::array_lengthof

LLVM contains a helpful function for getting the size of a C-style
array: `llvm::array_lengthof`. This is useful prior to C++17, but not as
helpful for C++17 or later: `std::size` already has support for C-style
arrays.

Change call sites to use `std::size` instead. Leave the few call sites that
use a locally defined `array_lengthof` that are meant to test previous bugs
with NTTPs in clang analyzer and SemaTemplate.

Differential Revision: https://reviews.llvm.org/D133520

show more ...

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<FileEntryRef>`.

With the API change in place, this patch then removes some uses of the deprecated `FileEntry::getName()` (e.g. in `DependencyGraph.cpp` and `ModuleDependencyCollector.cpp`).

Reviewed By: dexonsmith, bnbarham

Differential Revision: https://reviews.llvm.org/D123574

show more ...

[C++20][Modules][HU 5/5] Add fdirectives-only mode for preprocessing output.

When the -fdirectives-only option is used together with -E, the preprocessor
output reflects evaluation of if/then/else d

[C++20][Modules][HU 5/5] Add fdirectives-only mode for preprocessing output.

When the -fdirectives-only option is used together with -E, the preprocessor
output reflects evaluation of if/then/else directives.

As such, it preserves defines and undefs of macros that are still live after
such processing. The intent is that this output could be consumed as input
to generate considered a C++20 header unit.

We strip out any (unused) defines that come from built-in, built-in-file or
command line; these are re-added when the preprocessed source is consumed.

Differential Revision: https://reviews.llvm.org/D121099

show more ...

[clang] fix out of bounds access in an empty string when lexing a _Pragma with missing string token

The lexer can attempt to lex a _Pragma and crash with an out of bounds string access when it's
lex

[clang] fix out of bounds access in an empty string when lexing a _Pragma with missing string token

The lexer can attempt to lex a _Pragma and crash with an out of bounds string access when it's
lexing a _Pragma whose string token is an invalid buffer, e.g. when a module header file from which the macro
expansion for that token was deleted from the file system.

Differential Revision: https://reviews.llvm.org/D116052

show more ...

Revert "Rename llvm::array_lengthof into llvm::size to match std::size from C++17"

This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2.

- It conflicts with the existing llvm::size in STLEx

Revert "Rename llvm::array_lengthof into llvm::size to match std::size from C++17"

This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2.

- It conflicts with the existing llvm::size in STLExtras, which will now
never be called.
- Calling it without llvm:: breaks C++17 compat

show more ...

Rename llvm::array_lengthof into llvm::size to match std::size from C++17

As a conquence move llvm::array_lengthof from STLExtras.h to
STLForwardCompat.h (which is included by STLExtras.h so no buil

Rename llvm::array_lengthof into llvm::size to match std::size from C++17

As a conquence move llvm::array_lengthof from STLExtras.h to
STLForwardCompat.h (which is included by STLExtras.h so no build
breakage expected).

show more ...

[clang] Use true/false instead of 1/0 (NFC)

Identified with modernize-use-bool-literals.

[Preprocessor] Fix newline before/after _Pragma.

The PragmaAssumeNonNullHandler (and maybe others) passes an invalid
SourceLocation to its callback, hence PrintPreprocessedOutput does not
know how m

[Preprocessor] Fix newline before/after _Pragma.

The PragmaAssumeNonNullHandler (and maybe others) passes an invalid
SourceLocation to its callback, hence PrintPreprocessedOutput does not
know how many lines to insert between the previous token and the
pragma and does nothing.

With this patch we instead assume that the unknown token is on the same
line as the previous such that we can call the procedure that also emits
semantically significant whitespace.

Fixes bug reported here: https://reviews.llvm.org/D104601#3105044

show more ...

[Preprocessor] Fix warning: left and right subexpressions are identical. NFCI.

This is reported by msvc as
warning C6287: redundant code: the left and right subexpressions are identical

EmittedDire

[Preprocessor] Fix warning: left and right subexpressions are identical. NFCI.

This is reported by msvc as
warning C6287: redundant code: the left and right subexpressions are identical

EmittedDirectiveOnThisLine implies EmittedTokensOnThisLine
making this an NFC change. To be on the safe side and because both of
them are checked at other places as well, we continue to check both.

Compiler warning reported here:
https://reviews.llvm.org/D104601#2957333

show more ...

[clang] Let PPCallbacks::PragmaWarning() pass specifier as enum instead of string

Differential Revision: https://reviews.llvm.org/D110635