[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to av

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to avoid quite some code duplication.

Differential Revision: https://reviews.llvm.org/D158413

show more ...

[clang] llvm::Optional::value() && => operator*/operator->

std::optional::value() has undesired exception checking semantics and is
unavailable in older Xcode (see _LIBCPP_AVAILABILITY_BAD_OPTIONAL_

[clang] llvm::Optional::value() && => operator*/operator->

std::optional::value() has undesired exception checking semantics and is
unavailable in older Xcode (see _LIBCPP_AVAILABILITY_BAD_OPTIONAL_ACCESS). The
call sites block std::optional migration.

show more ...

[clang] Use value instead of getValue (NFC)

[clang, clang-tools-extra] Use has_value instead of hasValue (NFC)

Revert "Don't use Optional::hasValue (NFC)"

This reverts commit aa8feeefd3ac6c78ee8f67bf033976fc7d68bc6d.

Don't use Optional::hasValue (NFC)

[NFC][testing] Return underlying strings directly instead of OS.str()

This avoids an unnecessary copy required by 'return OS.str()', allowing
instead for NRVO or implicit move. The .str() call (whic

[NFC][testing] Return underlying strings directly instead of OS.str()

This avoids an unnecessary copy required by 'return OS.str()', allowing
instead for NRVO or implicit move. The .str() call (which flushes the
stream) is no longer required since 65b13610a5226b84889b923bae884ba395ad084d,
which made raw_string_ostream unbuffered by default.

Differential Revision: https://reviews.llvm.org/D115374

show more ...

[analyzer] Introduce MacroExpansionContext to libAnalysis

Introduce `MacroExpansionContext` to track what and how macros in a translation
unit expand. This is the first element of the patch-stack in

[analyzer] Introduce MacroExpansionContext to libAnalysis

Introduce `MacroExpansionContext` to track what and how macros in a translation
unit expand. This is the first element of the patch-stack in this direction.

The main goal is to substitute the current macro expansion generator in the
`PlistsDiagnostics`, but all the other `DiagnosticsConsumer` could benefit from
this.

`getExpandedText` and `getOriginalText` are the primary functions of this class.
The former can provide you the text that was the result of the macro expansion
chain starting from a `SourceLocation`.
While the latter will tell you **what text** was in the original source code
replaced by the macro expansion chain from that location.

Here is an example:

void bar();
#define retArg(x) x
#define retArgUnclosed retArg(bar()
#define BB CC
#define applyInt BB(int)
#define CC(x) retArgUnclosed

void unbalancedMacros() {
applyInt );
//^~~~~~~~~~^ is the substituted range
// Original text is "applyInt )"
// Expanded text is "bar()"
}

#define expandArgUnclosedCommaExpr(x) (x, bar(), 1
#define f expandArgUnclosedCommaExpr

void unbalancedMacros2() {
int x = f(f(1)) )); // Look at the parenthesis!
// ^~~~~~^ is the substituted range
// Original text is "f(f(1))"
// Expanded text is "((1,bar(),1,bar(),1"
}

Might worth investigating how to provide a reusable component, which could be
used for example by a standalone tool eg. expanding all macros to their
definitions.

I borrowed the main idea from the `PrintPreprocessedOutput.cpp` Frontend
component, providing a `PPCallbacks` instance hooking the preprocessor events.
I'm using that for calculating the source range where tokens will be expanded
to. I'm also using the `Preprocessor`'s `OnToken` callback, via the
`Preprocessor::setTokenWatcher` to reconstruct the expanded text.

Unfortunately, I concatenate the token's string representation without any
whitespaces except if the token is an identifier when I emit an extra space
to produce valid code for `int var` token sequences.
This could be improved later if needed.

Patch-stack:
1) D93222 (this one) Introduces the MacroExpansionContext class and unittests

2) D93223 Create MacroExpansionContext member in AnalysisConsumer and pass
down to the diagnostics consumers

3) D93224 Use the MacroExpansionContext for macro expansions in plists
It replaces the 'old' macro expansion mechanism.

4) D94673 API for CTU macro expansions
You should be able to get a `MacroExpansionContext` for each imported TU.
Right now it will just return `llvm::None` as this is not implemented yet.

5) FIXME: Implement macro expansion tracking for imported TUs as well.

It would also relieve us from bugs like:
- [fixed] D86135
- [confirmed] The `__VA_ARGS__` and other macro nitty-gritty, such as how to
stringify macro parameters, where to put or swallow commas, etc. are not
handled correctly.
- [confirmed] Unbalanced parenthesis are not well handled - resulting in
incorrect expansions or even crashes.
- [confirmed][crashing] https://bugs.llvm.org/show_bug.cgi?id=48358

Reviewed By: martong, Szelethus

Differential Revision: https://reviews.llvm.org/D93222

show more ...