[clang-reorder-fields] Reorder leading comments (#123740)

Similarly to https://github.com/llvm/llvm-project/pull/122918, leading
comments are currently not being moved.

```
struct Foo {
// T

[clang-reorder-fields] Reorder leading comments (#123740)

Similarly to https://github.com/llvm/llvm-project/pull/122918, leading
comments are currently not being moved.

```
struct Foo {
// This one is the cool field.
int a;
int b;
};
```

becomes:

```
struct Foo {
// This one is the cool field.
int b;
int a;
};
```

but should be:

```
struct Foo {
int b;
// This one is the cool field.
int a;
};
```

show more ...

[clang][refactor] Refactor `findNextTokenIncludingComments` (#123060)

We have two copies of the same code in clang-tidy and
clang-reorder-fields, and those are extremenly similar to
`Lexer::findNe

[clang][refactor] Refactor `findNextTokenIncludingComments` (#123060)

We have two copies of the same code in clang-tidy and
clang-reorder-fields, and those are extremenly similar to
`Lexer::findNextToken`, so just add an extra agument to the latter.

---------

Co-authored-by: cor3ntin <corentinjabot@gmail.com>

show more ...

Skip escaped newlines before checking for whitespace in Lexer::getRawToken. (#117548)

The Lexer used in getRawToken is not told to keep whitespace, so when it
skips over escaped newlines, it also i

Skip escaped newlines before checking for whitespace in Lexer::getRawToken. (#117548)

The Lexer used in getRawToken is not told to keep whitespace, so when it
skips over escaped newlines, it also ignores whitespace, regardless of
getRawToken's IgnoreWhiteSpace parameter.

Instead of letting this case fall through to lexing, check
for whitespace after skipping over any escaped newlines.

show more ...

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to av

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to avoid quite some code duplication.

Differential Revision: https://reviews.llvm.org/D158413

show more ...

[Lex] Preambles should contain the global module fragment.

For applications like clangd, the preamble remains an important optimization
when editing a module definition. The global module fragment i

[Lex] Preambles should contain the global module fragment.

For applications like clangd, the preamble remains an important optimization
when editing a module definition. The global module fragment is a good fit for
it as it by definition contains only preprocessor directives.
Before this patch, we would terminate the preamble immediately at the "module"
keyword.

Differential Revision: https://reviews.llvm.org/D158439

show more ...

Move from llvm::makeArrayRef to ArrayRef deduction guides - clang/ part

This is a follow-up to https://reviews.llvm.org/D140896, split into
several parts as it touches a lot of files.

Differential

Move from llvm::makeArrayRef to ArrayRef deduction guides - clang/ part

This is a follow-up to https://reviews.llvm.org/D140896, split into
several parts as it touches a lot of files.

Differential Revision: https://reviews.llvm.org/D141139

show more ...

Revert "Revert "[clang][Lex] Fix a crash on malformed string literals""

This reverts commit feea7ef23cb1bef92d363cc613052f8f3a878fc2.
Drops the test case, see https://reviews.llvm.org/D135161#3839510

Revert "[clang][Lex] Fix a crash on malformed string literals"

This reverts commit 36a200208facf58d454c9b7253c956c2f2a8b946.

[clang][Lex] Fix a crash on malformed string literals

Differential Revision: https://reviews.llvm.org/D135161

[clang, clang-tools-extra] Don't use Optional::{hasValue,getValue} (NFC)

[clang][Lexer] Fix tests after ff77071a4d67

[clang][Lexer] Make raw and normal lexer behave the same for line comments

Normally there are heruistics in lexer to treat `//*` specially in
language modes that don't have line comments (to emit `/

[clang][Lexer] Make raw and normal lexer behave the same for line comments

Normally there are heruistics in lexer to treat `//*` specially in
language modes that don't have line comments (to emit `/`). Unfortunately this
only applied to the first occurence of a line comment inside the file, as the
subsequent line comments were treated as if language had support for them.

This unfortunately only holds in normal lexing mode, as in raw mode all
occurences of line comments received this treatment, which created discrepancies
when comparing expanded and spelled tokens.

The proper fix would be to just make sure we treat all the line comments with a
subsequent `*` the same way, but it would imply breaking some code that's
accepted by clang today. So instead we introduce the same bug into raw lexing
mode.

Fixes https://github.com/clangd/clangd/issues/1003.

Differential Revision: https://reviews.llvm.org/D118471

show more ...

[Lexer] Fix bug in `makeFileCharRange` called on split tokens.

When the end loc of the specified range is a split token, `makeFileCharRange`
does not process it correctly. This patch adds proper su

[Lexer] Fix bug in `makeFileCharRange` called on split tokens.

When the end loc of the specified range is a split token, `makeFileCharRange`
does not process it correctly. This patch adds proper support for split tokens.

Differential Revision: https://reviews.llvm.org/D105365

show more ...

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include. Unfortunately it fails to do that for
predefined buffer since it doesn't have a valid insertion location.

As a result of that any file ID created for an include inside the
predefined buffers breaks the traversal logic in
SourceManager::computeMacroArgsCache.

To fix this issue we first record number of created FIDs for predefined
buffer, and then skip them explicitly in source manager.

Another solution would be to just give predefined buffers a valid source
location, but it is unclear where that should be..

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D78649

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[clang] Add test for FindNextToken in Lexer.

Reviewers: ilya-biryukov

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68565

llvm-svn: 373910

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
o

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942

show more ...

Remove cache for macro arg stringization

Summary:
The cache recorded the wrong expansion location for all but the first
stringization. It seems uncommon to stringize the same macro argument
multiple

Remove cache for macro arg stringization

Summary:
The cache recorded the wrong expansion location for all but the first
stringization. It seems uncommon to stringize the same macro argument
multiple times, so this cache doesn't seem that important.

Fixes PR39942

Reviewers: vsk, rsmith

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D65428

llvm-svn: 367337

show more ...

Re-apply r357823 "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."

It now comes with a follow-up fix for the clients of this API
in clangd and clang-tidy.

Differential Revision: https://rev

Re-apply r357823 "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."

It now comes with a follow-up fix for the clients of this API
in clangd and clang-tidy.

Differential Revision: https://reviews.llvm.org/D59977

llvm-svn: 359035

show more ...

Revert "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."

This reverts commit r357823.

Was breaking clang-tidy!

Differential Revision: https://reviews.llvm.org/D59977

llvm-svn: 357827

[Lexer] NFC: Fix an off-by-one bug in getAsCharRange().

As the unit test demonstrates, subtracting 1 from the offset was unnecessary.

The only user of this function was the plist file emitter (in S

[Lexer] NFC: Fix an off-by-one bug in getAsCharRange().

As the unit test demonstrates, subtracting 1 from the offset was unnecessary.

The only user of this function was the plist file emitter (in Static Analyzer
and ARCMigrator). It means that a lot of Static Analyzer's plist arrows
are in fact off by one character. The patch carefully preserves this
completely incorrect behavior and causes no functional change,
i.e. no plist format breakage.

Differential Revision: https://reviews.llvm.org/D59977

llvm-svn: 357823

show more ...

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module b

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.

Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.

llvm-svn: 355777

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

PR37189 Fix incorrect end source location and spelling for a split '>>' token.

When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an
extension) when we do the same for other t

PR37189 Fix incorrect end source location and spelling for a split '>>' token.

When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an
extension) when we do the same for other tokens starting with a '>', we can't
just use a location pointing to the first '>' as the location of the split
token, because that would result in our miscomputing the length and spelling
for the token. As a consequence, for example, a refactoring replacing 'A<X>'
with something else would sometimes replace one character too many, and
similarly diagnostics highlighting a template-id source range would highlight
one character too many.

Fix this by creating an expansion range covering the first character of the
'>>' token, whose spelling is '>'. For this to work, we generalize the
expansion range of a macro FileID to be either a token range (the common case)
or a character range (used in this new case).

llvm-svn: 331155

show more ...

[Lex] Avoid out-of-bounds dereference in LexAngledStringLiteral.

Fix makes the loop in LexAngledStringLiteral more like the loops in
LexStringLiteral, LexCharConstant. When we skip a character after

[Lex] Avoid out-of-bounds dereference in LexAngledStringLiteral.

Fix makes the loop in LexAngledStringLiteral more like the loops in
LexStringLiteral, LexCharConstant. When we skip a character after
backslash, we need to check if we reached the end of the file instead of
reading the next character unconditionally.

Discovered by OSS-Fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3832

rdar://problem/35572754

Reviewers: arphaman, kcc, rsmith, dexonsmith

Reviewed By: rsmith, dexonsmith

Subscribers: cfe-commits, rsmith, dexonsmith

Differential Revision: https://reviews.llvm.org/D41423

llvm-svn: 322390

show more ...