[clangd] Fix crash with null check for Token at Loc (#94528)

Fixes https://github.com/llvm/llvm-project/issues/94599

[clangd] Handle an expanded token range that ends in the `eof` token in TokenBuffer::spelledForExpanded() (#78092)

Such ranges can legitimately arise in the case of invalid code, such as
a declarat

[clangd] Handle an expanded token range that ends in the `eof` token in TokenBuffer::spelledForExpanded() (#78092)

Such ranges can legitimately arise in the case of invalid code, such as
a declaration missing an ending brace.

Fixes https://github.com/clangd/clangd/issues/1559

show more ...

[clang][tooling] Fix early termination when there are nested expansions

This also does some cleanups, I am happy to undo them (or send as
separate patches):
- Change the early exit to stop only once

[clang][tooling] Fix early termination when there are nested expansions

This also does some cleanups, I am happy to undo them (or send as
separate patches):
- Change the early exit to stop only once we hit an expansion inside the
main file, to make sure we keep following the nested expansions.
- Add more tests to cover all the cases mentioned in the implementation
- Drop the adjustments for prev/next tokens. We do the final checks
based on the expansion locations anyway, so any intermediate mapping
was a no-op.

Differential Revision: https://reviews.llvm.org/D154335

show more ...

[clang] Remove remaining uses of llvm::Optional (NFC)

This patch removes several "using" declarations and #include
"llvm/ADT/Optional.h".

This is part of an effort to migrate from llvm::Optional to

[clang] Remove remaining uses of llvm::Optional (NFC)

This patch removes several "using" declarations and #include
"llvm/ADT/Optional.h".

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[clang] Use std::optional instead of llvm::Optional (NFC)

This patch replaces (llvm::|)Optional< with std::optional<. I'll post
a separate patch to remove #include "llvm/ADT/Optional.h".

This is p

[clang] Use std::optional instead of llvm::Optional (NFC)

This patch replaces (llvm::|)Optional< with std::optional<. I'll post
a separate patch to remove #include "llvm/ADT/Optional.h".

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[clang] Add #include <optional> (NFC)

This patch adds #include <optional> to those files containing
llvm::Optional<...> or Optional<...>.

I'll post a separate patch to actually replace llvm::Option

[clang] Add #include <optional> (NFC)

This patch adds #include <optional> to those files containing
llvm::Optional<...> or Optional<...>.

I'll post a separate patch to actually replace llvm::Optional with
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[test] Split out Annotations from `TestingSupport`

The Annotations helper class does not have a gtest or gmock dependency, but because it's bundled with the rest of TestingSupport, it gets one. By s

[test] Split out Annotations from `TestingSupport`

The Annotations helper class does not have a gtest or gmock dependency, but because it's bundled with the rest of TestingSupport, it gets one. By splitting it out, a target can use it without being forced to use LLVM's copy of gtest.

Reviewed By: GMNGeoffrey, sammccall, gribozavr2

Differential Revision: https://reviews.llvm.org/D141175

show more ...

Move from llvm::makeArrayRef to ArrayRef deduction guides - clang/ part

This is a follow-up to https://reviews.llvm.org/D140896, split into
several parts as it touches a lot of files.

Differential

Move from llvm::makeArrayRef to ArrayRef deduction guides - clang/ part

This is a follow-up to https://reviews.llvm.org/D140896, split into
several parts as it touches a lot of files.

Differential Revision: https://reviews.llvm.org/D141139

show more ...

[clang] Don't including None.h (NFC)

These source files no longer use None, so they do not need to include
None.h.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https:

[clang] Don't including None.h (NFC)

These source files no longer use None, so they do not need to include
None.h.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[clang/unittests] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the a

[clang/unittests] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[Syntax] Fix macro-arg handling in TokenBuffer::spelledForExpanded

A few cases were not handled correctly. Notably:
#define ID(X) X
#define HIDE a ID(b)
HIDE
spelledForExpanded() would claim H

[Syntax] Fix macro-arg handling in TokenBuffer::spelledForExpanded

A few cases were not handled correctly. Notably:
#define ID(X) X
#define HIDE a ID(b)
HIDE
spelledForExpanded() would claim HIDE is an equivalent range of the 'b' it
contains, despite the fact that HIDE also covers 'a'.

While trying to fix this bug, I found findCommonRangeForMacroArgs hard
to understand (both the implementation and how it's used in spelledForExpanded).
It relies on details of the SourceLocation graph that are IMO fairly obscure.
So I've added/revised quite a lot of comments and made some naming tweaks.

Fixes https://github.com/clangd/clangd/issues/1289

Differential Revision: https://reviews.llvm.org/D134618

show more ...

[clang] createInvocationFromCommandLine -> createInvocation, delete former. NFC

(Followup from 40c13720a4b977d4347bbde53c52a4d0703823c2)

Differential Revision: https://reviews.llvm.org/D125012

[NFC] Remove duplicate include

[NFC][Clang][test] Inclusive language: Remove and rephrase uses of sanity test/check in clang/test

Part of work to use more inclusive terms in clang/llvm.

[clang][Syntax] Optimize expandedTokens for token ranges.

`expandedTokens(SourceRange)` used to do a binary search to get the
expanded tokens belonging to a source range. Each binary search uses
`is

[clang][Syntax] Optimize expandedTokens for token ranges.

`expandedTokens(SourceRange)` used to do a binary search to get the
expanded tokens belonging to a source range. Each binary search uses
`isBeforeInTranslationUnit` to order two source locations. This is
inherently very slow.
By profiling clangd we found out that users like clangd::SelectionTree
spend 95% of time in `isBeforeInTranslationUnit`. Also it is worth
noting that users of `expandedTokens(SourceRange)` majorly use ranges
provided by AST to query this funciton. The ranges provided by AST are
token ranges (starting at the beginning of a token and ending at the
beginning of another token).

Therefore we can avoid the binary search in majority of the cases by
maintaining an index of ExpandedToken by their SourceLocations. We still
do binary search for ranges which are not token ranges but such
instances are quite low.

Performance:
`~/build/bin/clangd --check=clang/lib/Serialization/ASTReader.cpp`
Before: Took 2:10s to complete.
Now: Took 1:13s to complete.

Differential Revision: https://reviews.llvm.org/D99086

show more ...

[clang] Mark re-injected tokens appropriately during pragma handling

This hides such tokens from TokenWatcher, preventing crashes in clients
trying to match spelled and expanded tokens.

Fixes https

[clang] Mark re-injected tokens appropriately during pragma handling

This hides such tokens from TokenWatcher, preventing crashes in clients
trying to match spelled and expanded tokens.

Fixes https://github.com/clangd/clangd/issues/712

Differential Revision: https://reviews.llvm.org/D98483

show more ...

[Syntax] expose API for expansions overlapping a spelled token range.

Summary:
This allows efficiently accessing all expansions (without iterating over each
token and searching), and also identifyin

[Syntax] expose API for expansions overlapping a spelled token range.

Summary:
This allows efficiently accessing all expansions (without iterating over each
token and searching), and also identifying tokens within a range that are
affected by the preprocessor (which is how clangd will use it).

Subscribers: ilya-biryukov, kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D84009

show more ...

[Syntax] Add mapping from spelled to expanded tokens for TokenBuffer

Summary:
Same restrictions apply as in the other direction: macro arguments are
not supported yet, only full macro expansions can

[Syntax] Add mapping from spelled to expanded tokens for TokenBuffer

Summary:
Same restrictions apply as in the other direction: macro arguments are
not supported yet, only full macro expansions can be mapped.

Taking over from https://reviews.llvm.org/D72581.

Reviewers: gribozavr2, sammccall

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77209

show more ...

[Syntax] Merge overlapping top-level macros in TokenBuffer

Summary:
Our previous definition of "top-level" was too informal, and didn't
allow for overlapping macros that each directly produce expand

[Syntax] Merge overlapping top-level macros in TokenBuffer

Summary:
Our previous definition of "top-level" was too informal, and didn't
allow for overlapping macros that each directly produce expanded tokens.
See D77507 for previous discussion.

Fixes http://bugs.llvm.org/show_bug.cgi?id=45428

Reviewers: kadircet, vabridgers

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77615

show more ...

[clang][Syntax] Handle macro arguments in spelledForExpanded

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D75446

[clang][Syntax] Add spelledTokenAt helper to TokenBuffer

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D75503

[clang][Tooling] Add a way to tokenize a FileRange

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D74962

[Syntax] Uppercase the first letter of the test name. NFC

To match naming style of other tests.

[Tooling/Syntax] Helpers to find spelled tokens touching a location.

Summary: Useful when positions are used to target nodes, with before/after ambiguity.

Reviewers: ilya-biryukov, kbobyrev

Subscr

[Tooling/Syntax] Helpers to find spelled tokens touching a location.

Summary: Useful when positions are used to target nodes, with before/after ambiguity.

Reviewers: ilya-biryukov, kbobyrev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71356

show more ...

Reland [clangd] Rethink how SelectionTree deals with macros and #includes.

This reverts commit 905b002c139f039a32ab9bf1fad63d745d12423f.

Avoid tricky (and invalid) comparator for std::set.