[clangd] Support .clangd command line modifications for C++ modules (#122606)

Tunnels `Manger` object into the `ScanningAllProjectModules` so it can
be used to perform necessary command-line modifi

[clangd] Support .clangd command line modifications for C++ modules (#122606)

Tunnels `Manger` object into the `ScanningAllProjectModules` so it can
be used to perform necessary command-line modifications (which also adds
`--resources` path previously added there explicitly). This allows using
the experimental C++ modules support with gcc.

This was discussed in the issue with @ChuanqiXu9 and @kadircet

Closes #112635

show more ...

[clangd] Fix the modification detection logic in `ClangdLSPServer::applyConfiguration`. (#115438)

Prior to this, the "old != new" check would always evaluate to true because it was comparing a pre-m

[clangd] Fix the modification detection logic in `ClangdLSPServer::applyConfiguration`. (#115438)

Prior to this, the "old != new" check would always evaluate to true because it was comparing a pre-mangling new command to a post-mangling old command.

show more ...

[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (#66462)

Alternatives to https://reviews.llvm.org/D153114.

Try to address https://github.com/clangd/clangd/issues/1293.

S

[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (#66462)

Alternatives to https://reviews.llvm.org/D153114.

Try to address https://github.com/clangd/clangd/issues/1293.

See the links for design ideas and the consensus so far. We want to have
some initial support in clang18.

This is the initial support for C++20 Modules in clangd.
As suggested by sammccall in https://reviews.llvm.org/D153114,
we should minimize the scope of the initial patch to make it easier
to review and understand so that every one are in the same page:

> Don't attempt any cross-file or cross-version coordination: i.e. don't
> try to reuse BMIs between different files, don't try to reuse BMIs
> between (preamble) reparses of the same file, don't try to persist the
> module graph. Instead, when building a preamble, synchronously scan
> for the module graph, build the required PCMs on the single preamble
> thread with filenames private to that preamble, and then proceed to
> build the preamble.

This patch reflects the above opinions.

# Testing in real-world project

I tested this with a modularized library:
https://github.com/alibaba/async_simple/tree/CXX20Modules. This library
has 3 modules (async_simple, std and asio) and 65 module units. (Note
that a module consists of multiple module units). Both `std` module and
`asio` module have 100k+ lines of code (maybe more, I didn't count). And
async_simple itself has 8k lines of code. This is the scale of the
project.

The result shows that it works pretty well, ..., well, except I need to
wait roughly 10s after opening/editing any file. And this falls in our
expectations. We know it is hard to make it perfect in the first move.

# What this patch does in detail

- Introduced an option `--experimental-modules-support` for the support
for C++20 Modules. So that no matter how bad this is, it wouldn't affect
current users. Following off the page, we'll assume the option is
enabled.
- Introduced two classes `ModuleFilesInfo` and
`ModuleDependencyScanner`. Now `ModuleDependencyScanner` is only used by
`ModuleFilesInfo`.
- The class `ModuleFilesInfo` records the built module files for
specific single source file. The module files can only be built by the
static member function `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef
File, ...)`.
- The class `PreambleData` adds a new member variable with type
`ModuleFilesInfo`. This refers to the needed module files for the
current file. It means the module files info is part of the preamble,
which is suggested in the first patch too.
- In `isPreambleCompatible()`, we add a call to
`ModuleFilesInfo::CanReuse()` to check if the built module files are
still up to date.
- When we build the AST for a source file, we will load the built module
files from ModuleFilesInfo.

# What we need to do next

Let's split the TODOs into clang part and clangd part to make things
more clear.

The TODOs in the clangd part include:
1. Enable reusing module files across source files. The may require us
to bring a ModulesManager like thing which need to handle `scheduling`,
`the possibility of BMI version conflicts` and `various events that can
invalidate the module graph`.
2. Get a more efficient method to get the `<module-name> ->
<module-unit-source>` map. Currently we always scan the whole project
during `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef File, ...)`.
This is clearly inefficient even if the scanning process is pretty fast.
I think the potential solutions include:
- Make a global scanner to monitor the state of every source file like I
did in the first patch. The pain point is that we need to take care of
the data races.
- Ask the build systems to provide the map just like we ask them to
provide the compilation database.
3. Persist the module files. So that we can reuse module files across
clangd invocations or even across clangd instances.

TODOs in the clang part include:
1. Clang should offer an option/mode to skip writing/reading the bodies
of the functions. Or even if we can requrie the parser to skip parsing
the function bodies.

And it looks like we can say the support for C++20 Modules is initially
workable after we made (1) and (2) (or even without (2)).

show more ...

[clang-tools-extra] Remove remaining uses of llvm::Optional (NFC)

This patch removes the unused "using" declaration and removes #include
"llvm/ADT/Optional.h".

This is part of an effort to migrate

[clang-tools-extra] Remove remaining uses of llvm::Optional (NFC)

This patch removes the unused "using" declaration and removes #include
"llvm/ADT/Optional.h".

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[clang-tools-extra] Use std::optional instead of llvm::Optional (NFC)

This patch replaces (llvm::|)Optional< with std::optional<. I'll post
a separate patch to clean up the "using" declarations, #i

[clang-tools-extra] Use std::optional instead of llvm::Optional (NFC)

This patch replaces (llvm::|)Optional< with std::optional<. I'll post
a separate patch to clean up the "using" declarations, #include
"llvm/ADT/Optional.h", etc.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[clang-tools-extra] Add #include <optional> (NFC)

This patch adds #include <optional> to those files containing
llvm::Optional<...> or Optional<...>.

I'll post a separate patch to actually replace

[clang-tools-extra] Add #include <optional> (NFC)

This patch adds #include <optional> to those files containing
llvm::Optional<...> or Optional<...>.

I'll post a separate patch to actually replace llvm::Optional with
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

JSON: llvm::Optional => std::optional

Many files are from language servers.

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

[clang-tools-extra] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the

[clang-tools-extra] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[clangd] Perform system include extraction inside CommandMangler

It needs to run after edits from config files are applied to
the compile command (because the config file may specify the
compiler),

[clangd] Perform system include extraction inside CommandMangler

It needs to run after edits from config files are applied to
the compile command (because the config file may specify the
compiler), and before resolveDriver() runs at the end of
CommandMangler.

As part of this change, QueryDriverDatabase is renamed to
SystemIncludeExtractor and is no longer a GlobalCompilationDatabase.

Fixes https://github.com/clangd/clangd/issues/1089
Fixes https://github.com/clangd/clangd/issues/1173
Fixes https://github.com/clangd/clangd/issues/1263

Differential Revision: https://reviews.llvm.org/D133757

show more ...

[clangd] Pass the entire tooling::CompileCommand to CommandMangler

This gives CommandMangler access to other fields of
tooling::CompileCommand as well, e.g. Directory.

Differential Revision: https:

[clangd] Pass the entire tooling::CompileCommand to CommandMangler

This gives CommandMangler access to other fields of
tooling::CompileCommand as well, e.g. Directory.

Differential Revision: https://reviews.llvm.org/D133756

show more ...

[clangd] Clean up unused includes. NFCI

Add includes where needed to fix build.
Haven't systematically added used headers, so there is still accidental
dependency on transitive includes.

[clang-tools-extra] Remove unused forward declarations (NFC)

[clangd] Allow configuration database to be specified in config.

This allows for more flexibility than -compile-commands-dir or ancestor
discovery.

See https://github.com/clangd/clangd/issues/116

[clangd] Allow configuration database to be specified in config.

This allows for more flexibility than -compile-commands-dir or ancestor
discovery.

See https://github.com/clangd/clangd/issues/116

Differential Revision: https://reviews.llvm.org/D95057

show more ...

[clangd] Move DirBasedCDB broadcasting onto its own thread.

This is on the critical path (it blocks getting the compile command for
the first file).

It's not trivially fast: it involves processing

[clangd] Move DirBasedCDB broadcasting onto its own thread.

This is on the critical path (it blocks getting the compile command for
the first file).

It's not trivially fast: it involves processing all filenames in the CDB
and doing some IO to look for shadowing CDBs.

And we may make this slower soon - making CDB configurable implies evaluating
the config for each listed to see which ones really are owned by the
broadcasted CDB.

Differential Revision: https://reviews.llvm.org/D94606

show more ...

[clangd] Allow CDBs to have background work to block on.

In preparation for moving DirectoryBasedCompilationDatabase broadcasting off
the main thread.

Differential Revision: https://reviews.llvm.or

[clangd] Allow CDBs to have background work to block on.

In preparation for moving DirectoryBasedCompilationDatabase broadcasting off
the main thread.

Differential Revision: https://reviews.llvm.org/D94603

show more ...

[clangd] Split out a base class for delegating GlobalCompilationDatabases. NFC

This prepares for adding another delegatable method (blockUntilIdle) to GCDB.

[clangd] Add hot-reload of compile_commands.json and compile_flags.txt

When querying the CDB, we stat the underlying file to check it hasn't changed.
We don't do this every time, but only if we didn

[clangd] Add hot-reload of compile_commands.json and compile_flags.txt

When querying the CDB, we stat the underlying file to check it hasn't changed.
We don't do this every time, but only if we didn't check within 5 seconds.

This behavior only exists for compile_commands.json and compile_flags.txt.
The CDB plugin system doesn't expose enough information to handle others.

Slight behavior change: we now only look for `build/compile_commands.json`
rather than trying every CDB strategy under `build` subdirectories.

Differential Revision: https://reviews.llvm.org/D92663

show more ...

Reland [clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This reverts commit 4d956af594c5adc9d566d1846d86dd89c70c9c0b.

Assertion failures on windows fixed by
965d71c69acce658e9e3d

Reland [clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This reverts commit 4d956af594c5adc9d566d1846d86dd89c70c9c0b.

Assertion failures on windows fixed by
965d71c69acce658e9e3de00b25a351b00937820

show more ...

Revert [clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This reverts commit 8a4390dc4768fcd929a7231717980ccb28f124f7.

(The reland did not have the bugfix, just trying to get more

Revert [clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This reverts commit 8a4390dc4768fcd929a7231717980ccb28f124f7.

(The reland did not have the bugfix, just trying to get more details
from the buildbots)

show more ...

Reland [clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This reverts commit de4f5519015cc97f28718d90cc6dac73c0a15161.

More debug output to try to pin down an impossible condition.

Revert "[clangd] Extract per-dir CDB cache to its own threadsafe class. NFC"

This reverts commit 634a377bd8cbaa515a58295cfd85dcb6a21381c1.
Breaks tests on Windows, see https://reviews.llvm.org/D9238

Revert "[clangd] Extract per-dir CDB cache to its own threadsafe class. NFC"

This reverts commit 634a377bd8cbaa515a58295cfd85dcb6a21381c1.
Breaks tests on Windows, see https://reviews.llvm.org/D92381#2443407

show more ...

[clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This is a step towards making compile_commands.json reloadable.

The idea is:
- in addition to rare CDB loads we're soon going to

[clangd] Extract per-dir CDB cache to its own threadsafe class. NFC

This is a step towards making compile_commands.json reloadable.

The idea is:
- in addition to rare CDB loads we're soon going to have somewhat-rare CDB
reloads and fairly-common stat() of files to validate the CDB
- so stop doing all our work under a big global lock, instead using it to
acquire per-directory structures with their own locks
- each directory can be refreshed from disk every N seconds, like filecache
- avoid locking these at all in the most common case: directory has no CDB

Differential Revision: https://reviews.llvm.org/D92381

show more ...

[clangd] Always retrieve ProjectInfo from Base in OverlayCDB

Summary:
Clangd is returning current working directory for overriden commands.
This can cause inconsistencies between:
- header and the m

[clangd] Always retrieve ProjectInfo from Base in OverlayCDB

Summary:
Clangd is returning current working directory for overriden commands.
This can cause inconsistencies between:
- header and the main files, as OverlayCDB only contains entries for the main
files it direct any queries for the headers to the base, creating a
discrepancy between the two.
- different clangd instances, as the results will be different depending on the
timing of execution of the query and override of the command. hence clangd
might see two different project infos for the same file between different
invocations.
- editors and the way user has invoked it, as current working directory of
clangd will depend on those, hence even when there's no underlying base CWD
might change depending on the editor, or the directory user has started the
editor in.

This patch gets rid of that discrepency by always directing queries to base or
returning llvm::None in absence of it.

For a sample bug see https://reviews.llvm.org/D83099#2154185.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D83934

show more ...

[clangd] Move non-clang base pieces into separate support/ lib. NFCI

Summary:
This enforces layering, reduces a sprawling clangd/ directory, and makes life
easier for embedders.

Reviewers: kbobyrev

[clangd] Move non-clang base pieces into separate support/ lib. NFCI

Summary:
This enforces layering, reduces a sprawling clangd/ directory, and makes life
easier for embedders.

Reviewers: kbobyrev

Subscribers: mgorny, ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D79014

show more ...

[clangd] (take 2) Try harder to find a plausible `clang` as argv0, particularly on Mac.

Summary:
This was originally committed in 88bccded8fa169481fa367debf5ec615640635a1,
and reverted in 93f77617ab

[clangd] (take 2) Try harder to find a plausible `clang` as argv0, particularly on Mac.

Summary:
This was originally committed in 88bccded8fa169481fa367debf5ec615640635a1,
and reverted in 93f77617abba512d2861e2fc50ce385883f587b6.

This version is now much more testable: the "detect toolchain properties" part
is still not tested but also not active in tests.
All the command manipulation based on the detected properties is
directly tested, and also not active in other tests.

Fixes https://github.com/clangd/clangd/issues/211
Fixes https://github.com/clangd/clangd/issues/178

Reviewers: kbobyrev, ilya-biryukov

Subscribers: mgorny, ormris, cfe-commits, usaxena95, kadircet, arphaman, jkorous, MaskRay

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71029

show more ...