[clang] Extend diagnose_if to accept more detailed warning information, take 2 (#119712)

This is take two of #70976. This iteration of the patch makes sure that
custom
diagnostics without any warnin

[clang] Extend diagnose_if to accept more detailed warning information, take 2 (#119712)

This is take two of #70976. This iteration of the patch makes sure that
custom
diagnostics without any warning group don't get promoted by `-Werror` or
`-Wfatal-errors`.

This implements parts of the extension proposed in
https://discourse.llvm.org/t/exposing-the-diagnostic-engine-to-c/73092/7.

Specifically, this makes it possible to specify a diagnostic group in an
optional third argument.

show more ...

Reapply "[NFC] Explicitly pass a VFS when creating DiagnosticsEngine (#115852)"

This reverts commit a1153cd6fedd4c906a9840987934ca4712e34cb2 with fixes
to lldb breakages.

Fixes https://github.com/l

Reapply "[NFC] Explicitly pass a VFS when creating DiagnosticsEngine (#115852)"

This reverts commit a1153cd6fedd4c906a9840987934ca4712e34cb2 with fixes
to lldb breakages.

Fixes https://github.com/llvm/llvm-project/issues/117145.

show more ...

Revert "[NFC] Explicitly pass a VFS when creating DiagnosticsEngine (#115852)"

Reverted for causing:
https://github.com/llvm/llvm-project/issues/117145

This reverts commit bdd10d9d249bd1c2a45e3de56

Revert "[NFC] Explicitly pass a VFS when creating DiagnosticsEngine (#115852)"

Reverted for causing:
https://github.com/llvm/llvm-project/issues/117145

This reverts commit bdd10d9d249bd1c2a45e3de56a5accd97e953458.

show more ...

[NFC] Explicitly pass a VFS when creating DiagnosticsEngine (#115852)

Starting with 41e3919ded78d8870f7c95e9181c7f7e29aa3cc4 DiagnosticsEngine
creation might perform IO. It was implicitly defaultin

[NFC] Explicitly pass a VFS when creating DiagnosticsEngine (#115852)

Starting with 41e3919ded78d8870f7c95e9181c7f7e29aa3cc4 DiagnosticsEngine
creation might perform IO. It was implicitly defaulting to
getRealFileSystem. This patch makes it explicit by pushing the decision
making to callers.

It uses ambient VFS if one is available, and keeps using
`getRealFileSystem` if there aren't any VFS.

show more ...

Revert "[clang] Extend diagnose_if to accept more detailed warning information (#70976)"

This reverts commit e39205654dc11c50bd117e8ccac243a641ebd71f.

There are further discussions in
https://githu

Revert "[clang] Extend diagnose_if to accept more detailed warning information (#70976)"

This reverts commit e39205654dc11c50bd117e8ccac243a641ebd71f.

There are further discussions in
https://github.com/llvm/llvm-project/pull/70976, happening for past two
weeks. Since there were no responses for couple weeks now, reverting
until author is back.

show more ...

[clang-tools-extra] Don't flush llvm::raw_string_ostream (NFC)

Don't call raw_string_ostream::flush(), which is essentially a no-op.
As specified in the docs, raw_string_ostream is always unbuffered

[clang-tools-extra] Don't flush llvm::raw_string_ostream (NFC)

Don't call raw_string_ostream::flush(), which is essentially a no-op.
As specified in the docs, raw_string_ostream is always unbuffered.
( 65b13610a5226b84889b923bae884ba395ad084d for further reference )

show more ...

Revert "[SystemZ][z/OS] Propagate IsText parameter to open text files as text (#107906)"

This reverts commit edf3b277a5f2ebe144827ed47463c22743cac5f9.

[SystemZ][z/OS] Propagate IsText parameter to open text files as text (#107906)

This patch adds an IsText parameter to the following functions
openFileForRead, getBufferForFile, getBufferForFileImp

[SystemZ][z/OS] Propagate IsText parameter to open text files as text (#107906)

This patch adds an IsText parameter to the following functions
openFileForRead, getBufferForFile, getBufferForFileImpl and determines
whether a file is text by querying the file tag on z/OS. The default is
set to OF_Text instead of OF_None, this change in value does not affect
any other platforms other than z/OS.

show more ...

Reapply "Reapply "[clang] Extend diagnose_if to accept more detailed warning information (#70976)" (#108453)"

This reverts commit e1bd9740faa62c11cc785a7b70ec1ad17e286bd1.

Fixes incorrect use of th

Reapply "Reapply "[clang] Extend diagnose_if to accept more detailed warning information (#70976)" (#108453)"

This reverts commit e1bd9740faa62c11cc785a7b70ec1ad17e286bd1.

Fixes incorrect use of the `DiagnosticsEngine` in the clangd tests.

show more ...

Revert "Reapply "[clang] Extend diagnose_if to accept more detailed warning information (#70976)" (#108453)"

This reverts commit e7f782e7481cea23ef452a75607d3d61f5bd0d22.

This had UBSan failures:

Revert "Reapply "[clang] Extend diagnose_if to accept more detailed warning information (#70976)" (#108453)"

This reverts commit e7f782e7481cea23ef452a75607d3d61f5bd0d22.

This had UBSan failures:

[----------] 1 test from ConfigCompileTests
[ RUN ] ConfigCompileTests.DiagnosticSuppression
Config fragment: compiling <unknown>:0 -> 0x00007B8366E2F7D8 (trusted=false)
/usr/local/google/home/fmayer/large/llvm-project/llvm/include/llvm/ADT/IntrusiveRefCntPtr.h:203:33: runtime error: reference binding to null pointer of type 'clang::DiagnosticIDs'

UndefinedBehaviorSanitizer: undefined-behavior /usr/local/google/home/fmayer/large/llvm-project/llvm/include/llvm/ADT/IntrusiveRefCntPtr.h:203:33

Pull Request: https://github.com/llvm/llvm-project/pull/108645

show more ...

Reapply "[clang] Extend diagnose_if to accept more detailed warning information (#70976)" (#108453)

This reverts commit e0cd11eba526234ca14a0b91f5598ca3363b6aca.

Update the use of `getWarningOpti

Reapply "[clang] Extend diagnose_if to accept more detailed warning information (#70976)" (#108453)

This reverts commit e0cd11eba526234ca14a0b91f5598ca3363b6aca.

Update the use of `getWarningOptionForDiag` in flang to use the
DiagnosticIDs.

show more ...

Revert "[clang] Extend diagnose_if to accept more detailed warning information (#70976)"

This reverts commit 030c6da7af826b641db005be925b20f956c3a6bb.

Several build bots are failing:
https://lab.ll

Revert "[clang] Extend diagnose_if to accept more detailed warning information (#70976)"

This reverts commit 030c6da7af826b641db005be925b20f956c3a6bb.

Several build bots are failing:
https://lab.llvm.org/buildbot/#/builders/89/builds/6211
https://lab.llvm.org/buildbot/#/builders/157/builds/7578
https://lab.llvm.org/buildbot/#/builders/140/builds/6429

show more ...

[clang] Extend diagnose_if to accept more detailed warning information (#70976)

This implements parts of the extension proposed in
https://discourse.llvm.org/t/exposing-the-diagnostic-engine-to-c/7

[clang] Extend diagnose_if to accept more detailed warning information (#70976)

This implements parts of the extension proposed in
https://discourse.llvm.org/t/exposing-the-diagnostic-engine-to-c/73092/7.

Specifically, this makes it possible to specify a diagnostic group in an
optional third argument.

show more ...

[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (#66462)

Alternatives to https://reviews.llvm.org/D153114.

Try to address https://github.com/clangd/clangd/issues/1293.

S

[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (#66462)

Alternatives to https://reviews.llvm.org/D153114.

Try to address https://github.com/clangd/clangd/issues/1293.

See the links for design ideas and the consensus so far. We want to have
some initial support in clang18.

This is the initial support for C++20 Modules in clangd.
As suggested by sammccall in https://reviews.llvm.org/D153114,
we should minimize the scope of the initial patch to make it easier
to review and understand so that every one are in the same page:

> Don't attempt any cross-file or cross-version coordination: i.e. don't
> try to reuse BMIs between different files, don't try to reuse BMIs
> between (preamble) reparses of the same file, don't try to persist the
> module graph. Instead, when building a preamble, synchronously scan
> for the module graph, build the required PCMs on the single preamble
> thread with filenames private to that preamble, and then proceed to
> build the preamble.

This patch reflects the above opinions.

# Testing in real-world project

I tested this with a modularized library:
https://github.com/alibaba/async_simple/tree/CXX20Modules. This library
has 3 modules (async_simple, std and asio) and 65 module units. (Note
that a module consists of multiple module units). Both `std` module and
`asio` module have 100k+ lines of code (maybe more, I didn't count). And
async_simple itself has 8k lines of code. This is the scale of the
project.

The result shows that it works pretty well, ..., well, except I need to
wait roughly 10s after opening/editing any file. And this falls in our
expectations. We know it is hard to make it perfect in the first move.

# What this patch does in detail

- Introduced an option `--experimental-modules-support` for the support
for C++20 Modules. So that no matter how bad this is, it wouldn't affect
current users. Following off the page, we'll assume the option is
enabled.
- Introduced two classes `ModuleFilesInfo` and
`ModuleDependencyScanner`. Now `ModuleDependencyScanner` is only used by
`ModuleFilesInfo`.
- The class `ModuleFilesInfo` records the built module files for
specific single source file. The module files can only be built by the
static member function `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef
File, ...)`.
- The class `PreambleData` adds a new member variable with type
`ModuleFilesInfo`. This refers to the needed module files for the
current file. It means the module files info is part of the preamble,
which is suggested in the first patch too.
- In `isPreambleCompatible()`, we add a call to
`ModuleFilesInfo::CanReuse()` to check if the built module files are
still up to date.
- When we build the AST for a source file, we will load the built module
files from ModuleFilesInfo.

# What we need to do next

Let's split the TODOs into clang part and clangd part to make things
more clear.

The TODOs in the clangd part include:
1. Enable reusing module files across source files. The may require us
to bring a ModulesManager like thing which need to handle `scheduling`,
`the possibility of BMI version conflicts` and `various events that can
invalidate the module graph`.
2. Get a more efficient method to get the `<module-name> ->
<module-unit-source>` map. Currently we always scan the whole project
during `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef File, ...)`.
This is clearly inefficient even if the scanning process is pretty fast.
I think the potential solutions include:
- Make a global scanner to monitor the state of every source file like I
did in the first patch. The pain point is that we need to take care of
the data races.
- Ask the build systems to provide the map just like we ask them to
provide the compilation database.
3. Persist the module files. So that we can reuse module files across
clangd invocations or even across clangd instances.

TODOs in the clang part include:
1. Clang should offer an option/mode to skip writing/reading the bodies
of the functions. Or even if we can requrie the parser to skip parsing
the function bodies.

And it looks like we can say the support for C++20 Modules is initially
workable after we made (1) and (2) (or even without (2)).

show more ...

[clangd] Fix data race surfaced in clangd-tsan buildbot

We can have concurrent accesses to same PreambleData (e.g.
code-completion and ast-builds). Hence we need to
deep copy TargetOpts.

[clangd] Use TargetOpts from preamble when building ASTs (#88381)

Building ASTs with compile flags that are incompatible to the ones used
for the Preamble are not really supported by clang and can

[clangd] Use TargetOpts from preamble when building ASTs (#88381)

Building ASTs with compile flags that are incompatible to the ones used
for the Preamble are not really supported by clang and can trigger
crashes.

In an ideal world, we should be re-using not only TargetOpts, but the
full ParseInputs from the Preamble to prevent such failures.

Unfortunately current contracts of ThreadSafeFS makes this a non-safe
change for certain implementations. As there are no guarantees that the
same ThreadSafeFS is going to be valid in the Context::current() we're
building the AST in.

show more ...

[clang] NFCI: Use `FileEntryRef` in `ASTReader::GetHeaderFileInfo()`

This is the `ASTReader` counterpart to PR #67383.

[clang] NFCI: Use `FileEntryRef` in `PreamblePatch`

[clang] NFCI: Change returned LanguageOptions pointer to reference

[clangd] SIGSEGV at clangd: DiagnosticConsumer Is Used After Free

This is a follow-up patch for D148088. The dynamic symbol index (`FileIndex::updatePreamble`) may run in a separate thread, and the

[clangd] SIGSEGV at clangd: DiagnosticConsumer Is Used After Free

This is a follow-up patch for D148088. The dynamic symbol index (`FileIndex::updatePreamble`) may run in a separate thread, and the `DiagnosticConsumer` that is set up in `buildPreamble` might go out of scope before it is used. This could result in a SIGSEGV when attempting to call any method of the `DiagnosticConsumer` class.

The function `buildPreamble` sets up the `DiagnosticConsumer` as follows:
```
... buildPreamble(...) {
...
StoreDiags PreambleDiagnostics;
...
llvm::IntrusiveRefCntPtr<DiagnosticsEngine> PreambleDiagsEngine =
CompilerInstance::createDiagnostics(&CI.getDiagnosticOpts(),
&PreambleDiagnostics,
/*ShouldOwnClient=*/false);
...
// The call might use the diagnostic consumer in a separate thread
PreambleCallback(...)
...
}
```
`PreambleDiagnostics` might be out of scope for `buildPreamble` function when we call it inside `PreambleCallback` in a separate thread.

The Fix
The fix involves replacing the client (DiagnosticConsumer) with an `IgnoringDiagConsumer` instance, which will print messages to the clangd log.

Alternatively, we can replace `PreambleDiagnostics` with an object that is owned by `DiagnosticsEngine`.

Note
There is no corresponding LIT/GTest for this issue, since there is a specific race condition that is difficult to reproduce within a test framework.

Test Plan:
```
ninja check-clangd
```

Reviewed By: kadircet, sammccall

Differential Revision: https://reviews.llvm.org/D159363

show more ...

[clang-tools-extra] Use range-based for loops (NFC)

Reland "[clangd] Always run preamble indexing on a separate thread"

This reverts commit 92c0546941190973e9201a08fa10edf27cb6992d.

Prevents tsan issues by resetting ref-counted-pointers eagerly, bef

Reland "[clangd] Always run preamble indexing on a separate thread"

This reverts commit 92c0546941190973e9201a08fa10edf27cb6992d.

Prevents tsan issues by resetting ref-counted-pointers eagerly, before
passing the copies into a new thread.

show more ...

[clangd] Update symbol collector to use include-cleaner.

Differential Revision: https://reviews.llvm.org/D152900

[clangd] Always allow diagnostics from stale preambles

We've been running this internally for months now, without any
stability or correctness concerns. It has ~40% speed up on incremental
diagnosti

[clangd] Always allow diagnostics from stale preambles

We've been running this internally for months now, without any
stability or correctness concerns. It has ~40% speed up on incremental
diagnostics latencies (as preamble can get invalidated through code completion
etc.).

Differential Revision: https://reviews.llvm.org/D153882

show more ...

[RFC][clangd] Move preamble index out of document open critical path

We would like to move the preamble index out of the critical path.
This patch is an RFC to get feedback on the correct implementa

[RFC][clangd] Move preamble index out of document open critical path

We would like to move the preamble index out of the critical path.
This patch is an RFC to get feedback on the correct implementation and potential pitfalls to keep into consideration.

I am not entirely sure if the lazy AST initialisation would create using Preamble AST in parallel. I tried with tsan enabled clangd but it seems to work OK (at least for the cases I tried)

Reviewed By: kadircet

Differential Revision: https://reviews.llvm.org/D148088

show more ...