Reapply "[clang] Introduce diagnostics suppression mappings (#112517)"

This reverts commit 5f140ba54794fe6ca379362b133eb27780e363d7.

Revert "[clang] Introduce diagnostics suppression mappings (#112517)"

This reverts commit 12e3ed8de8c6063b15916b3faf67c8c9cd17df1f.
This reverts commit 41e3919ded78d8870f7c95e9181c7f7e29aa3cc4.

The

Revert "[clang] Introduce diagnostics suppression mappings (#112517)"

This reverts commit 12e3ed8de8c6063b15916b3faf67c8c9cd17df1f.
This reverts commit 41e3919ded78d8870f7c95e9181c7f7e29aa3cc4.

There are some buildbot breakages in
https://lab.llvm.org/buildbot/#/builders/18/builds/6832.

show more ...

[clang] Introduce diagnostics suppression mappings (#112517)

This implements

https://discourse.llvm.org/t/rfc-add-support-for-controlling-diagnostics-severities-at-file-level-granularity-through-

[clang] Introduce diagnostics suppression mappings (#112517)

This implements

https://discourse.llvm.org/t/rfc-add-support-for-controlling-diagnostics-severities-at-file-level-granularity-through-command-line/81292.

Users now can suppress warnings for certain headers by providing a
mapping with globs, a sample file looks like:
```
[unused]
src:*
src:*clang/*=emit
```

This will suppress warnings from `-Wunused` group in all files that
aren't under `clang/` directory. This mapping file can be passed to
clang via `--warning-suppression-mappings=foo.txt`.

At a high level, mapping file is stored in DiagnosticOptions and then
processed with rest of the warning flags when creating a
DiagnosticsEngine. This is a functor that uses SpecialCaseLists
underneath to match against globs coming from the mappings file.

This implies processing warning options now performs IO, relevant
interfaces are updated to take in a VFS, falling back to RealFileSystem
when one is not available.

show more ...

[CommandLine] Avoid ManagedStatic.h include (NFC)

The two variables using ManagedStatic that are exported by this
header are not actually used anywhere -- they are used through
SubCommand::getTopLev

[CommandLine] Avoid ManagedStatic.h include (NFC)

The two variables using ManagedStatic that are exported by this
header are not actually used anywhere -- they are used through
SubCommand::getTopLevel() and SubCommand::getAll() instead.
Drop the extern declarations and the include.

show more ...

[NFC] [C++20] [Modules] [Reduced BMI] Make sure the size of reduced BMI is not large than full BMI

Before this patch, the size of the reduced BMI may be large than the
full BMI when the source codes

[NFC] [C++20] [Modules] [Reduced BMI] Make sure the size of reduced BMI is not large than full BMI

Before this patch, the size of the reduced BMI may be large than the
full BMI when the source codes is pretty small. This violates the design
principles. The root cause is an oversight that we skipped something
in full BMI but forgot to make it in reduced BMI.

show more ...

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
s

[clang][lex] Always pass suggested module to `InclusionDirective()` callback (#81061)

This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
suggested module, regardless of whether it was actually imported or not.
The extra `bool ModuleImported` parameter then denotes whether the
header `#include` will be automatically translated into import the the
module.

The main change is in `clang/lib/Lex/PPDirectives.cpp`, where we take
care to not modify `SuggestedModule` after it's been populated by
`LookupHeaderIncludeOrImport()`. We now exclusively use the `SM`
(`ModuleToImport`) variable instead, which has been equivalent to
`SuggestedModule` until now. This allows us to use the original
non-modified `SuggestedModule` for the callback itself.

(This patch turns out to be necessary for
https://github.com/apple/llvm-project/pull/8011).

show more ...

Reapply "[clang] NFCI: Adopt `SourceManager::getFileEntryRefForID()`"

This reapplies ddbcc10b9e26b18f6a70e23d0611b9da75ffa52f, except for a tiny part that was reverted separately: 65331da0032ab4253a

Reapply "[clang] NFCI: Adopt `SourceManager::getFileEntryRefForID()`"

This reapplies ddbcc10b9e26b18f6a70e23d0611b9da75ffa52f, except for a tiny part that was reverted separately: 65331da0032ab4253a4bc0ddcb2da67664bd86a9. That will be reapplied later on, since it turned out to be more involved.

This commit is enabled by 5523fefb01c282c4cbcaf6314a9aaf658c6c145f and f0f548a65a215c450d956dbcedb03656449705b9, specifically the part that makes 'clang-tidy/checkers/misc/header-include-cycle.cpp' separator agnostic.

show more ...

Revert "[clang] NFCI: Adopt `SourceManager::getFileEntryRefForID()`"

This reverts commit ddbcc10b9e26b18f6a70e23d0611b9da75ffa52f.

The 'clang-tidy/checkers/misc/header-include-cycle.cpp' test start

Revert "[clang] NFCI: Adopt `SourceManager::getFileEntryRefForID()`"

This reverts commit ddbcc10b9e26b18f6a70e23d0611b9da75ffa52f.

The 'clang-tidy/checkers/misc/header-include-cycle.cpp' test started failing on Windows: https://lab.llvm.org/buildbot/#/builders/216/builds/26855.

show more ...

[clang] NFCI: Adopt `SourceManager::getFileEntryRefForID()`

This commit replaces some calls to the deprecated `FileEntry::getName()` with `FileEntryRef::getName()` by swapping current usages of `Sou

[clang] NFCI: Adopt `SourceManager::getFileEntryRefForID()`

This commit replaces some calls to the deprecated `FileEntry::getName()` with `FileEntryRef::getName()` by swapping current usages of `SourceManager::getFileEntryForID()` with `SourceManager::getFileEntryRefForID()`. This lowers the number of usages of the deprecated `FileEntry::getName()` from 95 to 50.

show more ...

[clang] NFCI: Change returned LanguageOptions pointer to reference

[clang] NFCI: Use `DirectoryEntryRef` in `PrecompiledPreamble`

This removes some uses of the deprecated `DirectoryEntry::getName()`.

[libclang] Add API to override preamble storage path

TempPCHFile::create() calls llvm::sys::fs::createTemporaryFile() to
create a file named preamble-*.pch in a system temporary directory. This
comm

[libclang] Add API to override preamble storage path

TempPCHFile::create() calls llvm::sys::fs::createTemporaryFile() to
create a file named preamble-*.pch in a system temporary directory. This
commit allows overriding the directory where these often many and large
preamble-*.pch files are stored.

The referenced bug report requests the ability to override the temporary
directory path used by libclang. However, overriding the return value of
llvm::sys::path::system_temp_directory() was rejected during code review
as improper and because it would negatively affect multithreading
performance. Finding all places where libclang uses the temporary
directory is very difficult. Therefore this commit is limited to
override libclang's single known use of the temporary directory.

This commit allows to override the preamble storage path only during
CXIndex construction to avoid multithreading issues and ensure that all
preambles are stored in the same directory. For the same multithreading
and consistency reasons, this commit deprecates
clang_CXIndex_setGlobalOptions() and
clang_CXIndex_setInvocationEmissionPathOption() in favor of specifying
these options during CXIndex construction.

Adding a new CXIndex constructor function each time a new initialization
argument is needed leads to either a large number of function parameters
unneeded by most libclang users or to an exponential number of overloads
that support different usage requirements. Therefore this commit
introduces a new extensible struct CXIndexOptions and a general function
clang_createIndexWithOptions().

A libclang user passes a desired preamble storage path to
clang_createIndexWithOptions(), which stores it in
CIndexer::PreambleStoragePath. Whenever
clang_parseTranslationUnit_Impl() is called, it passes
CIndexer::PreambleStoragePath to ASTUnit::LoadFromCommandLine(), which
stores this argument in ASTUnit::PreambleStoragePath. Whenever
ASTUnit::getMainBufferWithPrecompiledPreamble() is called, it passes
ASTUnit::PreambleStoragePath to PrecompiledPreamble::Build().
PrecompiledPreamble::Build() forwards the corresponding StoragePath
argument to TempPCHFile::create(). If StoragePath is not empty,
TempPCHFile::create() stores the preamble-*.pch file in the directory at
the specified path rather than in the system temporary directory.

The analysis below proves that this passing around of the
PreambleStoragePath string is sufficient to guarantee that the libclang
user override is used in TempPCHFile::create(). The analysis ignores API
uses in test code.

TempPCHFile::create() is called only in PrecompiledPreamble::Build().
PrecompiledPreamble::Build() is called only in two places: one in
clangd, which is not used by libclang, and one in
ASTUnit::getMainBufferWithPrecompiledPreamble().
ASTUnit::getMainBufferWithPrecompiledPreamble() is called in 3 places:

ASTUnit::LoadFromCompilerInvocation() [analyzed below].
ASTUnit::Reparse(), which in turn is called only from
clang_reparseTranslationUnit_Impl(), which in turn is called only from
clang_reparseTranslationUnit(). clang_reparseTranslationUnit() is never
called in LLVM code, but is part of public libclang API. This function's
documentation requires its translation unit argument to have been built
with clang_createTranslationUnitFromSourceFile().
clang_createTranslationUnitFromSourceFile() delegates its work to
clang_parseTranslationUnit(), which delegates to
clang_parseTranslationUnit2(), which delegates to
clang_parseTranslationUnit2FullArgv(), which delegates to
clang_parseTranslationUnit_Impl(), which passes
CIndexer::PreambleStoragePath to the ASTUnit it creates.

ASTUnit::CodeComplete() passes AllowRebuild = false to
ASTUnit::getMainBufferWithPrecompiledPreamble(), which makes it return
nullptr before calling PrecompiledPreamble::Build().

Both ASTUnit::LoadFromCompilerInvocation() overloads (one of which
delegates its work to another) call
ASTUnit::getMainBufferWithPrecompiledPreamble() only if their argument
PrecompilePreambleAfterNParses > 0. LoadFromCompilerInvocation() is
called in:

ASTBuilderAction::runInvocation() keeps the default parameter value
of PrecompilePreambleAfterNParses = 0, meaning that the preamble file is
never created from here.

ASTUnit::LoadFromCommandLine().
ASTUnit::LoadFromCommandLine() is called in two places:

CrossTranslationUnitContext::ASTLoader::loadFromSource() keeps the
default parameter value of PrecompilePreambleAfterNParses = 0, meaning
that the preamble file is never created from here.

clang_parseTranslationUnit_Impl(), which passes
CIndexer::PreambleStoragePath to the ASTUnit it creates.

Therefore, the overridden preamble storage path is always used in
TempPCHFile::create().

TempPCHFile::create() uses PreambleStoragePath in the same way as
LibclangInvocationReporter() uses InvocationEmissionPath. The existing
documentation for clang_CXIndex_setInvocationEmissionPathOption() does
not specify ownership, encoding, separator or relative vs absolute path
requirements. So the documentation for
CXIndexOptions::PreambleStoragePath doesn't either. The assumptions are:

no ownership transfer;
UTF-8 encoding;
native separators.
Both relative and absolute paths are supported.

The added API works as expected in KDevelop:
https://invent.kde.org/kdevelop/kdevelop/-/merge_requests/283

Fixes: https://github.com/llvm/llvm-project/issues/51847
Differential Revision: https://reviews.llvm.org/D143418

show more ...

Use std::nullopt instead of None in comments (NFC)

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.

I tested this with GCC 7.5, the oldest supported GCC I had around.

Differential Revision: https://reviews.llvm.org/D140332

show more ...

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.

On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).

show more ...

[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional

[Frontend] when attaching a preamble, don't generate the long predefines buffer.

We know we're going to overwrite it anyway.
It'd be a bit of work to coordinate not generating it at all, but setting

[Frontend] when attaching a preamble, don't generate the long predefines buffer.

We know we're going to overwrite it anyway.
It'd be a bit of work to coordinate not generating it at all, but setting this
flag avoids generating ~10k of the 13k string.

Differential Revision: https://reviews.llvm.org/D125180

show more ...

Frontend: Delete output streams before closing CompilerInstance outputs

Delete the output streams coming from
CompilerInstance::createOutputFile() and friends once writes are
finished. Concretely, r

Frontend: Delete output streams before closing CompilerInstance outputs

Delete the output streams coming from
CompilerInstance::createOutputFile() and friends once writes are
finished. Concretely, replacing `OS->flush()` with `OS.reset()` in:

- `ExtractAPIAction::EndSourceFileAction()`
- `PrecompiledPreambleAction::setEmittedPreamblePCH()`
- `cc1_main()'s support for `-ftime-trace`

This fixes theoretical bugs related to proxy streams, which may have
cleanups to run in their destructor. For example, a proxy that
CompilerInstance sometimes uses is `buffer_ostream`, which wraps a
`raw_ostream` lacking pwrite support and adds it. `flush()` does not
promise that output is complete; `buffer_ostream` needs to wait until
the destructor to forward anything so that it can service later calls to
`pwrite()`. If the destructor isn't called then the proxied stream
hasn't received any content.

This also protects against some logic bugs, triggering a null
dereference on a later attempt to write to the stream.

No tests, since in practice these particular code paths never use
use `buffer_ostream`; you need to be writing a binary file to a
pipe (such as stdout) to hit it, but `-extract-api` writes a text file
and the other two use computed filenames that will never (in practice)
be a pipe. This is effectively NFC, for now.

But I have some other patches in the works that add guard rails,
crashing if the stream hasn't been destructed by the time the
CompilerInstance is told to keep the output file, since in most cases
this is a problem.

Differential Revision: https://reviews.llvm.org/D124635

show more ...

[Frontend] shrink in-memory PCH buffers to fit

After building a PCH, the vector capacity is on average ~1/3 unused.
If we're going to keep it in memory for a while, reallocate to the right size.
Tak

[Frontend] shrink in-memory PCH buffers to fit

After building a PCH, the vector capacity is on average ~1/3 unused.
If we're going to keep it in memory for a while, reallocate to the right size.
Take care to do this once clang is destroyed so that we can reuse its
memory rather than requesting more.

Differential Revision: https://reviews.llvm.org/D124242

show more ...

Reland [Frontend] avoid copy of PCH data when PrecompiledPreamble stores it in memory

This reverts commit eadf35270727ca743c11b07040bbfedd415ab6dc.

The reland fixes a couple of places in clang that

Reland [Frontend] avoid copy of PCH data when PrecompiledPreamble stores it in memory

This reverts commit eadf35270727ca743c11b07040bbfedd415ab6dc.

The reland fixes a couple of places in clang that were unneccesarily
requesting a null-terminated buffer of the PCH, and hitting assertions.

show more ...

Revert "[Frontend] avoid copy of PCH data when PrecompiledPreamble stores it in memory"

This reverts commit 6e22dac2e2955db1310c63aec215fc22d8da258e.

Seems to cause bot failures e.g.
https://lab.ll

Revert "[Frontend] avoid copy of PCH data when PrecompiledPreamble stores it in memory"

This reverts commit 6e22dac2e2955db1310c63aec215fc22d8da258e.

Seems to cause bot failures e.g.
https://lab.llvm.org/buildbot/#/builders/109/builds/37071

show more ...

[Frontend] avoid copy of PCH data when PrecompiledPreamble stores it in memory

Instead of unconditionally copying the PCHBuffer into an ostream which can be
backed either by a string or a file, just

[Frontend] avoid copy of PCH data when PrecompiledPreamble stores it in memory

Instead of unconditionally copying the PCHBuffer into an ostream which can be
backed either by a string or a file, just make the PCHBuffer itself the
in-memory storage.

Differential Revision: https://reviews.llvm.org/D124180

show more ...

[Frontend] Simplify PrecompiledPreamble::PCHStorage. NFC

- Remove fiddly union, preambles are heavyweight
- Remove fiddly move constructors in TempPCHFile and PCHStorage, use unique_ptr
- Remove unn

[Frontend] Simplify PrecompiledPreamble::PCHStorage. NFC

- Remove fiddly union, preambles are heavyweight
- Remove fiddly move constructors in TempPCHFile and PCHStorage, use unique_ptr
- Remove unneccesary accessors on PCHStorage
- Remove trivial InMemoryStorage
- Move implementation details into cpp file

This is a prefactoring, followup change will change the in-memory PCHStorage to
avoid extra string copies while creating it.

Differential Revision: https://reviews.llvm.org/D124177

show more ...

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<FileEntryRef>`.

With the API change in place, this patch then removes some uses of the deprecated `FileEntry::getName()` (e.g. in `DependencyGraph.cpp` and `ModuleDependencyCollector.cpp`).

Reviewed By: dexonsmith, bnbarham

Differential Revision: https://reviews.llvm.org/D123574

show more ...

[clang] Change ordering of PreableCallbacks to make sure PP can be referenced in them

Currently, BeforeExecute is called before BeginSourceFile which does not allow
using PP in the callbacks. Change

[clang] Change ordering of PreableCallbacks to make sure PP can be referenced in them

Currently, BeforeExecute is called before BeginSourceFile which does not allow
using PP in the callbacks. Change the ordering to ensure it is possible.
This is a prerequisite for D114370.

Originated from a discussion with @kadircet.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D114525

show more ...