[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are doc

[clang] Make deprecations of some `FileManager` APIs formal (#110014)

Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.

show more ...

[clang] Cleanup IncludeLocMap (#106241)

CompilerInstance can re-use same SourceManager across multiple
frontendactions. During this process it calls
`SourceManager::clearIDTables` to reset any cac

[clang] Cleanup IncludeLocMap (#106241)

CompilerInstance can re-use same SourceManager across multiple
frontendactions. During this process it calls
`SourceManager::clearIDTables` to reset any caches based on FileIDs.

It didn't reset IncludeLocMap, resulting in wrong include locations for
workflows that triggered multiple frontend-actions through same
CompilerInstance.

show more ...

[clang][NFC] Annotate `SourceManagerTest.cpp` with `preferred_type`

This helps debuggers to display values in bit-fields in a more helpful way.

[clang] Fix tests build after 537344fc

[clang][modules] Remove preloaded SLocEntries from PCM files (#66962)

This commit removes the list of SLocEntry offsets to preload eagerly
from PCM files. Commit introducing this functionality (258

[clang][modules] Remove preloaded SLocEntries from PCM files (#66962)

This commit removes the list of SLocEntry offsets to preload eagerly
from PCM files. Commit introducing this functionality (258ae54a) doesn't
clarify why this would be more performant than the lazy approach used
regularly.

Currently, the only SLocEntry the reader is supposed to preload is the
predefines buffer, but in my experience, it's not actually referenced in
most modules, so the time spent deserializing its SLocEntry is wasted.
This is especially noticeable in the dependency scanner, where this
change brings 4.56% speedup on my benchmark.

show more ...

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to av

[Lex] Introduce Preprocessor::LexTokensUntilEOF()

This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to avoid quite some code duplication.

Differential Revision: https://reviews.llvm.org/D158413

show more ...

[clang] NFCI: Use `FileEntryRef` in `SourceManager::overrideFileContents()`

Fix SourceManager::isBeforeInTranslationUnit bug with token-pasting

isBeforeInTranslationUnit compares SourceLocations across FileIDs by
mapping them onto a common ancestor file, following include/e

Fix SourceManager::isBeforeInTranslationUnit bug with token-pasting

isBeforeInTranslationUnit compares SourceLocations across FileIDs by
mapping them onto a common ancestor file, following include/expansion edges.

It is possible to get a tie in the common ancestor, because multiple
"chunks" of a macro arg will expand to the same macro param token in the body:
#define ID(X) X
#define TWO 2
ID(1 TWO)
Here two FileIDs both expand into `X` in ID's expansion:
- one containing `1` and spelled on line 3
- one containing `2` and spelled by the macro expansion of TWO
isBeforeInTranslationUnit breaks this tie by comparing the two FileIDs:
the one "on the left" is always created first and is numerically smaller.
This seems correct so far.

Prior to this patch it also takes a shortcut (unclear if intentionally).
Instead of comparing the two FileIDs that directly expand to the same location,
it compares the original FileIDs being compared. These may not be the
same if there are multiple macro expansions in between.
This *almost* always yields the right answer, because macro expansion
yields "trees" of FileIDs allocated in a contiguous range: when comparing tree A
to tree B, it doesn't matter what representative you pick.

However, the splitting of >> tokens is modeled as macro expansion (as if
the first '>' was a macro that expands to a '>' spelled a scratch buffer).
This splitting occurs retroactively when parsing, so the FileID allocated is
larger than expected if it were a real macro expansion performed during lexing.
As a result, macro tree A can be on the left of tree B, and yet contain
a token-split FileID whose numeric value is *greator* than those in B.
In this case the tiebreak gives the wrong answer.

Concretely:
#define ID(X) X
template <typename> class S{};
ID(
ID(S<S<int>> x);
int y;
)

Given Greater = (typeloc of S<int>).getEndLoc();
Y = (decl of y).getLocation();
isBeforeInTranslationUnit(Greater, Y) should return true, but returns false.

Here the common FileID of (Greater, Y) is the body of the outer ID
expansion, and they both expand to X within it.
With the current tiebreak rules, we compare the FileID of Greater (a split)
to the FileID of Y (a macro arg expansion into X of the outer ID).
The former is larger because the token split occurred relatively late.

This patch fixes the issue by removing the shortcut. It tracks the immediate
FileIDs used to reach the common file, and uses these IDs to break ties.
In the example, we now compare the macro arg expansion of the inner ID()
to the macro arg expansion of Y, and find that it is smaller.

This requires some changes to the InBeforeInTUCacheEntry (sic).
We store a little more data so it's probably slightly slower.
It was difficult to resist more invasive changes:
- performance: the sizing is very suspicious, and once the cache "fills up"
we're thrashing a single entry
- API: the class seems to be needlessly complicated
However I tried to avoid mixing these with subtle behavior changes, and
will send a followup instead.

Differential Revision: https://reviews.llvm.org/D134685

show more ...

Allow interfaces to operate on in-memory buffers with no source location info.

This patch is a part of the upstreaming efforts. Cling has the ability to spawn
child interpreters (mainly for auto com

Allow interfaces to operate on in-memory buffers with no source location info.

This patch is a part of the upstreaming efforts. Cling has the ability to spawn
child interpreters (mainly for auto completions). The child interpreter import
Decls using the ASTImporter which casuses the assertion here
https://github.com/llvm/llvm-project/blob/65eb74e94b414fcde6bfa810d1c30c7fcb136b77/clang/include/clang/Basic/SourceLocation.h#L322

The patch is co-developed with V. Vassilev.

Differential revision: https://reviews.llvm.org/D88780

show more ...

SourceManager: Change SourceManager::isMainFile to take a FileEntry, NFC

`SourceManager::isMainFile` does not use the filename, so it doesn't
need the full `FileEntryRef`; in fact, it's misleading t

SourceManager: Change SourceManager::isMainFile to take a FileEntry, NFC

`SourceManager::isMainFile` does not use the filename, so it doesn't
need the full `FileEntryRef`; in fact, it's misleading to take the name
because that makes it look relevant. Simplify the API, and in the
process remove some calls to `FileEntryRef::FileEntryRef` in the unit
tests (which were blocking making that private to `SourceManager`).

Differential Revision: https://reviews.llvm.org/D89507

show more ...

[clang][Preprocessor] Replace the slow translateFile call by a new, faster isMainFile check

The commit 3c28a2dc6bdc331e5a0d8097a5fa59d06682b9d0 introduced the check that checks if we're
trying to re

[clang][Preprocessor] Replace the slow translateFile call by a new, faster isMainFile check

The commit 3c28a2dc6bdc331e5a0d8097a5fa59d06682b9d0 introduced the check that checks if we're
trying to re-enter a main file when building a preamble. Unfortunately this slowed down the preamble
compilation by 80-90% in some test cases, as translateFile is really slow. This change checks
to see if the FileEntry is the main file without calling translateFile, but by using the new
isMainFile check instead. This speeds up preamble building by 1.5-2x for certain test cases that we have.

rdar://59361291

Differential Revision: https://reviews.llvm.org/D79834

show more ...

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include. Unfortunately it fails to do that for
predefined buffer since it doesn't have a valid insertion location.

As a result of that any file ID created for an include inside the
predefined buffers breaks the traversal logic in
SourceManager::computeMacroArgsCache.

To fix this issue we first record number of created FIDs for predefined
buffer, and then skip them explicitly in source manager.

Another solution would be to just give predefined buffers a valid source
location, but it is unclear where that should be..

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D78649

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[clang] Fix out-of-bounds memory access in ComputeLineNumbers

Differential Revision: https://reviews.llvm.org/D72409

[clang-format] Remove duplciate code from Invalid BOM detection

Summary:
Review comments on {D68767} asked that this duplicated code in clang-format was moved to one central location that being Sour

[clang-format] Remove duplciate code from Invalid BOM detection

Summary:
Review comments on {D68767} asked that this duplicated code in clang-format was moved to one central location that being SourceManager (where it had originally be copied from I assume)

Moved function into static function ContentCache::getInvalidBOM(...) - (closest class to where it was defined before)
Updated clang-format to call this static function

Added unit tests for said new function in BasicTests

Sorry not my normal code area so may have the wrong reviewers. (but your names were on the recent history)

Reviewers: bruno, arphaman, klimek, owenpan, mitchell-stellar, dexonsmith

Reviewed By: owenpan

Subscribers: cfe-commits

Tags: #clang, #clang-format, #clang-tools-extra

Differential Revision: https://reviews.llvm.org/D68914

show more ...

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
o

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942

show more ...

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module b

Modules: Rename MemoryBufferCache to InMemoryModuleCache

Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.

Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.

llvm-svn: 355777

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Add dump() method for SourceRange

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D50662

llvm-svn: 341140

Add #pragma clang module build/endbuild pragmas for performing a module build
as part of a compilation.

This is intended for two purposes:

1) Writing self-contained test cases for modules: we can n

Add #pragma clang module build/endbuild pragmas for performing a module build
as part of a compilation.

This is intended for two purposes:

1) Writing self-contained test cases for modules: we can now write a single
source file test that builds some number of module files on the side and
imports them.

2) Debugging / test case reduction. A single-source testcase is much more
amenable to reduction, compared to a VFS tarball or .pcm files.

llvm-svn: 305101

show more ...

Revert "Revert "PPCallbacks::MacroUndefined, change signature and add test.""

This reverts commit r301469. It isn't needed with r301470, which fixes
the API break introduced in the original commit.

Revert "Revert "PPCallbacks::MacroUndefined, change signature and add test.""

This reverts commit r301469. It isn't needed with r301470, which fixes
the API break introduced in the original commit.

llvm-svn: 301472

show more ...

Revert "PPCallbacks::MacroUndefined, change signature and add test."

This reverts commit r301449. It breaks the build with:

MacroPPCallbacks.h:114:50: error: non-virtual member function marked 'o

Revert "PPCallbacks::MacroUndefined, change signature and add test."

This reverts commit r301449. It breaks the build with:

MacroPPCallbacks.h:114:50: error: non-virtual member function marked 'override' hides virtual member function

llvm-svn: 301469

show more ...

PPCallbacks::MacroUndefined, change signature and add test.

Summary:
The PPCallbacks::MacroUndefined callback is currently insufficient for clients that need to track the MacroDirectives.
This patch

PPCallbacks::MacroUndefined, change signature and add test.

Summary:
The PPCallbacks::MacroUndefined callback is currently insufficient for clients that need to track the MacroDirectives.
This patch adds an additional argument to PPCallbacks::MacroUndefined that is the undef MacroDirective.

Reviewers: bruno, manmanren

Reviewed By: bruno

Subscribers: nemanjai, cfe-commits

Differential Revision: https://reviews.llvm.org/D29923

llvm-svn: 301449

show more ...

Reapply "Modules: Cache PCMs in memory and avoid a use-after-free"

This reverts commit r298185, effectively reapplying r298165, after fixing the
new unit tests (PR32338). The memory buffer generato

Reapply "Modules: Cache PCMs in memory and avoid a use-after-free"

This reverts commit r298185, effectively reapplying r298165, after fixing the
new unit tests (PR32338). The memory buffer generator doesn't null-terminate
the MemoryBuffer it creates; this version of the commit informs getMemBuffer
about that to avoid the assert.

Original commit message follows:

----

Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands). Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly. Furthermore, the LockFileManager is not
robust in every environment. Other -cc1 commands can stall until
timeout (after about eight minutes).

This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack. The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module. Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout. Perhaps we should
reconsider blocking at all.

This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.

The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename. Its ownership is shared by the
CompilerInstance and ModuleManager.

- The ModuleManager stores PCMs there that it loads from disk, never
touching the disk if the cache is hot.

- When modules fail to validate, they're removed from the cache.

- When a CompilerInstance is spawned to build a new module, each
already-loaded PCM is assumed to be valid, and is frozen to avoid
the use-after-free.

- Any newly-built module is written directly to the cache to avoid the
round-trip to the filesystem, making lock files unnecessary for
correctness.

Original patch by Manman Ren; most testcases by Adrian Prantl!

llvm-svn: 298278

show more ...

Revert "Modules: Cache PCMs in memory and avoid a use-after-free"

This reverts commit r298165, as it broke the ARM builds.

llvm-svn: 298185