[clang] Fix a warning

This patch fixes:

clang/lib/Basic/SourceManager.cpp:2131:72: warning: suggest
parentheses around ‘&&’ within ‘||’ [-Wparentheses]

[SourceManager] Fix the incorrect counting stats in getFileIDLoaded.

We were double-counting the number of binary search FileID scans.

[SourceManager] Improve getFileIDLocal.

Prune the search space -- If we know offset(LastFileIDLookup) < SearchOffset, we
can prune the initial binary-search range from [0, end) to [LastFileIDlookup,

[SourceManager] Improve getFileIDLocal.

Prune the search space -- If we know offset(LastFileIDLookup) < SearchOffset, we
can prune the initial binary-search range from [0, end) to [LastFileIDlookup, end).

It reduces the binary search scan by ~30%.

SemaExpr.cpp: 1393437 -> 1035426
FindTarget.cpp: 1275930 -> 920087

Linux kernel:
getFileIDLocal: 2.45% -> 2.15%

Differential Revision: https://reviews.llvm.org/D135132

show more ...

[SourceManager] Improve getFileIDLoaded.

Similar to getFileIDLocal patch, but for the version for load module.

Test with clangd (building AST with preamble), FileID scans in binary
search is reduce

[SourceManager] Improve getFileIDLoaded.

Similar to getFileIDLocal patch, but for the version for load module.

Test with clangd (building AST with preamble), FileID scans in binary
search is reduced:

SemaExpr.cpp: 142K -> 137K (-3%)
FindTarget.cpp: 368K -> 343K (-6%)

Differential Revision: https://reviews.llvm.org/D135258

show more ...

[clang] Update ModuleMap::getModuleMapFile* to use FileEntryRef

Update SourceManager::ContentCache::OrigEntry to keep the original
FileEntryRef, and use that to enable ModuleMap::getModuleMapFile* t

[clang] Update ModuleMap::getModuleMapFile* to use FileEntryRef

Update SourceManager::ContentCache::OrigEntry to keep the original
FileEntryRef, and use that to enable ModuleMap::getModuleMapFile* to
return the original FileEntryRef. This change should be NFC for
most users of SourceManager::ContentCache, but it could affect behaviour
for users of getNameAsRequested such as in compileModuleImpl. I have not
found a way to detect that difference without additional functional
changes, other than incidental cases like changes from / to \ on
Windows so there is no new test.

Differential Revision: https://reviews.llvm.org/D135220

show more ...

Fix SourceManager::isBeforeInTranslationUnit bug with token-pasting

isBeforeInTranslationUnit compares SourceLocations across FileIDs by
mapping them onto a common ancestor file, following include/e

Fix SourceManager::isBeforeInTranslationUnit bug with token-pasting

isBeforeInTranslationUnit compares SourceLocations across FileIDs by
mapping them onto a common ancestor file, following include/expansion edges.

It is possible to get a tie in the common ancestor, because multiple
"chunks" of a macro arg will expand to the same macro param token in the body:
#define ID(X) X
#define TWO 2
ID(1 TWO)
Here two FileIDs both expand into `X` in ID's expansion:
- one containing `1` and spelled on line 3
- one containing `2` and spelled by the macro expansion of TWO
isBeforeInTranslationUnit breaks this tie by comparing the two FileIDs:
the one "on the left" is always created first and is numerically smaller.
This seems correct so far.

Prior to this patch it also takes a shortcut (unclear if intentionally).
Instead of comparing the two FileIDs that directly expand to the same location,
it compares the original FileIDs being compared. These may not be the
same if there are multiple macro expansions in between.
This *almost* always yields the right answer, because macro expansion
yields "trees" of FileIDs allocated in a contiguous range: when comparing tree A
to tree B, it doesn't matter what representative you pick.

However, the splitting of >> tokens is modeled as macro expansion (as if
the first '>' was a macro that expands to a '>' spelled a scratch buffer).
This splitting occurs retroactively when parsing, so the FileID allocated is
larger than expected if it were a real macro expansion performed during lexing.
As a result, macro tree A can be on the left of tree B, and yet contain
a token-split FileID whose numeric value is *greator* than those in B.
In this case the tiebreak gives the wrong answer.

Concretely:
#define ID(X) X
template <typename> class S{};
ID(
ID(S<S<int>> x);
int y;
)

Given Greater = (typeloc of S<int>).getEndLoc();
Y = (decl of y).getLocation();
isBeforeInTranslationUnit(Greater, Y) should return true, but returns false.

Here the common FileID of (Greater, Y) is the body of the outer ID
expansion, and they both expand to X within it.
With the current tiebreak rules, we compare the FileID of Greater (a split)
to the FileID of Y (a macro arg expansion into X of the outer ID).
The former is larger because the token split occurred relatively late.

This patch fixes the issue by removing the shortcut. It tracks the immediate
FileIDs used to reach the common file, and uses these IDs to break ties.
In the example, we now compare the macro arg expansion of the inner ID()
to the macro arg expansion of Y, and find that it is smaller.

This requires some changes to the InBeforeInTUCacheEntry (sic).
We store a little more data so it's probably slightly slower.
It was difficult to resist more invasive changes:
- performance: the sizing is very suspicious, and once the cache "fills up"
we're thrashing a single entry
- API: the class seems to be needlessly complicated
However I tried to avoid mixing these with subtle behavior changes, and
will send a followup instead.

Differential Revision: https://reviews.llvm.org/D134685

show more ...

Avoid else-if after return, NFC

Signed-off-by: Jun Zhang <jun@junz.org>

[clang] SourceManager: fix at SourceManager::getFileIDLoaded for the case of invalid SLockEntry

There is a fix for the search procedure at `SourceManager::getFileIDLoaded`. It might return an invali

[clang] SourceManager: fix at SourceManager::getFileIDLoaded for the case of invalid SLockEntry

There is a fix for the search procedure at `SourceManager::getFileIDLoaded`. It might return an invalid (not loaded) entry. That might cause clang/clangd crashes. At my case the scenario was the following:
- `SourceManager::getFileIDLoaded` returned an invalid file id for a non loaded entry and incorrectly set `SourceManager::LastFileIDLookup` to the value
- `getSLocEntry` returned `SourceManager::FakeSLocEntryForRecovery` introduced at [D89748](https://reviews.llvm.org/D89748).
- The result entry is not tested at `SourceManager::isOffsetInFileID`and as result the call `return SLocOffset < getSLocEntryByID(FID.ID+1).getOffset();` returned `true` value because `FID.ID+1` pointed to a non-fake entry
- The tested offset was marked as one that belonged to the fake `SLockEntry`

Such behaviour might cause a weird clangd crash when preamble contains some header files that were removed just after the preamble created. Unfortunately it's not so easy to reproduce the crash in the form of a LIT test thus I provided the fix only.

Test Plan:
```
ninja check-clang
```

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D130847

show more ...

[Basic] SourceManager docs: macro expansion SLocs aren't a single token. NFC

And haven't been since 2011: https://github.com/llvm/llvm-project/commit/eeca36fe9ad767380b2eab76a6fe5ba410a47393

[clang][NFC] Inclusive terms: replace some uses of sanity in clang

Rewording of comments to avoid using `sanity test, sanity check`.

Reviewed By: aaron.ballman, Quuxplusone

Differential Revision:

[clang][NFC] Inclusive terms: replace some uses of sanity in clang

Rewording of comments to avoid using `sanity test, sanity check`.

Reviewed By: aaron.ballman, Quuxplusone

Differential Revision: https://reviews.llvm.org/D114025

show more ...

[clang] GCC directive extension extension: Hash NNN lines

Some time back I extended GCC's '# NNN' line marker semantics.
Specifically popping to a blank filename will restore the filename to
that of

[clang] GCC directive extension extension: Hash NNN lines

Some time back I extended GCC's '# NNN' line marker semantics.
Specifically popping to a blank filename will restore the filename to
that of the popped-to include. Restore to line 5 of including file
(escaped BOL #'s to avoid git eliding them):

\# 5 "" 2

Added documentation for this line control extension.

This was useful in developing modules tests, but turned out to also be
useful with machine-generated source code. Specifically, a generated
include file that itself includes fragments from elsewhere. The
ability to pop to the generated include file -- with its full path
prefix -- is useful for diagnostic & debug purposes. For instance
something like:

// Machine generated -- DO NOT EDIT
Type Var = {
\# 7 "encoded.dsl" 1 // push to snippet-container
{snippet, of, code}
\# 6 " 2 // Restore to machined-generated source
,
};

// user-code
...
\#include "dsl.h"
...

That pop to "" will restore the filename to '..includepath../dsl.h',
which is better than restoring to plain "dsl.h".

Differential Revision: https://reviews.llvm.org/D113425

show more ...

[clang] Introduce SourceLocation::[U]IntTy typedefs.

This is part of a patch series working towards the ability to make
SourceLocation into a 64-bit type to handle larger translation units.

NFC: th

[clang] Introduce SourceLocation::[U]IntTy typedefs.

This is part of a patch series working towards the ability to make
SourceLocation into a 64-bit type to handle larger translation units.

NFC: this patch introduces typedefs for the integer type used by
SourceLocation and makes all the boring changes to use the typedefs
everywhere, but for the moment, they are unconditionally defined to
uint32_t.

Patch originally by Mikhail Maltsev.

Reviewed By: tmatheson

Differential Revision: https://reviews.llvm.org/D105492

show more ...

[clang][SourceManager] Fix -Wparentheses warning (NFC)

GCC warning:
```
/llvm-project/clang/lib/Basic/SourceManager.cpp: In instantiation of ‘constexpr T likelyhasbetween(T, unsigned char, unsigned

[clang][SourceManager] Fix -Wparentheses warning (NFC)

GCC warning:
```
/llvm-project/clang/lib/Basic/SourceManager.cpp: In instantiation of ‘constexpr T likelyhasbetween(T, unsigned char, unsigned char) [with T = long unsigned int]’:
/llvm-project/clang/lib/Basic/SourceManager.cpp:1292:52: required from here
/llvm-project/clang/lib/Basic/SourceManager.cpp:1264:48: warning: suggest parentheses around ‘+’ in operand of ‘&’ [-Wparentheses]
1264 | (x & ~static_cast<T>(0) / 255 * 127) +
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
1265 | (~static_cast<T>(0) / 255 * (127 - (m - 1)))) &
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

show more ...

[NFC] Fix warning introduced in 20105b6b4874a85813f7a4a3d8ad2a0f023dda14

[clang] Speedup line offset mapping computation

Clang spends a decent amount of time in the LineOffsetMapping::get(...)
function. This function used to be vectorized (through SSE2) then the
optimiza

[clang] Speedup line offset mapping computation

Clang spends a decent amount of time in the LineOffsetMapping::get(...)
function. This function used to be vectorized (through SSE2) then the
optimization got dropped because the sequential version was on-par performance
wise.

This provides an optimization of the sequential version that works on a word at
a time, using (documented) bithacks to provide a portable vectorization.

When preprocessing the sqlite amalgamation, this yields a sweet 3% speedup.

This is a recommit of 6951b72334bbe4c189c71751edc1e361d7b5632c with endianness
and unsigned long vs uint64_t issues fixed (hopefully).

Differential Revision: https://reviews.llvm.org/D99409

show more ...

Revert "[clang] Speedup line offset mapping computation"

This reverts commit 6951b72334bbe4c189c71751edc1e361d7b5632c.
Breaks several bots, see comments on https://reviews.llvm.org/D99409

[clang] Speedup line offset mapping computation

Clang spends a decent amount of time in the LineOffsetMapping::get(...)
function. This function used to be vectorized (through SSE2) then the
optimiza

[clang] Speedup line offset mapping computation

Clang spends a decent amount of time in the LineOffsetMapping::get(...)
function. This function used to be vectorized (through SSE2) then the
optimization got dropped because the sequential version was on-par performance
wise.

This provides an optimization of the sequential version that works on a word at
a time, using (documented) bithacks to provide a portable vectorization.

When preprocessing the sqlite amalgamation, this yields a sweet 3% speedup.

Differential Revision: https://reviews.llvm.org/D99409

show more ...

Use a fast path when initializing LineOffsetMapping

Use the fact that the number of line break is lower than printable characters to
guide the optimization process. Also use a fuzzy test that catche

Use a fast path when initializing LineOffsetMapping

Use the fact that the number of line break is lower than printable characters to
guide the optimization process. Also use a fuzzy test that catches both \n and
\r in a single check to speedup the computation.

Differential Revision: https://reviews.llvm.org/D97320

show more ...

SourceManager: Migrate to FileEntryRef in getOrCreateContentCache, NFC

Change `SourceManager::getOrCreateContentCache` to take a `FileEntryRef`
and update call sites (mostly internal to SourceManage

SourceManager: Migrate to FileEntryRef in getOrCreateContentCache, NFC

Change `SourceManager::getOrCreateContentCache` to take a `FileEntryRef`
and update call sites (mostly internal to SourceManager.cpp). In a
couple of cases this temporarily relies on `FileEntry::getLastRef`, but
those can be cleaned up once other APIs switch over.

The one change outside of SourceManager.cpp is in ASTReader.cpp, which
stops relying on the auto-degrade-to-`FileEntry*` behaviour from
`InputFile::getFile` since it now needs a `FileEntryRef`.

No functionality change here.

Differential Revision: https://reviews.llvm.org/D92983

show more ...

SourceManager: Unify FileEntry/FileEntryRef versions of createFileID

Change `SourceManager::createFileID(const FileEntry*)` to defer to
`SourceManager::createFileID(FileEntryRef)`. This fixes an une

SourceManager: Unify FileEntry/FileEntryRef versions of createFileID

Change `SourceManager::createFileID(const FileEntry*)` to defer to
`SourceManager::createFileID(FileEntryRef)`. This fixes an unexercised
bug where the latter gained support for named pipes and the former
didn't, but since we're trying to remove all calls to the former it
doesn't really make sense to test this explicitly now that the
implementation is hollowed out.

This is a belated follow-up to 245218bb355599771ba43a0fe1449d1670f2666c,
which sunk named pipe support into FileManager and SourceManager. The
original version of that patch was based on top of
https://reviews.llvm.org/D92984, which removed the `FileEntry` overload
of `createFileID()`, and I missed the subtle difference when it was
rebased.

show more ...

Basic: Add native support for stdin to SourceManager and FileManager

Add support for stdin to SourceManager and FileManager. Adds
FileManager::getSTDIN, which adds a FileEntryRef for `<stdin>` and r

Basic: Add native support for stdin to SourceManager and FileManager

Add support for stdin to SourceManager and FileManager. Adds
FileManager::getSTDIN, which adds a FileEntryRef for `<stdin>` and reads
the MemoryBuffer, which is stored as `FileEntry::Content`.

Eventually the other buffers in `ContentCache` will sink to here as well
-- we probably usually want to load/save a MemoryBuffer eagerly -- but
it's happening early for stdin to get rid of
CompilerInstance::InitializeSourceManager's final call to
`SourceManager::overrideFileContents`.

clang/test/CXX/modules-ts/dcl.dcl/dcl.module/dcl.module.export/p1.cpp
relies on building a module from stdin; supporting that requires setting
ContentCache::BufferOverridden.

Differential Revision: https://reviews.llvm.org/D93148

show more ...

Basic: Support named pipes natively in SourceManager and FileManager

Handle named pipes natively in SourceManager and FileManager, removing a
call to `SourceManager::overrideFileContents` in
`Compil

Basic: Support named pipes natively in SourceManager and FileManager

Handle named pipes natively in SourceManager and FileManager, removing a
call to `SourceManager::overrideFileContents` in
`CompilerInstance::InitializeSourceManager` (removing a blocker for
sinking the content cache to FileManager (which will incidently sink
this new named pipe logic with it)).

SourceManager usually checks if the file entry's size matches the
eventually loaded buffer, but that's now skipped for named pipes since
the `stat` won't reflect the full size. Since we can't trust
`ContentsEntry->getSize()`, we also need shift the check for files that
are too large until after the buffer is loaded... and load the buffer
immediately in `createFileID` so that no client gets a bad value from
`ContentCache::getSize`. `FileManager::getBufferForFile` also needs to
treat these files as volatile when loading the buffer.

Native support in SourceManager / FileManager means that named pipes can
also be `#include`d, and clang/test/Misc/dev-fd-fs.c was expanded to
check for that.

This is a new version of 3b18a594c7717a328c33b9c1eba675e9f4bd367c, which
was reverted in b34632201987eed369bb7ef4646f341b901c95b8 since it was
missing the `SourceManager` changes.

Differential Revision: https://reviews.llvm.org/D92531

show more ...

Serialization: Change InputFile to use FileEntryRef and add getVirtualFileRef, NFC

Change the `InputFile` class to store `Optional<FileEntryRef>` instead
of `FileEntry*`. This paged in a few API cha

Serialization: Change InputFile to use FileEntryRef and add getVirtualFileRef, NFC

Change the `InputFile` class to store `Optional<FileEntryRef>` instead
of `FileEntry*`. This paged in a few API changes:

- Added `FileManager::getVirtualFileRef`, and converted `getVirtualFile`
to a wrapper of it.
- Updated `SourceManager::bypassFileContentsOverride` to take
`FileEntryRef` and return `Optional<FileEntryRef>`
(`ASTReader::getInputFile` is the only caller).

Differential Revision: https://reviews.llvm.org/D90053

show more ...

FileManager: Shrink FileEntryRef to the size of a pointer

Shrink `FileEntryRef` to the size of a pointer, by having it directly
reference the `StringMapEntry` the same way that `DirectoryEntryRef`
d

FileManager: Shrink FileEntryRef to the size of a pointer

Shrink `FileEntryRef` to the size of a pointer, by having it directly
reference the `StringMapEntry` the same way that `DirectoryEntryRef`
does. This makes `FileEntryRef::FileEntryRef` private as a side effect
(`FileManager` is a friend!).

There are two helper types added within `FileEntryRef`:

- `FileEntryRef::MapValue` is the type stored in
`FileManager::SeenFileEntries`. It's a replacement for
`SeenFileEntryOrRedirect`, where the second pointer type has been
changed from `StringRef*` to `MapEntry*` (see next bullet).
- `FileEntryRef::MapEntry` is the instantiation of `StringMapEntry<>`
where `MapValue` is stored. This is what `FileEntryRef` has a pointer
to, in order to grab the name in addition to the value.

Differential Revision: https://reviews.llvm.org/D89488

show more ...

SourceManager: Use the same fake SLocEntry whenever it fails to load

Instead of putting a fake `SLocEntry` at `LoadedSLocEntryTable[Index]`
when it fails to load in `SourceManager::loadSLocEntry`, a

SourceManager: Use the same fake SLocEntry whenever it fails to load

Instead of putting a fake `SLocEntry` at `LoadedSLocEntryTable[Index]`
when it fails to load in `SourceManager::loadSLocEntry`, allocate a fake
one. Unless someone is sniffing the address of the returned `SLocEntry`
(doubtful), this won't be a functionality change. Note that
`SLocEntryLoaded[Index]` wasn't being set to `true` either before or
after this change so no accessor is every going to look at
`LoadedSLocEntryTable[Index]`.

As a side effect, drop the `mutable` from `LoadedSLocEntryTable`.

Differential Revision: https://reviews.llvm.org/D89748

show more ...