[SourceManager] don't check invalid param of getLocalSLocEntry()

Forked from D80681.

getLocalSLocEntry() has an unused parameter used to satisfy an interface
of libclang (see getInclusions() in
cla

[SourceManager] don't check invalid param of getLocalSLocEntry()

Forked from D80681.

getLocalSLocEntry() has an unused parameter used to satisfy an interface
of libclang (see getInclusions() in
clang/tools/libclang/CIndexInclusionStack.cpp). It's pointless for
callers to construct/pass/check this inout parameter that can never
signify that a FileID is invalid.

Reviewed By: kadircet

Differential Revision: https://reviews.llvm.org/D82498

show more ...

[Clang][SourceManager] optimize getFileIDLocal()

Summary:
A recent Linux kernel commit exposed a performance cliff in Clang. Calls
to SourceManager::getFileIDLocal() when there's a cache miss agains

[Clang][SourceManager] optimize getFileIDLocal()

Summary:
A recent Linux kernel commit exposed a performance cliff in Clang. Calls
to SourceManager::getFileIDLocal() when there's a cache miss against
LastFileIDLookup can be relatively expensive, as getFileIDLocal() tries
a few linear probes, then falls back to binary search. The use of
SourceManager::isOffsetInFileID() is also relatively expensive (both
isOffsetInFileID and getFileIDLocal dominated a trace of the performance
cliff case).

As a FIXME notes (and as @kadircet helpfully noted in review of D80681),
there's a few optimizations we can do here since we've already
identified that an offset is local (as opposed to "loaded").

This patch was forked off of D80681, which additionally did this and
modified some caching behavior, as we expect this change to be less
controversial.

In terms of optimizations, we've already determined that the SLocOffset
parameter to SourceManager::getFileIDLocal() is local in the caller
SourceManager::getFileIDSlow(). Also, there's an early continue in the
binary search loop in getFileIDLocal() that are duplicated in
isOffsetInFileID() as pointed out by @kadircet.

Take advantage of these to optimize the binary search patch, and remove
the FIXME.

Reviewers: kadircet

Reviewed By: kadircet

Subscribers: cfe-commits, kadircet, srhines

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82497

show more ...

[clang][Preprocessor] Replace the slow translateFile call by a new, faster isMainFile check

The commit 3c28a2dc6bdc331e5a0d8097a5fa59d06682b9d0 introduced the check that checks if we're
trying to re

[clang][Preprocessor] Replace the slow translateFile call by a new, faster isMainFile check

The commit 3c28a2dc6bdc331e5a0d8097a5fa59d06682b9d0 introduced the check that checks if we're
trying to re-enter a main file when building a preamble. Unfortunately this slowed down the preamble
compilation by 80-90% in some test cases, as translateFile is really slow. This change checks
to see if the FileEntry is the main file without calling translateFile, but by using the new
isMainFile check instead. This speeds up preamble building by 1.5-2x for certain test cases that we have.

rdar://59361291

Differential Revision: https://reviews.llvm.org/D79834

show more ...

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include. Unfortunately it fails to do that for
predefined buffer since it doesn't have a valid insertion location.

As a result of that any file ID created for an include inside the
predefined buffers breaks the traversal logic in
SourceManager::computeMacroArgsCache.

To fix this issue we first record number of created FIDs for predefined
buffer, and then skip them explicitly in source manager.

Another solution would be to just give predefined buffers a valid source
location, but it is unclear where that should be..

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D78649

show more ...

Avoid including FileManager.h from SourceManager.h

Most clients of SourceManager.h need to do things like turning source
locations into file & line number pairs, but this doesn't require
bringing in

Avoid including FileManager.h from SourceManager.h

Most clients of SourceManager.h need to do things like turning source
locations into file & line number pairs, but this doesn't require
bringing in FileManager.h and LLVM's FS headers.

The main code change here is to sink SM::createFileID into the cpp file.
I reason that this is not performance critical because it doesn't happen
on the diagnostic path, it happens along the paths of macro expansion
(could be hot) and new includes (less hot).

Saves some includes:
309 - /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileManager.h
272 - /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileSystemOptions.h
271 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/VirtualFileSystem.h
267 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/FileSystem.h
266 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Chrono.h

Differential Revision: https://reviews.llvm.org/D75406

show more ...

Detect source location overflow due includes

Summary:
As discussed in http://lists.llvm.org/pipermail/cfe-dev/2019-October/063459.html
the overflow of the souce locations (limited to 2^31 chars) can

Detect source location overflow due includes

Summary:
As discussed in http://lists.llvm.org/pipermail/cfe-dev/2019-October/063459.html
the overflow of the souce locations (limited to 2^31 chars) can generate all sorts of
weird things (bogus warnings, hangs, crashes, miscompilation and correct compilation).
In debug mode this assert would fail. So it might be a good start, as in PR42301,
to detect the failure and exit with a proper error message.

Reviewers: rsmith, thakis, miyuki

Reviewed By: miyuki

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70183

show more ...

[clang] Fix out-of-bounds memory access in ComputeLineNumbers

Differential Revision: https://reviews.llvm.org/D72409

[clang-format] Remove duplciate code from Invalid BOM detection

Summary:
Review comments on {D68767} asked that this duplicated code in clang-format was moved to one central location that being Sour

[clang-format] Remove duplciate code from Invalid BOM detection

Summary:
Review comments on {D68767} asked that this duplicated code in clang-format was moved to one central location that being SourceManager (where it had originally be copied from I assume)

Moved function into static function ContentCache::getInvalidBOM(...) - (closest class to where it was defined before)
Updated clang-format to call this static function

Added unit tests for said new function in BasicTests

Sorry not my normal code area so may have the wrong reviewers. (but your names were on the recent history)

Reviewers: bruno, arphaman, klimek, owenpan, mitchell-stellar, dexonsmith

Reviewed By: owenpan

Subscribers: cfe-commits

Tags: #clang, #clang-format, #clang-tools-extra

Differential Revision: https://reviews.llvm.org/D68914

show more ...

ASTReader: Bypass overridden files when reading PCHs

If contents of a file that is part of a PCM are overridden when reading
it, but weren't overridden when the PCM was being built, the ASTReader
wi

ASTReader: Bypass overridden files when reading PCHs

If contents of a file that is part of a PCM are overridden when reading
it, but weren't overridden when the PCM was being built, the ASTReader
will emit an error. Now it creates a separate FileEntry for recovery,
bypassing the overridden content instead of discarding it. The
pre-existing testcase clang/test/PCH/remap-file-from-pch.cpp confirms
that the new recovery method works correctly.

This resolves a long-standing FIXME to avoid hypothetically invalidating
another precompiled module that's already using the overridden contents.

This also removes ContentCache-related API that would be unsafe to use
across `CompilerInstance`s in an implicit modules build. This helps to
unblock us sinking it from SourceManager into FileManager in the future,
which would allow us to delete `InMemoryModuleCache`.

https://reviews.llvm.org/D66710

llvm-svn: 370546

show more ...

ContentCache: Drop getBuffer's dependency on SourceManager

Refactor ContentCache::IsSystemFile to IsFileVolatile, checking
SourceManager::userFilesAreVolatile at construction time. This is a
step t

ContentCache: Drop getBuffer's dependency on SourceManager

Refactor ContentCache::IsSystemFile to IsFileVolatile, checking
SourceManager::userFilesAreVolatile at construction time. This is a
step toward lowering ContentCache down from SourceManager to
FileManager.

No functionality change intended.

https://reviews.llvm.org/D66713

llvm-svn: 369958

show more ...

Introduce FileEntryRef and use it when handling includes to report correct dependencies
when the FileManager is reused across invocations

This commit introduces a parallel API to FileManager's getFi

Introduce FileEntryRef and use it when handling includes to report correct dependencies
when the FileManager is reused across invocations

This commit introduces a parallel API to FileManager's getFile: getFileEntryRef, which returns
a reference to the FileEntry, and the name that was used to access the file. In the case of
a VFS with 'use-external-names', the FileEntyRef contains the external name of the file,
not the filename that was used to access it.

The new API is adopted only in the HeaderSearch and Preprocessor for include file lookup, so that the
accessed path can be propagated to SourceManager's FileInfo. SourceManager's FileInfo now can report this accessed path, using
the new getName method. This API is then adopted in the dependency collector, which now correctly reports dependencies when a file
is included both using a symlink and a real path in the case when the FileManager is reused across multiple Preprocessor invocations.

Note that this patch does not fix all dependency collector issues, as the same problem is still present in other cases when dependencies
are obtained using FileSkipped, InclusionDirective, and HasInclude. This will be fixed in follow-up commits.

Differential Revision: https://reviews.llvm.org/D65907

llvm-svn: 369680

show more ...

NFCI: Simplify SourceManager::translateFile by removing code path that should never be taken

I noticed that SourceManager::translateFile has code that doesn't really make sense.
In particular, if it

NFCI: Simplify SourceManager::translateFile by removing code path that should never be taken

I noticed that SourceManager::translateFile has code that doesn't really make sense.
In particular, if it fails to find a FileID by comparing FileEntry * values, it tries to
look through files that have the same filename, to see if they have a matching inode to try to
find the right FileID. However, the inode comparison seem redundant, as Clang's FileManager
already deduplicates FileEntry * values by inode.
Thus the comparisons between inodes should never actually succeed, and the comparison between FileEntry * values should be sufficient here.

Differential Revision: https://reviews.llvm.org/D65481

llvm-svn: 369585

show more ...

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
o

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942

show more ...

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616

show more ...

Use a unique_ptr instead of manual memory management for LineTable

llvm-svn: 366088

Fix MSVC "signed/unsigned mismatch" warning. NFCI.

Fixes PR42426.

llvm-svn: 365019

Change std::{lower,upper}_bound to llvm::{lower,upper}_bound or llvm::partition_point. NFC

llvm-svn: 365006

PR42301: Abort cleanly if we encounter a huge source file rather than
crashing.

Ideally we wouldn't care about the size of a file so long as it fits in
memory, but in practice we have lots of hardoc

PR42301: Abort cleanly if we encounter a huge source file rather than
crashing.

Ideally we wouldn't care about the size of a file so long as it fits in
memory, but in practice we have lots of hardocded assumptions that
unsigned can be used to index files, string literals, and so on.

llvm-svn: 364103

show more ...

[DebugInfo] Don't emit checksums when compiling a preprocessed CPP

Fixes PR41215

Differential Revision: https://reviews.llvm.org/D60283

llvm-svn: 361296

[Preamble] Stop circular inclusion of main file when building preamble

If a header file was processed for the second time, we could end up with a
wrong conditional stack and skipped ranges:

In the

[Preamble] Stop circular inclusion of main file when building preamble

If a header file was processed for the second time, we could end up with a
wrong conditional stack and skipped ranges:

In the particular example, if the header guard is evaluated the second time and
it is decided to skip the conditional block, the corresponding "#endif" is
never seen since the preamble does not include it and we end up in the
Tok.is(tok::eof) case with a wrong conditional stack.

Detect the circular inclusion, emit a diagnostic and stop processing the
inclusion.

llvm-svn: 360418

show more ...

[clang] Fix a bug that reports UTF32 (LE) files as UTF16 (LE) ones

Also fix a typo for the SCSU byte order mark.

Differential Revision: https://reviews.llvm.org/D61628

llvm-svn: 360256

Make SourceManager::createFileID(UnownedTag, ...) take a const llvm::MemoryBuffer*

Requires making the llvm::MemoryBuffer* stored by SourceManager const,
which in turn requires making the accessors

Make SourceManager::createFileID(UnownedTag, ...) take a const llvm::MemoryBuffer*

Requires making the llvm::MemoryBuffer* stored by SourceManager const,
which in turn requires making the accessors for that return const
llvm::MemoryBuffer*s and updating all call sites.

The original motivation for this was to use it and fix the TODO in
CodeGenAction.cpp's ConvertBackendLocation() by using the UnownedTag
version of createFileID, and since llvm::SourceMgr* hands out a const
llvm::MemoryBuffer* this is required. I'm not sure if fixing the TODO
this way actually works, but this seems like a good change on its own
anyways.

No intended behavior change.

Differential Revision: https://reviews.llvm.org/D60247

llvm-svn: 357724

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

ComputeLineNumbers: delete SSE2 vectorization

Summary:
SSE2 vectorization was added in 2012, but it is 2018 now and I can't
observe any performance boost (testing clang -E [all Sema/* CodeGen/* with

ComputeLineNumbers: delete SSE2 vectorization

Summary:
SSE2 vectorization was added in 2012, but it is 2018 now and I can't
observe any performance boost (testing clang -E [all Sema/* CodeGen/* with proper -I options]) with the existing _mm_movemask_epi8+countTrailingZeros or the following SSE4.2 (compiling with -msse4.2):

__m128i C = _mm_setr_epi8('\r','\n',0,0,0,0,0,0,0,0,0,0,0,0,0,0);
_mm_cmpestri(C, 2, Chunk, 16, _SIDD_UBYTE_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_POSITIVE_POLARITY | _SIDD_LEAST_SIGNIFICANT)

Delete the vectorization to simplify the code.

Also simplify the code a bit and don't check the line ending sequence \n\r

Reviewers: bkramer, #clang

Reviewed By: bkramer

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D55484

llvm-svn: 348777

show more ...

SourceManager: insert(make_pair(..)) -> try_emplace. NFC

llvm-svn: 348709