[clang] Migrate away from PointerUnion::{is,get} (NFC) (#119724)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn

[clang] Migrate away from PointerUnion::{is,get} (NFC) (#119724)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[clang] Migrate away from PointerUnion::{is,get} (NFC) (#119654)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn

[clang] Migrate away from PointerUnion::{is,get} (NFC) (#119654)

Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:

// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>

I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.

show more ...

[SystemZ][z/OS] Add new openFileForReadBinary function, and pass IsText parameter to getBufferForFile (#111723)

This patch adds an IsText parameter to the following getBufferForFile,
getBufferForFi

[SystemZ][z/OS] Add new openFileForReadBinary function, and pass IsText parameter to getBufferForFile (#111723)

This patch adds an IsText parameter to the following getBufferForFile,
getBufferForFileImpl. We introduce a new virtual function
openFileForReadBinary which defaults to openFileForRead except in
RealFileSystem which uses the OF_None flag instead of OF_Text.

The default is set to OF_Text instead of OF_None, this change in value
does not affect any other platforms other than z/OS. Setting this
parameter correctly is required to open files on z/OS in the correct
encoding. The IsText parameter is based on the context of where we open
files, for example, in the ASTReader, HeaderMap requires that files
always be opened in binary even though they might be tagged as text.

show more ...

Revert "[SystemZ][z/OS] Propagate IsText parameter to open text files as text (#107906)"

This reverts commit edf3b277a5f2ebe144827ed47463c22743cac5f9.

[SystemZ][z/OS] Propagate IsText parameter to open text files as text (#107906)

This patch adds an IsText parameter to the following functions
openFileForRead, getBufferForFile, getBufferForFileImp

[SystemZ][z/OS] Propagate IsText parameter to open text files as text (#107906)

This patch adds an IsText parameter to the following functions
openFileForRead, getBufferForFile, getBufferForFileImpl and determines
whether a file is text by querying the file tag on z/OS. The default is
set to OF_Text instead of OF_None, this change in value does not affect
any other platforms other than z/OS.

show more ...

[clang][deps] Print tracing VFS data (#108056)

Clang's `-cc1 -print-stats` shows lots of useful internal data including
basic `FileManager` stats. Since this layer caches some results, it is
uncle

[clang][deps] Print tracing VFS data (#108056)

Clang's `-cc1 -print-stats` shows lots of useful internal data including
basic `FileManager` stats. Since this layer caches some results, it is
unclear how that information translates to actual filesystem accesses.
This PR uses `llvm::vfs::TracingFileSystem` to provide that missing
information.

Similar mechanism is implemented for `clang-scan-deps`'s verbose mode
(`-v`). IO contention proved to be a real bottleneck a couple of times
already and this new feature should make those easier to detect in the
future. The tracing VFS is inserted below the caching FS and above the
real FS.

show more ...

Reland [clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (#95802)

This commit implements the entirety of the now-accepted [N3017
-Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg1

Reland [clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (#95802)

This commit implements the entirety of the now-accepted [N3017
-Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm) and
its sister C++ paper [p1967](https://wg21.link/p1967). It implements
everything in the specification, and includes an implementation that
drastically improves the time it takes to embed data in specific
scenarios (the initialization of character type arrays). The mechanisms
used to do this are used under the "as-if" rule, and in general when the
system cannot detect it is initializing an array object in a variable
declaration, will generate EmbedExpr AST node which will be expanded by
AST consumers (CodeGen or constant expression evaluators) or expand
embed directive as a comma expression.

This reverts commit
https://github.com/llvm/llvm-project/commit/682d461d5a231cee54d65910e6341769419a67d7.

---------

Co-authored-by: The Phantom Derpstorm <phdofthehouse@gmail.com>
Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
Co-authored-by: H. Vetinari <h.vetinari@gmx.com>

show more ...

Revert "✨ [Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy)" (#95299)

Reverts llvm/llvm-project#68620

Introduce or expose a memory leak and UB, see llvm/llvm-pro

Revert "✨ [Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy)" (#95299)

Reverts llvm/llvm-project#68620

Introduce or expose a memory leak and UB, see llvm/llvm-project#68620

show more ...

[clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy) (#68620)

This commit implements the entirety of the now-accepted [N3017 -
Preprocessor
Embed](https://www

[clang][Sema, Lex, Parse] Preprocessor embed in C and C++ (and Obj-C and Obj-C++ by-proxy) (#68620)

This commit implements the entirety of the now-accepted [N3017 -
Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm) and
its sister C++ paper [p1967](https://wg21.link/p1967). It implements
everything in the specification, and includes an implementation that
drastically improves the time it takes to embed data in specific
scenarios (the initialization of character type arrays). The mechanisms
used to do this are used under the "as-if" rule, and in general when the
system cannot detect it is initializing an array object in a variable
declaration, will generate EmbedExpr AST node which will be expanded
by AST consumers (CodeGen or constant expression evaluators) or
expand embed directive as a comma expression.

---------

Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
Co-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: Podchishchaeva, Mariya <mariya.podchishchaeva@intel.com>

show more ...

[clang] Processing real directories added as virtual ones (#91645)

The `FileManager` might create a virtual directory that can be used
later as a search path. This is the case when we use remapping

[clang] Processing real directories added as virtual ones (#91645)

The `FileManager` might create a virtual directory that can be used
later as a search path. This is the case when we use remapping, as
demonstrated in the suggested LIT test.

We might encounter a 'false cache miss' and add the same directory
twice into `FileManager::SeenDirEntries` if the added record is a real
directory that is present on a disk:
- Once as a virtual directory
- And once as a real one

This isn't a problem if the added directories have the same name, as in
this case, we will get a cache hit. However, it could lead to
compilation errors if the directory names are different but point to the
same folder. For example, one might use an absolute name and another a
relative one. For instance, the **implicit-module-remap.cpp** LIT test
will fail with the following message:
```
/.../implicit-module-remap.cpp.tmp/test.cpp:1:2: fatal error: module 'a'
was built in directory '/.../implicit-module-remap.cpp.tmp' but now
resides in directory '.'
1 | #include "a.h"
| ^
1 error generated.
```

The suggested fix checks if the added virtual directory is present on
the disk and handles it as a real one if that is the case.

show more ...

[clang-scan-deps] Fix contention when updating `TrackingStatistic`s in hot code paths in `FileManager`. (#88427)

`FileManager::getDirectoryRef()` and `FileManager::getFileRef()` are hot code paths i

[clang-scan-deps] Fix contention when updating `TrackingStatistic`s in hot code paths in `FileManager`. (#88427)

`FileManager::getDirectoryRef()` and `FileManager::getFileRef()` are hot code paths in `clang-scan-deps`. These functions are updating on every call a few atomics related to printing statistics, which causes contention on high core count machines.

![Screenshot 2024-04-10
214123](https://github.com/llvm/llvm-project/assets/37383324/5756b1bc-cab5-4612-8769-ee7e03a66479)

![Screenshot 2024-04-10
214246](https://github.com/llvm/llvm-project/assets/37383324/3d560e89-61c7-4fb9-9330-f9e660e8fc8b)

![Screenshot 2024-04-10
214315](https://github.com/llvm/llvm-project/assets/37383324/006341fc-49d4-4720-a348-7af435c21b17)

After this patch we make the variables local to the `FileManager`.

In our test case, this saves about 49 sec over 1 min 47 sec of `clang-scan-deps` run time (1 min 47 sec before, 58 sec after). These figures are after applying my suggestion in https://github.com/llvm/llvm-project/pull/88152#issuecomment-2049803229, that is:
```
static bool shouldCacheStatFailures(StringRef Filename) {
return true;
}
```
Without the above, there's just too much OS noise from the high volume of `status()` calls with regular non-modules C++ code. Tested on Windows with clang-cl.

show more ...

[clang][CodeGen] Shift relink option implementation away from module cloning (#81693)

We recently implemented a new option allowing relinking of bitcode
modules via the "-mllvm -relink-builtin-bitc

[clang][CodeGen] Shift relink option implementation away from module cloning (#81693)

We recently implemented a new option allowing relinking of bitcode
modules via the "-mllvm -relink-builtin-bitcode-postop"
option.

This implementation relied on llvm::CloneModule() in order to pass
copies to modules and preserve the original modules for later relinking.
However, cloning modules has been found to be prohibitively expensive,
significantly increasing compilation time for large bitcode libraries.

In this patch, we shift the relink option implementation to instead link
the original modules initially, and reload modules from the file system
if relinking is requested. This approach results in significantly
reduced overhead.

We accomplish this by creating a new ReloadModules() routine that can be
called from a BackendConsumer class, to mimic the behavior of
ASTConsumer's loadLinkModules(), but without access to the
CompilerInstance.

Because loading the bitcodes from the filesystem requires access to the
FileManager class, we also forward a reference to the CompilerInstance
class to the BackendConsumer. This mirrors what is already done for
several CompilerInstance members, such as TargetOptions and
CodeGenOptions.

Finally, we needed to add a const specifier to the
FileManager::getBufferForFile() routine to allow it to be called using
the const reference returned from CompilerInstance::getFileManager()

show more ...

[clang][DependencyScanner] Remove unused -ivfsoverlay files (#73734)

`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tre

[clang][DependencyScanner] Remove unused -ivfsoverlay files (#73734)

`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.

Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.

show more ...

[clang] NFC: Remove `{File,Directory}Entry::getName()` (#74910)

The files and directories that Clang accesses are uniqued by their
inode. For each inode `FileManager` will create exactly one `FileE

[clang] NFC: Remove `{File,Directory}Entry::getName()` (#74910)

The files and directories that Clang accesses are uniqued by their
inode. For each inode `FileManager` will create exactly one `FileEntry`
or `DirectoryEntry` object, which makes answering the question _"Are
these two files/directories the same?"_ a simple pointer equality check.

However, since the same inode can be accessed through multiple different
paths, asking the `FileEntry` or `DirectoryEntry` object _"What is your
name?"_ doesn't have clear semantics. In c0ff9908 we started reporting
the most recent name used to access the entry, which turned out to be
necessary for Clang modules. However, the long-term solution has always
been to explicitly track the as-requested name. This has been
implemented in 4dc5573a as `FileEntryRef` and `DirectoryEntryRef`.

The `DirectoryEntry::getName()` interface has been deprecated since the
Clang 17 release and `FileEntry::getName()` since Clang 18. We have
replaced uses of these deprecated APIs in `main` with
`DirectoryEntryRef::getName()` and `FileEntryRef::getName()`
respectively.

This makes it possible to remove `{File,Directory}Entry::getName()` for
good along with the `FileManager` code that implements them.

show more ...

[clang] Use SmallString::operator std::string (NFC)

[clang][modules] Adopt `FileEntryRef` in the `HeaderFileInfo` block in PCM files (#67383)

This patch adopts `FileEntryRef` in the `HeaderFileInfo`-writing part of
`ASTWriter`.

First, this patch

[clang][modules] Adopt `FileEntryRef` in the `HeaderFileInfo` block in PCM files (#67383)

This patch adopts `FileEntryRef` in the `HeaderFileInfo`-writing part of
`ASTWriter`.

First, this patch removes the loop over
`FileManager::VirtualFileEntries`. It's redundant, since all virtual
file entries are also present in `SeenFileEntries` and thus already in
`UIDToFiles`.

Second, since we now no longer rely on
`FileEntry::getLastRef()`/`FileEntry::getName()`, this patch takes care
to establish which path gets used for each UID by picking the
`FileEntryRef` with the most "`<`" name (instead of just relying on the
`StringMap` iteration order).

Note that which `FileEntry`/`FileEntryRef` objects we pick for each UID
for serialization into the `llvm::OnDiskChainedHashTable` doesn't really
matter. The hash function only includes the file size and modification
time. The file name only plays role during resolution of hash
collisions, in which case it goes through `FileManager` and resolves to
a `FileEntry` that gets pointer-compared with the queried `FileEntry`.

(Reincarnation of [D143414](https://reviews.llvm.org/D143414) and
[D142780](https://reviews.llvm.org/D142780).)

show more ...

[clang] NFCI: Use `FileEntryRef` in `FileManager::getCanonicalName()`

[clang] NFCI: Use `FileEntryRef` in `FileManager::getBufferForFile()`

[lit][clang] Avoid realpath on Windows due to MAX_PATH limitations

Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths

[lit][clang] Avoid realpath on Windows due to MAX_PATH limitations

Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths short and avoid hitting MAX_PATH limitations.

Changes lit logic to:

Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we
can work around using substitute drives, which `os.path.realpath` would
resolve.

Use `os.path.realpath` on Unix, where the current directory always has
symlinks resolved, so it is impossible to preserve symlinks in the
presence of relative paths, and so we must make sure that all code paths
use real paths.

Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI`
code to avoid resolving substitute drives (i.e. resolving to a path
under a different root).

How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows

Differential Revision: https://reviews.llvm.org/D154130
Reviewed By: @benlangmuir

Patch by Tristan Labelle <tristan@thebrowser.company>!

show more ...

[clang] NFC: Use `DirectoryEntryRef` in `FileManager::getCanonicalName()`

This patch removes the last use of deprecated `DirectoryEntry::getName()`.

Depends on D151855.

Reviewed By: benlangmuir

D

[clang] NFC: Use `DirectoryEntryRef` in `FileManager::getCanonicalName()`

This patch removes the last use of deprecated `DirectoryEntry::getName()`.

Depends on D151855.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D151922

show more ...

[clang] Make `FileEntryRef::getDir()` return the as-requested `DirectoryEntryRef`

For redirected file entries, `FileEntryRef::getDir()` returns the parent directory entry of the target file entry. T

[clang] Make `FileEntryRef::getDir()` return the as-requested `DirectoryEntryRef`

For redirected file entries, `FileEntryRef::getDir()` returns the parent directory entry of the target file entry. This differs from `FileEntry::getDir()` that always returns the parent directory that was last used to look up that file.

After switching from `FileEntry` to `FileEntryRef` for umbrella headers in D142113, this discrepancy became observable and caused Clang to emit incorrect diagnostics.

This patch changes Clang so that it always associates `FileEntryRef` with the parent directory that was used to look it up. This brings its behavior closer to `FileEntry`, but without the hacky mutation.

This also ensures that `llvm::sys::path::parent_path(FileRef->getNameAsRequested()) == FileRef->getDir()->getName()`. Previously, `FileRef->getDir()` would fall underneath the redirecting VFS into the world of on-disk paths.

Reviewed By: benlangmuir, rmaz

Differential Revision: https://reviews.llvm.org/D151398

show more ...

[clang] NFC: Remove GCC 5.1 workaround in FileEntry

We no longer support GCC 5.1 (D122976), so let's remove the workaround in FileEntry.

Reviewed By: benlangmuir

Differential Revision: https://rev

[clang] NFC: Remove GCC 5.1 workaround in FileEntry

We no longer support GCC 5.1 (D122976), so let's remove the workaround in FileEntry.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D143427

show more ...

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around

[Clang] Prepare for llvm::Optional becoming std::optional.

The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.

I tested this with GCC 7.5, the oldest supported GCC I had around.

Differential Revision: https://reviews.llvm.org/D140332

show more ...

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to

Revert "[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional"

This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.

The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.

On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).

show more ...

[clang] Convert OptionalFileEntryRefDegradesToFileEntryPtr to std::optional