[llvm] Use value_or instead of getValueOr (NFC)

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally p

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).

For now this is accomplished through the use of a new
`ExposesExternalVFSPath` field on `vfs::Status`. This flag is true when
the `Status` has an external path that's different from its virtual
path, ie. the contained path is the external path. See the plan in
`FileManager::getFileRef` for where this is going - eventually we won't
need `IsVFSMapped` any more and all returned paths should be virtual.

Resolves rdar://90578880 and llvm-project#53306.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D123398

show more ...

Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstanc

Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstances. Specifically,
since crash reproducers have `UseExternalNames` set to false, the
`File->getFileEntry().getDir()->getName()` call in `DoFrameworkLookup`
would use the *cached* directory name instead of the directory of the
looked-up file.

The plan is to re-commit this patch but to *add*
`ExposesExternalVFSPath` rather than replace `IsVFSMapped`.

Differential Revision: https://reviews.llvm.org/D123103

show more ...

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
p

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).

This also renames `IsVFSMapped` to `ExposesExternalVFSPath` and only
sets it if `UseExternalName` is true. This flag then represents that the
`Status` has an external path that's different from its virtual path.
Right now the contained path is still the external path, but further PRs
will change this to *always* be the virtual path. Clients that need the
external can then request it specifically.

Note that even though `ExposesExternalVFSPath` isn't set for all
VFS-mapped paths, `IsVFSMapped` was only being used by a hack in
`FileManager` that was specific to module searching. In that case
`UseExternalNames` is always `true` and so that hack still applies.

Resolves rdar://90578880 and llvm-project#53306.

Differential Revision: https://reviews.llvm.org/D122549

show more ...

[VFS] Add print/dump to the whole FileSystem hierarchy

For now most are implemented by printing out the name of the filesystem,
but this can be expanded in the future. Only `OverlayFileSystem` and
`

[VFS] Add print/dump to the whole FileSystem hierarchy

For now most are implemented by printing out the name of the filesystem,
but this can be expanded in the future. Only `OverlayFileSystem` and
`RedirectingFileSystem` are properly implemented in this patch.
- `OverlayFileSystem`: Prints each filesystem in the order that any
operations are actually run on them. Optionally prints recursively.
- `RedirectingFileSystem`: Prints out all mappings, as well as the
`ExternalFS`. Most of this was already implemented other than the
handling for the `DirectoryRemap` case and to actually print out the
mapping.

Each FS should implement `printImpl` rather than `print`, where the
latter just fowards to the former. This is to avoid spreading the
default arguments through to the subclasses (where we may miss updating
in the future).

Differential Revision: https://reviews.llvm.org/D121421

show more ...

[VFS] Rename `RedirectingFileSystem::dump` to `print`

The rest of LLVM uses `print` for the method taking the `raw_ostream`
and `dump` only for the method with no parameters. Use the same for
`Redir

[VFS] Rename `RedirectingFileSystem::dump` to `print`

The rest of LLVM uses `print` for the method taking the `raw_ostream`
and `dump` only for the method with no parameters. Use the same for
`RedirectingFileSystem`.

Differential Revision: https://reviews.llvm.org/D121494

show more ...

[VFS] Use generic_category for errors generated from the VFS

Errors are generally checked in clients by comparing to the portable
error condition in `std::errc`, which will have the `generic_categor

[VFS] Use generic_category for errors generated from the VFS

Errors are generally checked in clients by comparing to the portable
error condition in `std::errc`, which will have the `generic_category`
(eg. `std::errc::no_such_file_or_directory`). While in practice these
are usually equivalent for the standard errno's, they are not in *all*
implementations. One such example is CentOS 7.

Differential Revision: https://reviews.llvm.org/D120299

show more ...

[VFS] Add back setFallthrough for downstream users

This fixes lldb's build. We can remove this in the future if we want but
for now this will be nicer to existing consumers.

[VFS] Add a "redirecting-with" field to overlays

Extend "fallthrough" to allow a third option: "fallback". Fallthrough
allows the original path to used if the redirected (or mapped) path
fails. Fall

[VFS] Add a "redirecting-with" field to overlays

Extend "fallthrough" to allow a third option: "fallback". Fallthrough
allows the original path to used if the redirected (or mapped) path
fails. Fallback is the reverse of this, ie. use the original path and
fallback to the mapped path otherwise.

While this result *can* be achieved today using multiple overlays, this
adds a much more intuitive option. As an example, take two directories
"A" and "B". We would like files from "A" to be used, unless they don't
exist, in which case the VFS should fallback to those in "B".

With the current fallthrough option this is possible by adding two
overlays: one mapping from A -> B and another mapping from B -> A. Since
the frontend *nests* the two RedirectingFileSystems, the result will
be that "A" is mapped to "B" and back to "A", unless it isn't in "A" in
which case it fallsthrough to "B" (or fails if it exists in neither).

Using "fallback" semantics allows a single overlay instead: one mapping
from "A" to "B" but only using that mapping if the operation in "A"
fails first.

"redirect-only" is used to represent the current "fallthrough: false"
case.

Differential Revision: https://reviews.llvm.org/D117937

show more ...

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. R

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. Removing unused headers in .cpp files. No impact expected.
3. Removing unused headers in .h files. This removes implicit dependencies and
is generally considered a good thing, but this may break downstream builds.
I've updated llvm, clang, lld, lldb and mlir deps, and included a list of the
modification in the second part of the commit.
4. Replacing header inclusion by forward declaration. This has the same impact
as 3.

Notable changes:

- llvm/Support/TargetParser.h no longer includes llvm/Support/AArch64TargetParser.h nor llvm/Support/ARMTargetParser.h
- llvm/Support/TypeSize.h no longer includes llvm/Support/WithColor.h
- llvm/Support/YAMLTraits.h no longer includes llvm/Support/Regex.h
- llvm/ADT/SmallVector.h no longer includes llvm/Support/MemAlloc.h nor llvm/Support/ErrorHandling.h

You may need to add some of these headers in your compilation units, if needs be.

As an hint to the impact of the cleanup, running

clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Support/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l

before: 8000919 lines
after: 7917500 lines

Reduced dependencies also helps incremental rebuilds and is more ccache
friendly, something not shown by the above metric :-)

Discourse thread on the topic: https://llvm.discourse.group/t/include-what-you-use-include-cleanup/5831

show more ...

[llvm][vfs] Abstract in-memory node creation

The creation of in-memory VFS nodes happens in a single function that deduces what kind of node to create from the arguments. This leads to complicated i

[llvm][vfs] Abstract in-memory node creation

The creation of in-memory VFS nodes happens in a single function that deduces what kind of node to create from the arguments. This leads to complicated if-then-else logic that's difficult to cleanly extend.

This patch abstracts away in-memory node creation via a type-erased factory function that's passed instead.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D117648

show more ...

[llvm][vfs] NFC: Virtualize in-memory `getStatus`

This patch virtualizes the `getStatus` function on `InMemoryNode` in LLVM VFS. Currently, this is implemented via top-level function `getNodeStatus`

[llvm][vfs] NFC: Virtualize in-memory `getStatus`

This patch virtualizes the `getStatus` function on `InMemoryNode` in LLVM VFS. Currently, this is implemented via top-level function `getNodeStatus` that tries to cast `InMemoryNode *` into each subtype. Virtual functions seem to be the simpler solution here.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D117649

show more ...

[clang] support relative roots to vfs overlays

This diff adds support for relative roots to VFS overlays. The directory root
will be made absolute from the current working directory and will be used

[clang] support relative roots to vfs overlays

This diff adds support for relative roots to VFS overlays. The directory root
will be made absolute from the current working directory and will be used to
determine the path style to use. This supports the use of VFS overlays with
remote build systems that might use a different working directory for each
compilation.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D116174

show more ...

[clang][deps] Squash caches for original and minimized files

The minimizing and caching filesystem used by the dependency scanner keeps minimized and original files in separate caches.

This setup i

[clang][deps] Squash caches for original and minimized files

The minimizing and caching filesystem used by the dependency scanner keeps minimized and original files in separate caches.

This setup is not well suited for dealing with files that are sometimes minimized and sometimes not. Such files are being stat-ed and read twice, which is wasteful and also means the two versions of the file can get "out of sync".

This patch squashes the two caches together. When a file is stat-ed or read, its original contents are populated. If a file needs to be minimized, we give the minimizer the already loaded contents instead of reading the file again.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D115346

show more ...

reland: [VFS] Use original path when falling back to external FS

This reverts commit f0cf544d6f6fe6cbca4c07772998272d6bb433d8.

Just a small change to fix:

```
/home/buildbot/as-builder-4/llvm-clan

reland: [VFS] Use original path when falling back to external FS

This reverts commit f0cf544d6f6fe6cbca4c07772998272d6bb433d8.

Just a small change to fix:

```
/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp: In static member function ‘static llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> > llvm::vfs::File::getWithPath(llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >, const llvm::Twine&)’:
/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp:2084:10: error: could not convert ‘F’ from ‘std::unique_ptr<llvm::vfs::File>’ to ‘llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >’
return F;
^
```

Differential Revision: https://reviews.llvm.org/D113832

show more ...

Revert "[VFS] Use original path when falling back to external FS"

```
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp: In static member function 'sta

Revert "[VFS] Use original path when falling back to external FS"

```
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp: In static member function 'static llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> > llvm::vfs::File::getWithPath(llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >, const llvm::Twine&)':
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp:2084:10: error: could not convert 'F' from 'std::unique_ptr<llvm::vfs::File>' to 'llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >'
return F;
^
```

This reverts commit c972175649f4bb50d40d911659a04d5620ce6fe0.

show more ...

[VFS] Use original path when falling back to external FS

This is a follow up to 0be9ca7c0f9a733f846bb6bc4e8e36d46b518162 to make
paths in the case of falling back to the external file system use the

[VFS] Use original path when falling back to external FS

This is a follow up to 0be9ca7c0f9a733f846bb6bc4e8e36d46b518162 to make
paths in the case of falling back to the external file system use the
original format, preserving relative paths, and allow the external
filesystem to canonicalize them if needed.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D109128

show more ...

[Support] [VirtualFileSystem] Detect the windows_slash path style

This fixes the following clang VFS tests, if `windows_slash` is the
default style:

Clang :: VFS/implicit-include.c
Clang :: VFS

[Support] [VirtualFileSystem] Detect the windows_slash path style

This fixes the following clang VFS tests, if `windows_slash` is the
default style:

Clang :: VFS/implicit-include.c
Clang :: VFS/relative-path.c
Clang-Unit :: Frontend/./FrontendTests.exe/CompilerInstance.DefaultVFSOverlayFromInvocation

Also clarify a couple references to `Style::windows` into
`Style::windows_backslash`, to make it clearer that each of them are
opinionated in different directions (even if it doesn't matter for
calls to e.g. `is_absolute`).

Differential Revision: https://reviews.llvm.org/D113272

show more ...

Put implementation details into anonymous namespaces. NFCI.

[VFS] InMemoryFilesystem's UniqueIDs are a function of path and content.

This ensures that re-creating "the same" FS results in the same UIDs for files.
In turn, this means that creating a clang mod

[VFS] InMemoryFilesystem's UniqueIDs are a function of path and content.

This ensures that re-creating "the same" FS results in the same UIDs for files.
In turn, this means that creating a clang module (preamble) using one in-memory
filesystem and consuming it using another doesn't create duplicate FileEntrys
for files that are the same in both FSes.

It's tempting to give the creator control over the UIDs instead. However that
requires fiddly API changes, e.g. what should the UIDs of intermediate
directories be?
This change is more "magic" but seems safe given:
- InMemoryFilesystem is used in testing more than production
- comparing UIDs across filesystems is unusual
- files with the same path and content are usually logically equivalent

(The usual reason for re-creating virtual filesystems rather than reusing them
is that typical use involves mutating their CWD and so is not threadsafe).

Differential Revision: https://reviews.llvm.org/D110711

show more ...

[Support] Qualify auto (NFC)

Identified with readability-qualified-auto.

PR51018: Remove explicit conversions from SmallString to StringRef to future-proof against C++23

C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible wit

PR51018: Remove explicit conversions from SmallString to StringRef to future-proof against C++23

C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible with C++23 (& a follow-up change I've made
will make this ambiguous/invalid even in <C++23 so we don't regress
this & it generally improves the code anyway)

show more ...

[llvm] Rename StringRef _lower() method calls to _insensitive()

This is a mechanical change. This actually also renames the
similarly named methods in the SmallString class, however these
methods do

[llvm] Rename StringRef _lower() method calls to _insensitive()

This is a mechanical change. This actually also renames the
similarly named methods in the SmallString class, however these
methods don't seem to be used outside of the llvm subproject, so
this doesn't break building of the rest of the monorepo.

show more ...

[clang] Don't crash when loading invalid VFS for the module dep collector

The VFS is null when it's invalid so return early in collectVFSFromYAML.

[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

Previously file entries in the -ivfsoverlay yaml could map to a file in the
external file system

[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

Previously file entries in the -ivfsoverlay yaml could map to a file in the
external file system, but directories had to list their contents in the form of
other file entries or directories. Allowing directory entries to map to a
directory in the external file system makes it possible to present an external
directory's contents in a different location and (in combination with the
'fallthrough' option) overlay one directory's contents on top of another.

rdar://problem/72485443
Differential Revision: https://reviews.llvm.org/D94844

show more ...