[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally p

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).

For now this is accomplished through the use of a new
`ExposesExternalVFSPath` field on `vfs::Status`. This flag is true when
the `Status` has an external path that's different from its virtual
path, ie. the contained path is the external path. See the plan in
`FileManager::getFileRef` for where this is going - eventually we won't
need `IsVFSMapped` any more and all returned paths should be virtual.

Resolves rdar://90578880 and llvm-project#53306.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D123398

show more ...

[RGT] Use GTEST_SKIP() in more places where we skip a test

Simply returning will report the test as PASSED when it didn't
really do anything. SKIPPED is the correct result for these.

Found by the R

[RGT] Use GTEST_SKIP() in more places where we skip a test

Simply returning will report the test as PASSED when it didn't
really do anything. SKIPPED is the correct result for these.

Found by the Rotten Green Tests project.

show more ...

Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstanc

Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstances. Specifically,
since crash reproducers have `UseExternalNames` set to false, the
`File->getFileEntry().getDir()->getName()` call in `DoFrameworkLookup`
would use the *cached* directory name instead of the directory of the
looked-up file.

The plan is to re-commit this patch but to *add*
`ExposesExternalVFSPath` rather than replace `IsVFSMapped`.

Differential Revision: https://reviews.llvm.org/D123103

show more ...

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
p

[VFS] RedirectingFileSystem only replace path if not already mapped

If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).

This also renames `IsVFSMapped` to `ExposesExternalVFSPath` and only
sets it if `UseExternalName` is true. This flag then represents that the
`Status` has an external path that's different from its virtual path.
Right now the contained path is still the external path, but further PRs
will change this to *always* be the virtual path. Clients that need the
external can then request it specifically.

Note that even though `ExposesExternalVFSPath` isn't set for all
VFS-mapped paths, `IsVFSMapped` was only being used by a hack in
`FileManager` that was specific to module searching. In that case
`UseExternalNames` is always `true` and so that hack still applies.

Resolves rdar://90578880 and llvm-project#53306.

Differential Revision: https://reviews.llvm.org/D122549

show more ...

[VFS] Add print/dump to the whole FileSystem hierarchy

For now most are implemented by printing out the name of the filesystem,
but this can be expanded in the future. Only `OverlayFileSystem` and
`

[VFS] Add print/dump to the whole FileSystem hierarchy

For now most are implemented by printing out the name of the filesystem,
but this can be expanded in the future. Only `OverlayFileSystem` and
`RedirectingFileSystem` are properly implemented in this patch.
- `OverlayFileSystem`: Prints each filesystem in the order that any
operations are actually run on them. Optionally prints recursively.
- `RedirectingFileSystem`: Prints out all mappings, as well as the
`ExternalFS`. Most of this was already implemented other than the
handling for the `DirectoryRemap` case and to actually print out the
mapping.

Each FS should implement `printImpl` rather than `print`, where the
latter just fowards to the former. This is to avoid spreading the
default arguments through to the subclasses (where we may miss updating
in the future).

Differential Revision: https://reviews.llvm.org/D121421

show more ...

[VFS] Add a "redirecting-with" field to overlays

Extend "fallthrough" to allow a third option: "fallback". Fallthrough
allows the original path to used if the redirected (or mapped) path
fails. Fall

[VFS] Add a "redirecting-with" field to overlays

Extend "fallthrough" to allow a third option: "fallback". Fallthrough
allows the original path to used if the redirected (or mapped) path
fails. Fallback is the reverse of this, ie. use the original path and
fallback to the mapped path otherwise.

While this result *can* be achieved today using multiple overlays, this
adds a much more intuitive option. As an example, take two directories
"A" and "B". We would like files from "A" to be used, unless they don't
exist, in which case the VFS should fallback to those in "B".

With the current fallthrough option this is possible by adding two
overlays: one mapping from A -> B and another mapping from B -> A. Since
the frontend *nests* the two RedirectingFileSystems, the result will
be that "A" is mapped to "B" and back to "A", unless it isn't in "A" in
which case it fallsthrough to "B" (or fails if it exists in neither).

Using "fallback" semantics allows a single overlay instead: one mapping
from "A" to "B" but only using that mapping if the operation in "A"
fails first.

"redirect-only" is used to represent the current "fallthrough: false"
case.

Differential Revision: https://reviews.llvm.org/D117937

show more ...

[NFCI][Support] Avoid ASSERT_/EXPECT_TRUE(A <op> B)

The error messages in tests are far better when a test fails if the test
is written using ASSERT_/EXPECT_<operator>(A, B) rather than
ASSERT_/EXPE

[NFCI][Support] Avoid ASSERT_/EXPECT_TRUE(A <op> B)

The error messages in tests are far better when a test fails if the test
is written using ASSERT_/EXPECT_<operator>(A, B) rather than
ASSERT_/EXPECT_TRUE(A <operator> B).

This commit updates all of llvm/unittests/Support to use these macros
where possible.

This change has not been possible in:
- llvm/unittests/Support/FSUniqueIDTest.cpp - due to not overloading
operators beyond ==, != and <.
- llvm/unittests/Support/BranchProbabilityTest.cpp - where the unchanged
tests are of the operator overloads themselves.

There are other possibilities of this conversion not being valid, which
have not applied in these tests, as they do not use NULL (they use
nullptr), and they do not use const char* (they use std::string or
StringRef).

Reviewed By: mubashar_

Differential Revision: https://reviews.llvm.org/D117319

show more ...

[clang] support relative roots to vfs overlays

This diff adds support for relative roots to VFS overlays. The directory root
will be made absolute from the current working directory and will be used

[clang] support relative roots to vfs overlays

This diff adds support for relative roots to VFS overlays. The directory root
will be made absolute from the current working directory and will be used to
determine the path style to use. This supports the use of VFS overlays with
remote build systems that might use a different working directory for each
compilation.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D116174

show more ...

reland: [VFS] Use original path when falling back to external FS

This reverts commit f0cf544d6f6fe6cbca4c07772998272d6bb433d8.

Just a small change to fix:

```
/home/buildbot/as-builder-4/llvm-clan

reland: [VFS] Use original path when falling back to external FS

This reverts commit f0cf544d6f6fe6cbca4c07772998272d6bb433d8.

Just a small change to fix:

```
/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp: In static member function ‘static llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> > llvm::vfs::File::getWithPath(llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >, const llvm::Twine&)’:
/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp:2084:10: error: could not convert ‘F’ from ‘std::unique_ptr<llvm::vfs::File>’ to ‘llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >’
return F;
^
```

Differential Revision: https://reviews.llvm.org/D113832

show more ...

Revert "[VFS] Use original path when falling back to external FS"

```
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp: In static member function 'sta

Revert "[VFS] Use original path when falling back to external FS"

```
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp: In static member function 'static llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> > llvm::vfs::File::getWithPath(llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >, const llvm::Twine&)':
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp:2084:10: error: could not convert 'F' from 'std::unique_ptr<llvm::vfs::File>' to 'llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >'
return F;
^
```

This reverts commit c972175649f4bb50d40d911659a04d5620ce6fe0.

show more ...

[VFS] Use original path when falling back to external FS

This is a follow up to 0be9ca7c0f9a733f846bb6bc4e8e36d46b518162 to make
paths in the case of falling back to the external file system use the

[VFS] Use original path when falling back to external FS

This is a follow up to 0be9ca7c0f9a733f846bb6bc4e8e36d46b518162 to make
paths in the case of falling back to the external file system use the
original format, preserving relative paths, and allow the external
filesystem to canonicalize them if needed.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D109128

show more ...

[VFS] InMemoryFilesystem's UniqueIDs are a function of path and content.

This ensures that re-creating "the same" FS results in the same UIDs for files.
In turn, this means that creating a clang mod

[VFS] InMemoryFilesystem's UniqueIDs are a function of path and content.

This ensures that re-creating "the same" FS results in the same UIDs for files.
In turn, this means that creating a clang module (preamble) using one in-memory
filesystem and consuming it using another doesn't create duplicate FileEntrys
for files that are the same in both FSes.

It's tempting to give the creator control over the UIDs instead. However that
requires fiddly API changes, e.g. what should the UIDs of intermediate
directories be?
This change is more "magic" but seems safe given:
- InMemoryFilesystem is used in testing more than production
- comparing UIDs across filesystems is unusual
- files with the same path and content are usually logically equivalent

(The usual reason for re-creating virtual filesystems rather than reusing them
is that typical use involves mutating their CWD and so is not threadsafe).

Differential Revision: https://reviews.llvm.org/D110711

show more ...

[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

Previously file entries in the -ivfsoverlay yaml could map to a file in the
external file system

[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

Previously file entries in the -ivfsoverlay yaml could map to a file in the
external file system, but directories had to list their contents in the form of
other file entries or directories. Allowing directory entries to map to a
directory in the external file system makes it possible to present an external
directory's contents in a different location and (in combination with the
'fallthrough' option) overlay one directory's contents on top of another.

rdar://problem/72485443
Differential Revision: https://reviews.llvm.org/D94844

show more ...

[VFS] Fix inconsistencies between relative paths and fallthrough.

This patch addresses inconsistencies in the way fallthrough is handled
in the RedirectingFileSystem. Rather than trying to change th

[VFS] Fix inconsistencies between relative paths and fallthrough.

This patch addresses inconsistencies in the way fallthrough is handled
in the RedirectingFileSystem. Rather than trying to change the working
directory of the external filesystem, the RedirectingFileSystem will
canonicalize every path before handing it down. This guarantees that
relative paths are resolved relative to the RedirectingFileSystem's
working directory.

This allows us to have a strictly virtual working directory, and still
fallthrough for absolute paths, but not for relative paths that would
get resolved incorrectly at the lower layer (for example, in case of the
RealFileSystem, because the strictly virtual path does not exist).

Differential revision: https://reviews.llvm.org/D95188

show more ...

Support: Add RedirectingFileSystem::create from simple list of redirections

Add an overload of `RedirectingFileSystem::create` that builds a
redirecting filesystem off of a simple vector of string p

Support: Add RedirectingFileSystem::create from simple list of redirections

Add an overload of `RedirectingFileSystem::create` that builds a
redirecting filesystem off of a simple vector of string pairs. This is
intended to be used to support `clang::arcmt::FileRemapper` and
`clang::PreprocessorOptions::RemappedFiles`.

Differential Revision: https://reviews.llvm.org/D91317

show more ...

VFS: Return new file systems as uniquely owned when possible, almost NFC

Uniformly return uniquely-owned filesystems from VFS creation APIs. The
one exception is `getRealFileSystem`, which has a sin

VFS: Return new file systems as uniquely owned when possible, almost NFC

Uniformly return uniquely-owned filesystems from VFS creation APIs. The
one exception is `getRealFileSystem`, which has a single instance and
needs to be shared.

This is almost NFC, except that it fixes a memory leak in
`vfs::collectVFSFromYAML()`.

Depends on https://reviews.llvm.org/D92888

Differential Revision: https://reviews.llvm.org/D92890

show more ...

Support: Change InMemoryFileSystem::addFileNoOwn to take a MemoryBufferRef, NFC

Found this by chance when looking at the InMemoryFileSystem API, seems
like an easy cleanup.

Differential Revision: h

Support: Change InMemoryFileSystem::addFileNoOwn to take a MemoryBufferRef, NFC

Found this by chance when looking at the InMemoryFileSystem API, seems
like an easy cleanup.

Differential Revision: https://reviews.llvm.org/D90893

show more ...

[llvm] [unittests] Remove temporary files after they're not needed

Some LLVM unit tests forget to clean up temporary files and
directories. Introduce RAII classes for cleaning them up.

Refactor the

[llvm] [unittests] Remove temporary files after they're not needed

Some LLVM unit tests forget to clean up temporary files and
directories. Introduce RAII classes for cleaning them up.

Refactor the tests to use those classes.

Differential Revision: https://reviews.llvm.org/D83228

show more ...

Revert "Relands "[YAMLVFSWriter][Test][NFC] Add couple tests" vol. 2"

This reverts commit 30949926f98576fbff8d5ad0390be5124ffacd7e.

This was failing on bot here:
http://lab.llvm.org:8011/builders/l

Revert "Relands "[YAMLVFSWriter][Test][NFC] Add couple tests" vol. 2"

This reverts commit 30949926f98576fbff8d5ad0390be5124ffacd7e.

This was failing on bot here:
http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/32749

show more ...

Relands "[YAMLVFSWriter][Test][NFC] Add couple tests" vol. 2

This reverts commit e4e3e41905d182c0f3d5b0b9406e3cbf2aabb30f.

Fixed dangling StringRef in test.

[YAMLVFSWriter] Fix for delimiters

Differential Revision: https://reviews.llvm.org/D79809

[VirtualFileSystem] Add unit test that showcases another YAMLVFSWriter bug

This scenario generates another broken YAML mapping as illustrated below.

{
'type': 'directory',
'name': "c",

[VirtualFileSystem] Add unit test that showcases another YAMLVFSWriter bug

This scenario generates another broken YAML mapping as illustrated below.

{
'type': 'directory',
'name': "c",
'contents': [
,
{
'type': 'directory',
'name': "d",
'contents': [
,
{
'type': 'directory',
'name': "e",
'contents': [
{
'type': 'file',
'name': "f",
'external-contents': "//root/a/c/d/e/f"
} {
'type': 'file',
'name': "g",
'external-contents': "//root/a/c/d/e/g"
}
]
}
]
}
]
},

show more ...

[VirtualFileSystem] Add unit test that showcases YAMLVFSWriter bug

This scenario generates a broken YAML mapping as illustrated below.

{
'type': 'directory',
'name': "c",
'contents': [

[VirtualFileSystem] Add unit test that showcases YAMLVFSWriter bug

This scenario generates a broken YAML mapping as illustrated below.

{
'type': 'directory',
'name': "c",
'contents': [
{
'type': 'file',
'name': "d",
'external-contents': "//root/a/c/d"
} {
'type': 'file',
'name': "e",
'external-contents': "//root/a/c/e"
} {
'type': 'file',
'name': "f",
'external-contents': "//root/a/c/f"
}
]
},

show more ...

Revert "Relands "[YAMLVFSWriter][Test][NFC] Add couple tests""

This reverts commit 49b32d80416288b6eb8e26f76c40a8e32c20a361.

Relands "[YAMLVFSWriter][Test][NFC] Add couple tests"

Fixed test for Windows.

Differential Revision: https://reviews.llvm.org/D79552