[Support] Move llvm::MemoryBuffer to sys::fs::file_t

Summary:
On Windows, Posix integer file descriptors are a compatibility layer
over native file handles provided by the C runtime. There is a hard

[Support] Move llvm::MemoryBuffer to sys::fs::file_t

Summary:
On Windows, Posix integer file descriptors are a compatibility layer
over native file handles provided by the C runtime. There is a hard
limit on the maximum number of file descriptors that a process can open,
and the limit is 8192. LLD typically doesn't run into this limit because
it opens input files, maps them into memory, and then immediately closes
the file descriptor. This prevents it from running out of FDs.

For various reasons, I'd like to open handles to every input file and
keep them open during linking. That requires migrating MemoryBuffer over
to taking open native file handles instead of integer FDs.

Reviewers: aganea, Bigcheese

Reviewed By: aganea

Subscribers: smeenai, silvas, mehdi_amini, hiraditya, steven_wu, dexonsmith, dang, llvm-commits, zturner

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D63453

llvm-svn: 365588

show more ...

[VirtualFileSystem] Fix uninitialized variable warning. NFCI.

llvm-svn: 361371

VFS: Avoid some unnecessary std::string copies

Thread Twine a little deeper through the VFS to avoid unnecessarily
constructing the same std::string twice in a parameter sequence:

Twine -> std:

VFS: Avoid some unnecessary std::string copies

Thread Twine a little deeper through the VFS to avoid unnecessarily
constructing the same std::string twice in a parameter sequence:

Twine -> std::string -> StringRef -> std::string

Changing a few parameters from StringRef to Twine avoids the early call
to `Twine::str()`.

llvm-svn: 354739

show more ...

Reapply [VFS] Allow multiple RealFileSystem instances with independent CWDs.

This reverts commit r351091.
The original mac breakages are addressed by ensuring the root directory
we're working from i

Reapply [VFS] Allow multiple RealFileSystem instances with independent CWDs.

This reverts commit r351091.
The original mac breakages are addressed by ensuring the root directory
we're working from is fully symlink-resolved before starting.

Differential Revision: https://reviews.llvm.org/D58169

llvm-svn: 354026

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

[Support] Remove error return value from one overload of fs::make_absolute

Summary:
The version of make_absolute which accepted a specific directory to use
as the "base" for the computation could ne

[Support] Remove error return value from one overload of fs::make_absolute

Summary:
The version of make_absolute which accepted a specific directory to use
as the "base" for the computation could never fail, even though it
returned a std::error_code. The reason for that seems to be historical
-- the CWD flavour (which can fail due to failure to retrieve CWD) was
there first, and the new version was implemented by extending that.

This removes the error return value from the non-CWD overload and
reimplements the CWD version on top of that. This enables us to remove
some dead code where people were pessimistically trying to handle the
errors returned from this function.

Reviewers: zturner, sammccall

Subscribers: hiraditya, kristina, llvm-commits

Differential Revision: https://reviews.llvm.org/D56599

llvm-svn: 351317

show more ...

[VFS] Move RedirectingFileSystem interface into header (NFC)

This moves the RedirectingFileSystem into the header so it can be
extended. This is needed in LLDB we need a way to obtain the external
p

[VFS] Move RedirectingFileSystem interface into header (NFC)

This moves the RedirectingFileSystem into the header so it can be
extended. This is needed in LLDB we need a way to obtain the external
path to deal with FILE* and file descriptor APIs.

Discussion on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-November/127755.html

Differential revision: https://reviews.llvm.org/D54277

llvm-svn: 351265

show more ...

Revert "[VFS] Allow multiple RealFileSystem instances with independent CWDs."
This reverts commit r351079, r351069 and r351050 as it broken the greendragon bots on macOS.

llvm-svn: 351091

[VFS] Fix unused variable warning. NFC

llvm-svn: 351069

[VFS] Allow multiple RealFileSystem instances with independent CWDs.

Summary:
Previously only one RealFileSystem instance was available, and its working
directory is shared with the process. This do

[VFS] Allow multiple RealFileSystem instances with independent CWDs.

Summary:
Previously only one RealFileSystem instance was available, and its working
directory is shared with the process. This doesn't work well for multithreaded
programs that want to work with relative paths - the vfs::FileSystem is assumed
to provide the working directory, but a thread cannot control this exclusively.

The new vfs::createPhysicalFileSystem() factory copies the process's working
directory initially, and then allows it to be independently modified.

This implementation records the working directory path, and glues it to relative
paths to provide the correct absolute path to the sys::fs:: functions.
This will give different results in unusual situations (e.g. the CWD is moved).

The main alternative is the use of openat(), fstatat(), etc to ask the OS to
resolve paths relative to a directory handle which can be kept open. This is
more robust. There are two reasons not to do this initially:
1. these functions are not available on all supported Unixes, and are somewhere
between difficult and unavailable on Windows. So we need a path-based
fallback anyway.
2. this would mean also adding support at the llvm::sys::fs level, which is a
larger project. My clearest idea is an OS-specific `BaseDirectory` object
that can be optionally passed to functions there. Eventually this could be
backed by either paths or a fd where openat() is supported.
This is a large project, and demonstrating here that a path-based fallback
works is a useful prerequisite.

There is some subtlety to the path-manipulation mechanism:
- when setting the working directory, both Specified=makeAbsolute(path) and
Resolved=realpath(path) are recorded. These may differ in the presence of
symlinks.
- getCurrentWorkingDirectory() and makeAbsolute() use Specified - this is
similar to the behavior of $PWD and sys::path::current_path
- IO operations like openFileForRead use Resolved. This is similar to the
behavior of an openat() based implementation, that doesn't see changes
in symlinks.
There may still be combinations of operations and FS states that yield unhelpful
behavior. This is hard to avoid with symlinks and FS abstractions :(

The caching behavior of the current working directory is removed in this patch.
getRealFileSystem() is now specified to link to the process CWD, so the caching
is incorrect.
The user who needed this so far is clangd, which will immediately switch to
createPhysicalFileSystem().

Reviewers: ilya-biryukov, bkramer, labath

Subscribers: ioeric, kadircet, kristina, llvm-commits

Differential Revision: https://reviews.llvm.org/D56545

llvm-svn: 351050

show more ...

Add vtable anchor to classes.

llvm-svn: 350142

[VFS] Implement `RedirectingFileSystem::getRealPath`.

It fixes the case when Objective-C framework is added as a subframework
through a symlink. When parent framework infers a module map and fails
t

[VFS] Implement `RedirectingFileSystem::getRealPath`.

It fixes the case when Objective-C framework is added as a subframework
through a symlink. When parent framework infers a module map and fails
to detect a symlink, it would add a subframework as a submodule. And
when we parse module map for the subframework, we would encounter an
error like

> error: umbrella for module 'WithSubframework.Foo' already covers this directory

By implementing `getRealPath` "an egregious but useful hack" in
`ModuleMap::inferFrameworkModule` works as expected.

rdar://problem/45821279

Reviewers: bruno, benlangmuir, erik.pilkington

Reviewed By: bruno

Subscribers: hiraditya, dexonsmith, JDevlieghere, cfe-commits, llvm-commits

Differential Revision: https://reviews.llvm.org/D54245

llvm-svn: 347009

show more ...

Revert "[VFS] Add "expand tilde" argument to getRealPath."

This reverts commit r346453.
This is a complex change to a widely-used interface, and was not reviewed.

llvm-svn: 346500

[VFS] Add "expand tilde" argument to getRealPath.

Add an optional argument to expand tildes in the path to mirror llvm's
implementation of the corresponding function.

llvm-svn: 346453

Extend virtual file system with `isLocal` method

Expose the `llvm::sys::fs::is_local` function through the VFS.

Differential revision: https://reviews.llvm.org/D54127

llvm-svn: 346372

[VFS] Add support for "no_push" to VFS recursive iterators.

The "regular" file system has a useful feature that makes it possible to
stop recursing when using the recursive directory iterators. This

[VFS] Add support for "no_push" to VFS recursive iterators.

The "regular" file system has a useful feature that makes it possible to
stop recursing when using the recursive directory iterators. This
functionality was missing for the VFS recursive iterator and this patch
adds that.

Differential revision: https://reviews.llvm.org/D53465

llvm-svn: 345793

show more ...

Add parens to fix incorrect assert check.

&& has higher priority than ||, so this assert works really oddly. Add
parens to match the programmer's intent.

Change-Id: I3abe1361ee0694462190c5015779db6

Add parens to fix incorrect assert check.

&& has higher priority than ||, so this assert works really oddly. Add
parens to match the programmer's intent.

Change-Id: I3abe1361ee0694462190c5015779db664012f3d4
llvm-svn: 345543

show more ...

[VFS] Add property 'fallthrough' that controls fallback to real file system.

Default property value 'true' preserves current behavior. Value 'false' can be
used to create VFS "root", file system tha

[VFS] Add property 'fallthrough' that controls fallback to real file system.

Default property value 'true' preserves current behavior. Value 'false' can be
used to create VFS "root", file system that gives better control over which
files compiler can use during compilation as there are no unpredictable
accesses to real file system.

Non-fallthrough use case changes how we treat multiple VFS overlay
files. Instead of all of them being at the same level just above a real
file system, now they are nested and subsequent overlays can refer to
files in previous overlays.

rdar://problem/39465552

Reviewers: bruno, benlangmuir

Reviewed By: bruno

Subscribers: dexonsmith, cfe-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D50539

llvm-svn: 345431

show more ...

[VFS] Remove 'ignore-non-existent-contents' attribute for YAML-based VFS.

'ignore-non-existent-contents' stopped working after r342232 in a way
that the actual attribute value isn't used and it work

[VFS] Remove 'ignore-non-existent-contents' attribute for YAML-based VFS.

'ignore-non-existent-contents' stopped working after r342232 in a way
that the actual attribute value isn't used and it works as if it is
always `true`.

Common use case for VFS iteration is iterating through files in umbrella
directories for modules. Ability to detect if some VFS entries point to
non-existing files is nice but non-critical. Instead of adding back
support for `'ignore-non-existent-contents': false` I am removing the
attribute, because such scenario isn't used widely enough and stricter
checks don't provide enough value to justify the maintenance.

Change is done both in LLVM and Clang, corresponding Clang commit is r345212.

rdar://problem/45176119

Reviewers: bruno

Reviewed By: bruno

Subscribers: hiraditya, dexonsmith, sammccall, cfe-commits

Differential Revision: https://reviews.llvm.org/D53228

llvm-svn: 345213

show more ...

Lift VFS from clang to llvm (NFC)

This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang

Lift VFS from clang to llvm (NFC)

This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.

RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html

Differential revision: https://reviews.llvm.org/D52783

llvm-svn: 344140

show more ...