Implement sys::fs::copy_file using the macOS copyfile(3) API
to support APFS clones.

This patch adds a Darwin-specific implementation of
llvm::sys::fs::copy_file() that uses the macOS copyfile(3) AP

Implement sys::fs::copy_file using the macOS copyfile(3) API
to support APFS clones.

This patch adds a Darwin-specific implementation of
llvm::sys::fs::copy_file() that uses the macOS copyfile(3) API to
support APFS copy-on-write clones, which should be faster and much
more space efficient.

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/ToolsandAPIs/ToolsandAPIs.html

Differential Revision: https://reviews.llvm.org/D60802

This reapplies 358628 with an additional bugfix handling the case
where the destination file already exists. (Caught by the clang testsuite).

llvm-svn: 358716

show more ...

Revert Implement sys::fs::copy_file using the macOS copyfile(3) API to support APFS clones.

This reverts r358628 (git commit 91a06bee788262a294527b815354f380d99dfa9b)
while investigating a crash rep

Revert Implement sys::fs::copy_file using the macOS copyfile(3) API to support APFS clones.

This reverts r358628 (git commit 91a06bee788262a294527b815354f380d99dfa9b)
while investigating a crash reproducer bot failure.

llvm-svn: 358634

show more ...

Implement sys::fs::copy_file using the macOS copyfile(3) API
to support APFS clones.

This patch adds a Darwin-specific implementation of
llvm::sys::fs::copy_file() that uses the macOS copyfile(3) AP

Implement sys::fs::copy_file using the macOS copyfile(3) API
to support APFS clones.

This patch adds a Darwin-specific implementation of
llvm::sys::fs::copy_file() that uses the macOS copyfile(3) API to
support APFS copy-on-write clones, which should be faster and much
more space efficient.

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/ToolsandAPIs/ToolsandAPIs.html

Differential Revision: https://reviews.llvm.org/D60802

llvm-svn: 358628

show more ...

[llvm-objcopy] Make .build-id linking atomic

This change makes linking into .build-id atomic and safe to use.
Some users under particular workflows are reporting that this races
more than half the t

[llvm-objcopy] Make .build-id linking atomic

This change makes linking into .build-id atomic and safe to use.
Some users under particular workflows are reporting that this races
more than half the time under particular conditions.

llvm-svn: 356404

show more ...

[Support] Fix TempFile::discard to not leave behind temporary files

Moved the remove of the temporary file to after the close to avoid
remove failures caused by ETXTBSY errors.

This issue was seen

[Support] Fix TempFile::discard to not leave behind temporary files

Moved the remove of the temporary file to after the close to avoid
remove failures caused by ETXTBSY errors.

This issue was seen when FileOutputBuffer falls back to an in memory
buffer due to the inability to mmap the on disk file. This occurred when
running LLD on an Ubuntu VM in VirtualBox on a Windows host attempting
to write the output to a VirtualBox shared folder.

Differential Revision: https://reviews.llvm.org/D57960

llvm-svn: 354017

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

[Support] Remove error return value from one overload of fs::make_absolute

Summary:
The version of make_absolute which accepted a specific directory to use
as the "base" for the computation could ne

[Support] Remove error return value from one overload of fs::make_absolute

Summary:
The version of make_absolute which accepted a specific directory to use
as the "base" for the computation could never fail, even though it
returned a std::error_code. The reason for that seems to be historical
-- the CWD flavour (which can fail due to failure to retrieve CWD) was
there first, and the new version was implemented by extending that.

This removes the error return value from the non-CWD overload and
reimplements the CWD version on top of that. This enables us to remove
some dead code where people were pessimistically trying to handle the
errors returned from this function.

Reviewers: zturner, sammccall

Subscribers: hiraditya, kristina, llvm-commits

Differential Revision: https://reviews.llvm.org/D56599

llvm-svn: 351317

show more ...

Use llvm::copy. NFC

llvm-svn: 347126

Remove dead function user_cache_directory()

It's been unused since it was added almost 3 years ago in
https://reviews.llvm.org/D13801

Motivated by https://reviews.llvm.org/rL342002 since it removes

Remove dead function user_cache_directory()

It's been unused since it was added almost 3 years ago in
https://reviews.llvm.org/D13801

Motivated by https://reviews.llvm.org/rL342002 since it removes one of the
functions keeping a ref to SHGetKnownFolderPath.

Differential Revision: https://reviews.llvm.org/D52184

llvm-svn: 342485

show more ...

[Support] sys::fs::directory_entry includes the file_type.

This is available on most platforms (Linux/Mac/Win/BSD) with no extra syscalls.
On other platforms (e.g. Solaris) we stat() if this informa

[Support] sys::fs::directory_entry includes the file_type.

This is available on most platforms (Linux/Mac/Win/BSD) with no extra syscalls.
On other platforms (e.g. Solaris) we stat() if this information is requested.

This will allow switching clang's VFS to efficiently expose (path, type) when
traversing a directory. Currently it exposes an entire Status, but does so by
calling fs::status() on all platforms.
Almost all callers only need the path, and all callers only need (path, type).

Patch by sammccall (Sam McCall)

Differential Revision: https://reviews.llvm.org/D51918

llvm-svn: 342089

show more ...

[Windows FS] Allow moving files in TempFile::keep

In r338216 / D49860 TempFile::keep was extended to allow keeping across
filesystems. The aim on Windows was to have this happen in rename_internal
u

[Windows FS] Allow moving files in TempFile::keep

In r338216 / D49860 TempFile::keep was extended to allow keeping across
filesystems. The aim on Windows was to have this happen in rename_internal
using the existing system API. However, to fix an issue and preserve the
idea of "renaming" not being a move, put Windows keep-across-filesystem in
TempFile::keep.

Differential Revision: https://reviews.llvm.org/D50048

llvm-svn: 338841

show more ...

[Support] [NFC] change comment about retries in createUniqueEntity

Rewording as requested on D50126 after the change was pushed.

llvm-svn: 338755

[Support] fix TempFile infinite loop and permission denied errors

Summary:
On Windows, TempFile::create() was prone to failing with permission
denied errors when a process created many tempfiles wit

[Support] fix TempFile infinite loop and permission denied errors

Summary:
On Windows, TempFile::create() was prone to failing with permission
denied errors when a process created many tempfiles without providing
a model large enough to accommodate them. There was also a problem
with createUniqueEntity getting into an infinite loop when all names
permitted by the model are in use. This change fixes both of these
problems and adds a unit test for them.

Reviewers: pcc, rnk, zturner

Reviewed By: zturner

Subscribers: inglorion, hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D50126

llvm-svn: 338745

show more ...

[dsymutil] Simplify temporary file handling.

Dsymutil's update functionality was broken on Windows because we tried
to rename a file while we're holding open handles to that file. TempFile
provides

[dsymutil] Simplify temporary file handling.

Dsymutil's update functionality was broken on Windows because we tried
to rename a file while we're holding open handles to that file. TempFile
provides a solution for this through its keep(Twine) method. This patch
changes dsymutil to make use of that functionality.

Differential revision: https://reviews.llvm.org/D49860

llvm-svn: 338216

show more ...

Fix typo in lib/Support/Path.cpp to test commit access

llvm-svn: 336216

Add a flag to FileOutputBuffer that allows modification.

FileOutputBuffer creates a temp file and on commit atomically
renames the temp file to the destination file. Sometimes we
want to modify an

Add a flag to FileOutputBuffer that allows modification.

FileOutputBuffer creates a temp file and on commit atomically
renames the temp file to the destination file. Sometimes we
want to modify an existing file in place, but still have the
atomicity guarantee. To do this we can initialize the contents
of the temp file from the destination file (if it exists), that
way the resulting FileOutputBuffer can have only selective
bytes modified. Committing will then atomically replace the
destination file as desired.

llvm-svn: 335902

show more ...

LTO: Keep file handles open for memory mapped files.

On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping

LTO: Keep file handles open for memory mapped files.

On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping can
sometimes be incorrect. That was what we did when adding an entry to the
ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing
intermittent build failures on Chromium's ThinLTO bots on Windows. More
details are in the associated Chromium bug (crbug.com/786127).

We can prevent this from happening by keeping a handle to the file open while
the mapping is active. So this patch changes the mapped_file_region class to
duplicate the file handle when mapping the file and close it upon unmapping it.

One gotcha is that the file handle that we keep open must not have been
created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system
will prevent other processes from opening the file. We can achieve this
by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead,
we use SetFileInformationByHandle with FileDispositionInfo to manage the
delete-on-close bit. This lets us remove the hack that we used to use to
clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE.

A downside of using SetFileInformationByHandle/FileDispositionInfo as
opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using
CreateFile to open the file while the flag is set, even within the same
process. This doesn't seem to matter for almost every client of TempFile,
except for LockFileManager, which calls sys::fs::create_link to create a
hard link from the lock file, and in the process of doing so tries to open
the file. To prevent this change from breaking LockFileManager I changed it
to stop using TempFile by effectively reverting r318550.

Differential Revision: https://reviews.llvm.org/D48051

llvm-svn: 334630

show more ...

[FileSystem] Split up the OpenFlags enumeration.

This breaks the OpenFlags enumeration into two separate
enumerations: OpenFlags and CreationDisposition. The first
controls the behavior of the API

[FileSystem] Split up the OpenFlags enumeration.

This breaks the OpenFlags enumeration into two separate
enumerations: OpenFlags and CreationDisposition. The first
controls the behavior of the API depending on whether or not
the target file already exists, and is not a flags-based
enum. The second controls more flags-like values.

This yields a more easy to understand API, while also allowing
flags to be passed to the openForRead api, where most of the
values didn't make sense before. This also makes the apis more
testable as it becomes easy to enumerate all the configurations
which make sense, so I've added many new tests to exercise all
the different values.

llvm-svn: 334221

show more ...

[FileSystem] Remove OpenFlags param from several functions.

There was only one place in the entire codebase where a non
default value was being passed, and that place was already hidden
in an implem

[FileSystem] Remove OpenFlags param from several functions.

There was only one place in the entire codebase where a non
default value was being passed, and that place was already hidden
in an implementation file. So we can delete the extra parameter
and all existing clients continue to work as they always have,
while making the interface a bit simpler.

Differential Revision: https://reviews.llvm.org/D47789

llvm-svn: 334046

show more ...

Revert 332508 as it caused problems in the clang test suite.

llvm-svn: 332555

Fix llvm::sys::path::remove_dots() to return "." instead of an empty path.

Differential Revision: https://reviews.llvm.org/D46887

llvm-svn: 332508

[Support/Path] Make handling of paths like "///" consistent

Summary:
Various path functions were not treating paths consisting of slashes
alone consistently. For example, the iterator-based accessor

[Support/Path] Make handling of paths like "///" consistent

Summary:
Various path functions were not treating paths consisting of slashes
alone consistently. For example, the iterator-based accessors decomposed the
path "///" into two elements: "/" and ".". This is not too bad, but it
is different from the behavior specified by posix:
```
A pathname that contains ***at least one non-slash character*** and that
ends with one or more trailing slashes shall be resolved as if a single
dot character ( '.' ) were appended to the pathname.
```
More importantly, this was different from how we treated the same path
in the filename+parent_path functions, which decomposed this path into
"." and "". This was completely wrong as it lost the information that
this was an absolute path which referred to the root directory.

This patch fixes this behavior by making sure all functions treat paths
consisting of (back)slashes alone the same way as "/". I.e., the
iterator-based functions will just report one component ("/"), and the
filename+parent_path will decompose them into "/" and "".

A slightly controversial topic here may be the treatment of "//". Posix
says that paths beginning with "//" may have special meaning and indeed
we have code which parses paths like "//net/foo/bar" specially. However,
as we were already not being consistent in parsing the "//" string
alone, and any special parsing for it would complicate the code further,
I chose to treat it the same way as longer sequences of slashes (which
are guaranteed to be the same as "/").

Another slight change of behavior is in the parsing of paths like
"//net//". Previously the last component of this path was ".". However,
as in our parsing the "//net" part in this path was the same as the
"drive" part in "c:\" and the next slash was the "root directory", it
made sense to treat "//net//" the same way as "//net/" (i.e., not to add
the extra "." component at the end).

Reviewers: zturner, rnk, dblaikie, Bigcheese

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D45942

llvm-svn: 331876

show more ...

IWYU for llvm-config.h in llvm, additions.

See r331124 for how I made a list of files missing the include.
I then ran this Python script:

for f in open('filelist.txt'):
f = f.strip()

IWYU for llvm-config.h in llvm, additions.

See r331124 for how I made a list of files missing the include.
I then ran this Python script:

for f in open('filelist.txt'):
f = f.strip()
fl = open(f).readlines()

found = False
for i in xrange(len(fl)):
p = '#include "llvm/'
if not fl[i].startswith(p):
continue
if fl[i][len(p):] > 'Config':
fl.insert(i, '#include "llvm/Config/llvm-config.h"\n')
found = True
break
if not found:
print 'not found', f
else:
open(f, 'w').write(''.join(fl))

and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p`
and tried to fix include ordering and whatnot.

No intended behavior change.

llvm-svn: 331184

show more ...

s/LLVM_ON_WIN32/_WIN32/, llvm

LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of

s/LLVM_ON_WIN32/_WIN32/, llvm

LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of a reinvented one.

See thread "Replacing LLVM_ON_WIN32 with just _WIN32" on llvm-dev and cfe-dev.
No intended behavior change.

This moves over all uses of the macro, but doesn't remove the definition
of it in (llvm-)config.h yet.

llvm-svn: 331127

show more ...

Changed createTemporaryFile without FD to actually create a file.

Summary:
This commit changes semantics of createUniqueFile and
createTemporaryFile variants that do not return file descriptors.
Pre

Changed createTemporaryFile without FD to actually create a file.

Summary:
This commit changes semantics of createUniqueFile and
createTemporaryFile variants that do not return file descriptors.
Previously they only checked if files exist, therefore being subject
to race conditions. Now they will create an empty file to avoid them.

Functions that do not create a file are now called
getPotentiallyUniqueTempFileName and getPotentiallyUniqueFileName.

Reviewers: klimek, bkramer, krasimir, JDevlieghere, espindola

Reviewed By: klimek

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D36827

llvm-svn: 327851

show more ...