[Support] Allow FileOutputBuffer::create to create an empty file

Size==0 triggers `assert(Size != 0)` in mapped_file_region::init.
I plan to use an empty file in D79339 (llvm-objcopy --dump-section)

[Support] Allow FileOutputBuffer::create to create an empty file

Size==0 triggers `assert(Size != 0)` in mapped_file_region::init.
I plan to use an empty file in D79339 (llvm-objcopy --dump-section).

According to POSIX, "If len is zero, mmap() shall fail and no mapping
shall be established." Just specialize case Size=0 to use
createInMemoryBuffer.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D79338

show more ...

[LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap

Summary:
Add a flag `F_no_mmap` to `FileOutputBuffer` to support
`--[no-]mmap-output-file` in ELF LLD. LLD currently explicitly ignores
this

[LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap

Summary:
Add a flag `F_no_mmap` to `FileOutputBuffer` to support
`--[no-]mmap-output-file` in ELF LLD. LLD currently explicitly ignores
this flag for compatibility with GNU ld and gold.

We need this flag to speed up link time for large binaries in certain
scenarios. When we link some of our larger binaries we find that LLD
takes 50+ GB of memory, which causes memory pressure. The memory
pressure causes the VM to flush dirty pages of the output file to disk.
This is normally okay, since we should be flushing cold pages. However,
when using BtrFS with compression we need to write 128KB at a time when
we flush a page. If any page in that 128KB block is written again, then
it must be flushed a second time, and so on. Since LLD doesn't write
sequentially this causes write amplification. The same 128KB block will
end up being flushed multiple times, causing the linker to many times
more IO than necessary. We've observed 3-5x faster builds with
-no-mmap-output-file when we hit this scenario.

The bad scenario only applies to compressed filesystems, which group
together multiple pages into a single compressed block. I've tested
BtrFS, but the problem will be present for any compressed filesystem
on Linux, since it is caused by the VM.

Silently ignoring --no-mmap-output-file caused a silent regression when
we switched from gold to lld. We pass --no-mmap-output-file to fix this
edge case, but since lld silently ignored the flag we didn't realize it
wasn't being respected.

Benchmark building a 9 GB binary that exposes this edge case. I linked 3
times with --mmap-output-file and 3 times with --no-mmap-output-file and
took the average. The machine has 24 cores @ 2.4 GHz, 112 GB of RAM,
BtrFS mounted with -compress-force=zstd, and an 80% full disk.

| Mode | Time |
|---------|-------|
| mmap | 894 s |
| no mmap | 126 s |

When compression is disabled, BtrFS performs just as well with and
without mmap on this benchmark.

I was unable to reproduce the regression with any binaries in
lld-speed-test.

Reviewed By: ruiu, MaskRay

Differential Revision: https://reviews.llvm.org/D69294

show more ...

Revert "raw_ostream: add operator<< overload for std::error_code"

This reverts commit r368849, because it breaks some bots (e.g.
llvm-clang-x86_64-win-fast).

It turns out this is not as NFC as we h

Revert "raw_ostream: add operator<< overload for std::error_code"

This reverts commit r368849, because it breaks some bots (e.g.
llvm-clang-x86_64-win-fast).

It turns out this is not as NFC as we had hoped, because operator== will
consider two std::error_codes to be distinct even though they both hold
"success" values if they have different categories.

llvm-svn: 368854

show more ...

raw_ostream: add operator<< overload for std::error_code

Summary:
The main motivation for this is unit tests, which contain a large macro
for pretty-printing std::error_code, and this macro is dupli

raw_ostream: add operator<< overload for std::error_code

Summary:
The main motivation for this is unit tests, which contain a large macro
for pretty-printing std::error_code, and this macro is duplicated in
every file that needs to do this. However, the functionality may be
useful elsewhere too.

In this patch I have reimplemented the existing ASSERT_NO_ERROR macros
to reuse the new functionality, but I have kept the macro (as a
one-liner) as it is slightly more readable than ASSERT_EQ(...,
std::error_code()).

Reviewers: sammccall, ilya-biryukov

Subscribers: zturner, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D65643

llvm-svn: 368849

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Remove F_modify flag from FileOutputBuffer.

This code is dead. There is no use of the feature in the entire LLVM codebase.

Differential Revision: https://reviews.llvm.org/D56939

llvm-svn: 351613

[SupportTests] Silence -Wsign-compare warnings

llvm-svn: 335921

Add a flag to FileOutputBuffer that allows modification.

FileOutputBuffer creates a temp file and on commit atomically
renames the temp file to the destination file. Sometimes we
want to modify an

Add a flag to FileOutputBuffer that allows modification.

FileOutputBuffer creates a temp file and on commit atomically
renames the temp file to the destination file. Sometimes we
want to modify an existing file in place, but still have the
atomicity guarantee. To do this we can initialize the contents
of the temp file from the destination file (if it exists), that
way the resulting FileOutputBuffer can have only selective
bytes modified. Committing will then atomically replace the
destination file as desired.

llvm-svn: 335902

show more ...

Convert FileOutputBuffer::commit to Error.

llvm-svn: 317656

Update unittest too.

llvm-svn: 317651

Untabify.

llvm-svn: 311875

Re-sort #include lines for unittests. This uses a slightly modified
clang-format (https://reviews.llvm.org/D33932) to keep primary headers
at the top and handle new utility headers like 'gmock' consi

Re-sort #include lines for unittests. This uses a slightly modified
clang-format (https://reviews.llvm.org/D33932) to keep primary headers
at the top and handle new utility headers like 'gmock' consistently with
other utility headers.

No other change was made. I did no manual edits, all of this is
clang-format.

This should allow other changes to have more clear and focused diffs,
and is especially motivated by moving some headers into more focused
libraries.

llvm-svn: 304786

show more ...

Fix a real temp file leak in FileOutputBuffer

If we failed to commit the buffer but did not die to a signal, the temp
file would remain on disk on Windows. Having an open file mapping and
file handl

Fix a real temp file leak in FileOutputBuffer

If we failed to commit the buffer but did not die to a signal, the temp
file would remain on disk on Windows. Having an open file mapping and
file handle prevents the file from being deleted. I am choosing not to
add an assertion of success on the temp file removal, since virus
scanners and other environmental things can often cause removal to fail
in real world tools.

Also fix more temp file leaks in unit tests.

llvm-svn: 280445

show more ...

Return ErrorOr from FileOutputBuffer::create. NFC.

llvm-svn: 244848

[cleanup] Re-sort all the #include lines in LLVM using
utils/sort_includes.py.

I clearly haven't done this in a while, so more changed than usual. This
even uncovered a missing include from the Inst

[cleanup] Re-sort all the #include lines in LLVM using
utils/sort_includes.py.

I clearly haven't done this in a while, so more changed than usual. This
even uncovered a missing include from the InstrProf library that I've
added. No functionality changed here, just mechanical cleanup of the
include order.

llvm-svn: 225974

show more ...

Remove unused feature. NFC.

llvm-svn: 224135

Misc cleanups to the FileSytem api.

The main difference is the removal of

std::error_code exists(const Twine &path, bool &result);

It was an horribly redundant interface since a file not existing

Misc cleanups to the FileSytem api.

The main difference is the removal of

std::error_code exists(const Twine &path, bool &result);

It was an horribly redundant interface since a file not existing is also a valid
error_code. Now we have an access function that returns just an error_code. This
is the only function that has to be implemented for Unix and Windows. The
functions can_write, exists and can_execute an now just wrappers.

One still has to be very careful using these function to avoid introducing
race conditions (Time of check to time of use).

llvm-svn: 217625

show more ...

Remove the last uses of 'using std::error_code'

This finishes the transition to std::error_code.

llvm-svn: 210877

Don't use 'using std::error_code' in include/llvm.

This should make sure that most new uses use the std prefix.

llvm-svn: 210835

Remove unused has_magic.

This will allow inlining get_magic, which should in turn fix one of the mingw
build problems after the switch to std::error_code.

llvm-svn: 210712

Use std::unique_ptr instead of OwningPtr in the MemoryBuffer unittests.

llvm-svn: 209102

Fix known typos

Sweep the codebase for common typos. Includes some changes to visible function
names that were misspelt.

llvm-svn: 200018

Remove remove_all. A compiler has no need for recursively deleting a directory.

llvm-svn: 198955

Add a convenience createUniqueDirectory function.

There are a few valid situation where we care about the structure inside a
directory, but not about the directory itself. A simple example is for un

Add a convenience createUniqueDirectory function.

There are a few valid situation where we care about the structure inside a
directory, but not about the directory itself. A simple example is for unit
testing directory traversal.

PathV1 had a function like this, add one to V2 and port existing users of the
created temp file and delete it hack to using it.

llvm-svn: 185059

show more ...

Move PathV2.h to Path.h

Most clients have already been moved from Path V1 to V2. The ones using V1
now include PathV1.h explicitly.

llvm-svn: 183801