[Support] Remove unused includes (NFC) (#116752)

Identified with misc-include-cleaner.

[NFC] Add explicit #include llvm-config.h where its macros are used. (#106621)

Without these explicit includes, removing other headers, who implicitly
include llvm-config.h, may have non-trivial si

[NFC] Add explicit #include llvm-config.h where its macros are used. (#106621)

Without these explicit includes, removing other headers, who implicitly
include llvm-config.h, may have non-trivial side effects.

show more ...

[llvm] Avoid 'raw_string_ostream::str' (NFC)

Since `raw_string_ostream` doesn't own the string buffer, it is
desirable (in terms of memory safety) for users to directly reference
the string buffer r

[llvm] Avoid 'raw_string_ostream::str' (NFC)

Since `raw_string_ostream` doesn't own the string buffer, it is
desirable (in terms of memory safety) for users to directly reference
the string buffer rather than use `raw_string_ostream::str()`.

Work towards TODO comment to remove `raw_string_ostream::str()`.

show more ...

Use StringRef::find_first_of(char), etc (NFC) (#92841)

[llvm][Support] Add and use errnoAsErrorCode (#84423)

LLVM is inconsistent about how it converts `errno` to `std::error_code`.
This can cause problems because values outside of `std::errc` compare

[llvm][Support] Add and use errnoAsErrorCode (#84423)

LLVM is inconsistent about how it converts `errno` to `std::error_code`.
This can cause problems because values outside of `std::errc` compare
differently if one is system and one is generic on POSIX systems.

This is even more of a problem on Windows where use of the system
category is just wrong, as that is for Windows errors, which have a
completely different mapping than POSIX/generic errors. This patch fixes
one instance of this mistake in `JSONTransport.cpp`.

This patch adds `errnoAsErrorCode()` which makes it so people do not
need to think about this issue in the future. It also cleans up a lot of
usage of `errno` in LLVM and Clang.

show more ...

[llvm][Support] Call clear_error in LockFileManager to avoid report_fatal_error (#83655)

As per the comment in `raw_fd_ostream`'s destructor, you must call
`clear_error()` to prevent a call to `rep

[llvm][Support] Call clear_error in LockFileManager to avoid report_fatal_error (#83655)

As per the comment in `raw_fd_ostream`'s destructor, you must call
`clear_error()` to prevent a call to `report_fatal_error()`. There's not
really a way to test this, but we did encounter it in the wild.

rdar://117347895

show more ...

[llvm][Support] Add ExponentialBackoff helper (#81206)

This provides a simple way to implement exponential backoff using a do
while loop.

Usage example (also see the change to LockFileManager.cp

[llvm][Support] Add ExponentialBackoff helper (#81206)

This provides a simple way to implement exponential backoff using a do
while loop.

Usage example (also see the change to LockFileManager.cpp):
```
ExponentialBackoff Backoff(10s);
do {
if (tryToDoSomething())
return ItWorked;
} while (Backoff.waitForNextAttempt());
return Timeout;
```

Abstracting this out of `LockFileManager` as the module build daemon
will need it.

show more ...

[Support] Use SmallString::operator std::string (NFC)

[Support] llvm::Optional => std::optional

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

[ADT] Don't including None.h (NFC)

These source files no longer use None, so they do not need to include
None.h.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://

[ADT] Don't including None.h (NFC)

These source files no longer use None, so they do not need to include
None.h.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[llvm] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of ma

[llvm] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[libc++] Remove unused macro in __config

That macro was being defined but not used anywhere in libc++, so it
must be safe to remove it.

As a fly-by fix, also remove mentions of this macro in other

[libc++] Remove unused macro in __config

That macro was being defined but not used anywhere in libc++, so it
must be safe to remove it.

As a fly-by fix, also remove mentions of this macro in other places
in LLVM, to make sure they were not depending on the value defined in
libc++.

Differential Revision: https://reviews.llvm.org/D110289

show more ...

Introduce llvm::sys::Process::getProcessId() and adopt it

Differential Revision: https://reviews.llvm.org/D78022

Improve module.pcm lock file performance on machines with high core counts

Summary:
When building a large Xcode project with multiple module dependencies, and mixed Objective-C & Swift, I observed a

Improve module.pcm lock file performance on machines with high core counts

Summary:
When building a large Xcode project with multiple module dependencies, and mixed Objective-C & Swift, I observed a large number of clang processes stalling at zero CPU for 30+ seconds throughout the build. This was especially prevalent on my 18-core iMac Pro.

After some sampling, the major cause appears to be the lock file implementation for precompiled modules in the module cache. When the lock is heavily contended by multiple clang processes, the exponential backoff runs in lockstep, with some of the processes sleeping for 30+ seconds in order to acquire the file lock.

In the attached patch, I implemented a more aggressive polling mechanism that limits the sleep interval to a max of 500ms, and randomizes the wait time. I preserved a limited form of exponential backoff. I also updated the code to use cross-platform timing, thread sleep, and random number capabilities available in C++11.

On iMac Pro (2.3 GHz Intel Xeon W, 18 core):

Xcode 11.1 bundled clang:

502.2 seconds (average of 5 runs)

Custom clang build with LockFileManager patch applied:

276.6 seconds (average of 5 runs)

This is a 1.82x speedup for this use case.

On MacBook Pro (4 core 3.1GHz Intel i7):

Xcode 11.1 bundled clang:

539.4 seconds (average of 2 runs)

Custom clang build with LockFileManager patch applied:

509.5 seconds (average of 2 runs)

As expected, machines with fewer cores benefit less from this change.

```
Call graph:
2992 Thread_393602 DispatchQueue_1: com.apple.main-thread (serial)
2992 start (in libdyld.dylib) + 1 [0x7fff6a1683d5]
2992 main (in clang) + 297 [0x1097a1059]
2992 driver_main(int, char const**) (in clang) + 2803 [0x1097a5513]
2992 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (in clang) + 1608 [0x1097a7cc8]
2992 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (in clang) + 3299 [0x1097dace3]
2992 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (in clang) + 509 [0x1097dcc1d]
2992 clang::FrontendAction::Execute() (in clang) + 42 [0x109818b3a]
2992 clang::ParseAST(clang::Sema&, bool, bool) (in clang) + 185 [0x10981b369]
2992 clang::Parser::ParseFirstTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) (in clang) + 37 [0x10983e9b5]
2992 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) (in clang) + 141 [0x10983ecfd]
2992 clang::Parser::ParseExternalDeclaration(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) (in clang) + 695 [0x10983f3b7]
2992 clang::Parser::ParseObjCAtDirectives(clang::Parser::ParsedAttributesWithRange&) (in clang) + 637 [0x10a9be9bd]
2992 clang::Parser::ParseModuleImport(clang::SourceLocation) (in clang) + 170 [0x10c4841ba]
2992 clang::Parser::ParseModuleName(clang::SourceLocation, llvm::SmallVectorImpl<std::__1::pair<clang::IdentifierInfo*, clang::SourceLocation> >&, bool) (in clang) + 503 [0x10c485267]
2992 clang::Preprocessor::Lex(clang::Token&) (in clang) + 316 [0x1098285cc]
2992 clang::Preprocessor::LexAfterModuleImport(clang::Token&) (in clang) + 690 [0x10cc7af62]
2992 clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::__1::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) (in clang) + 7989 [0x10bba6535]
2992 compileAndLoadModule(clang::CompilerInstance&, clang::SourceLocation, clang::SourceLocation, clang::Module*, llvm::StringRef) (in clang) + 296 [0x10bba8318]
2992 llvm::LockFileManager::waitForUnlock() (in clang) + 91 [0x10b6953ab]
2992 nanosleep (in libsystem_c.dylib) + 199 [0x7fff6a22c914]
2992 __semwait_signal (in libsystem_kernel.dylib) + 10 [0x7fff6a2a0f32]

```

Differential Revision: https://reviews.llvm.org/D69575

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[LockFileManager] Make default waitForUnlock timeout a parameter, NFC

Patch by Xi Ge!

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

[Support] Fix some Wundef warnings

For the _WIN32 macro, it is the definedness that matters rather than
the value. Most uses of the macro already rely on the definedness.
This commit fixes the few

[Support] Fix some Wundef warnings

For the _WIN32 macro, it is the definedness that matters rather than
the value. Most uses of the macro already rely on the definedness.
This commit fixes the few remaining uses that relied on the value.

Differential Revision: https://reviews.llvm.org/D51105

llvm-svn: 340520

show more ...

LTO: Keep file handles open for memory mapped files.

On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping

LTO: Keep file handles open for memory mapped files.

On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping can
sometimes be incorrect. That was what we did when adding an entry to the
ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing
intermittent build failures on Chromium's ThinLTO bots on Windows. More
details are in the associated Chromium bug (crbug.com/786127).

We can prevent this from happening by keeping a handle to the file open while
the mapping is active. So this patch changes the mapped_file_region class to
duplicate the file handle when mapping the file and close it upon unmapping it.

One gotcha is that the file handle that we keep open must not have been
created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system
will prevent other processes from opening the file. We can achieve this
by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead,
we use SetFileInformationByHandle with FileDispositionInfo to manage the
delete-on-close bit. This lets us remove the hack that we used to use to
clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE.

A downside of using SetFileInformationByHandle/FileDispositionInfo as
opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using
CreateFile to open the file while the flag is set, even within the same
process. This doesn't seem to matter for almost every client of TempFile,
except for LockFileManager, which calls sys::fs::create_link to create a
hard link from the lock file, and in the process of doing so tries to open
the file. To prevent this change from breaking LockFileManager I changed it
to stop using TempFile by effectively reverting r318550.

Differential Revision: https://reviews.llvm.org/D48051

llvm-svn: 334630

show more ...

Remove \brief commands from doxygen comments.

We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they ar

Remove \brief commands from doxygen comments.

We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.

Patch produced by

for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done

Differential Revision: https://reviews.llvm.org/D46290

llvm-svn: 331272

show more ...

IWYU for llvm-config.h in llvm, additions.

See r331124 for how I made a list of files missing the include.
I then ran this Python script:

for f in open('filelist.txt'):
f = f.strip()

IWYU for llvm-config.h in llvm, additions.

See r331124 for how I made a list of files missing the include.
I then ran this Python script:

for f in open('filelist.txt'):
f = f.strip()
fl = open(f).readlines()

found = False
for i in xrange(len(fl)):
p = '#include "llvm/'
if not fl[i].startswith(p):
continue
if fl[i][len(p):] > 'Config':
fl.insert(i, '#include "llvm/Config/llvm-config.h"\n')
found = True
break
if not found:
print 'not found', f
else:
open(f, 'w').write(''.join(fl))

and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p`
and tried to fix include ordering and whatnot.

No intended behavior change.

llvm-svn: 331184

show more ...

s/LLVM_ON_WIN32/_WIN32/, llvm

LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of

s/LLVM_ON_WIN32/_WIN32/, llvm

LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of a reinvented one.

See thread "Replacing LLVM_ON_WIN32 with just _WIN32" on llvm-dev and cfe-dev.
No intended behavior change.

This moves over all uses of the macro, but doesn't remove the definition
of it in (llvm-)config.h yet.

llvm-svn: 331127

show more ...

[Support] Replace hand-written scope_exit with make_scope_exit.

No functionality change intended.

llvm-svn: 325460

Use TempFile in the implementation of LockFileManager.

This move some of the complexity over to the lower level TempFile.

It also makes it a bit more explicit where errors are ignored since we
now

Use TempFile in the implementation of LockFileManager.

This move some of the complexity over to the lower level TempFile.

It also makes it a bit more explicit where errors are ignored since we
now have a call to consumeError.

llvm-svn: 318550

show more ...

Simplify and rename variable.

std::error_code can represent success, so we don't need a
Optional<std::error_code>.

Rename the variable to avoid confusion with the type Error.

llvm-svn: 318111