[Support] Remove unused includes (NFC) (#116752)

Identified with misc-include-cleaner.

Define llvm::thread::DefaultStackSize to 4 megabytes on AIX

Link time thinLTO spawns pthreads to parallelize optimization and
codegen of the input bitcode files. On AIX, the default pthread
stack si

Define llvm::thread::DefaultStackSize to 4 megabytes on AIX

Link time thinLTO spawns pthreads to parallelize optimization and
codegen of the input bitcode files. On AIX, the default pthread
stack size limit is ~192k for 64-bit programs; insufficient for a
normal LLVM compilation.

Reviewed By: ZarkoCA, MaskRay

Differential Revision: https://reviews.llvm.org/D155731

show more ...

Include <optional> instead of "llvm/ADT/Optional.h"

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getv

Include <optional> instead of "llvm/ADT/Optional.h"

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[Support] llvm::Optional => std::optional

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

Threading: Convert Optional to std::optional

dthorn: Relanding after fix-forward.

Revert "Threading: Convert Optional to std::optional"

This reverts commit 5e50b8089aee249d77542ea858d956568ec6581f.

This commit breaks the build for BOLT:
bolt/lib/Profile/DataAggregator.cpp:264:

Revert "Threading: Convert Optional to std::optional"

This reverts commit 5e50b8089aee249d77542ea858d956568ec6581f.

This commit breaks the build for BOLT:
bolt/lib/Profile/DataAggregator.cpp:264:66: error: no viable
conversion from 'Optional<StringRef>[3]' to
'ArrayRef<std::optional<StringRef>>'

show more ...

Threading: Convert Optional to std::optional

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing f

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing functions in `Support/Threading.h`, so I moved it into there.
The function is also renamed to `llvm::get_physical_cores()` to match
the style of threading's functions.

The big change here is that now if you have threading disabled,
`llvm::get_physical_cores()` will return -1, as if it had not been able
to work out the right info. This is due to how Threading.cpp includes
OS-specific code/headers. This seems ok, as if threading is disabled,
LLVM should not need to know the number of physical cores.

Differential Revision: https://reviews.llvm.org/D137836

show more ...

Revert "[Support] Move getHostNumPhysicalCores to Threading.h"

This reverts commit 5577207d6d3e0642ea047a8dfbfcf3ad372a7f25.

This breaks building LLVM on recent macOS. Error messages below:

llvm/l

Revert "[Support] Move getHostNumPhysicalCores to Threading.h"

This reverts commit 5577207d6d3e0642ea047a8dfbfcf3ad372a7f25.

This breaks building LLVM on recent macOS. Error messages below:

llvm/lib/Support/Threading.cpp:190:3: error: use of undeclared
identifier 'sysctlbyname'
sysctlbyname("hw.physicalcpu", &count, &len, NULL, 0);
^

llvm/lib/Support/Threading.cpp:193:13: error: use of undeclared
identifier 'CTL_HW'
nm[0] = CTL_HW;
^

llvm/lib/Support/Threading.cpp:194:13: error: use of undeclared identifier 'HW_AVAILCPU'
nm[1] = HW_AVAILCPU;
^

llvm/lib/Support/Threading.cpp:195:5: error: use of undeclared identifier 'sysctl'
sysctl(nm, 2, &count, &len, NULL, 0);
^

show more ...

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing f

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing functions in `Support/Threading.h`, so I moved it into there.
The function is also renamed to `llvm::get_physical_cores()` to match
the style of threading's functions.

Differential Revision: https://reviews.llvm.org/D137836

show more ...

Host: Internalize computeHostNumPhysicalCores/computeHostNumHardwareThreads

Windows computeHostNumPhysicalCores is defined by Threading.cpp. Leave it
unchanged.

Revert "Host: Internalize computeHostNumPhysicalCores/computeHostNumHardwareThreads"

This reverts commit 9969ceb36b440eaafa17c486f29a69c7a7da3b3b.

On Windows:

lld-link: error: undefined symbol: in

Revert "Host: Internalize computeHostNumPhysicalCores/computeHostNumHardwareThreads"

This reverts commit 9969ceb36b440eaafa17c486f29a69c7a7da3b3b.

On Windows:

lld-link: error: undefined symbol: int __cdecl computeHostNumPhysicalCores(void)
>>> referenced by LLVMSupport.lib(Support.Host.obj):(int __cdecl llvm::sys::getHostNumPhysicalCores(void))

show more ...

Host: Internalize computeHostNumPhysicalCores/computeHostNumHardwareThreads

Remove redundant initialization of Optional (NFC)

[Support] move llvm::llvm_is_multithread to header, NFC

This allow optimization without LTO. Also remove some useless else-ifs.
Signed-off-by: Jun Zhang <jun@junz.org>

Differential Revision: https:

[Support] move llvm::llvm_is_multithread to header, NFC

This allow optimization without LTO. Also remove some useless else-ifs.
Signed-off-by: Jun Zhang <jun@junz.org>

Differential Revision: https://reviews.llvm.org/D131313

show more ...

[Support] reorder Threading includes to avoid conflict with FreeBSD headers

FreeBSD's condvar.h (included by user.h in Threading.inc) uses a "struct
thread" that conflicts with llvm::thread if both

[Support] reorder Threading includes to avoid conflict with FreeBSD headers

FreeBSD's condvar.h (included by user.h in Threading.inc) uses a "struct
thread" that conflicts with llvm::thread if both are visible when it's
included.

So this moves our #include after the FreeBSD code.

show more ...

Recommit: Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that all

Recommit: Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to set the new thread's
stack size. On Darwin even the default size is boosted to 8MB to match the main
thread.

It also switches all users of the older C-style `llvm_execute_on_thread` API
family over to `llvm::thread` followed by either a `detach` or `join` call and
removes the old API.

Moved definition of DefaultStackSize into the .cpp file to hopefully
fix the build on some (GCC-6?) machines.

show more ...

Revert "Support: add llvm::thread class that supports specifying stack size."

It's causing build failures because DefaultStackSize isn't defined everywhere
it should be and I need time to investigat

Revert "Support: add llvm::thread class that supports specifying stack size."

It's causing build failures because DefaultStackSize isn't defined everywhere
it should be and I need time to investigate.

show more ...

Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to

Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to set the new thread's
stack size. On Darwin even the default size is boosted to 8MB to match the main
thread.

It also switches all users of the older C-style `llvm_execute_on_thread` API
family over to `llvm::thread` followed by either a `detach` or `join` call and
removes the old API.

show more ...

[llvm-cov] Prevent llvm-cov from using too many threads

As reported here: https://reviews.llvm.org/D75153#1987272

Before, each instance of llvm-cov was creating one thread per hardware core, which

[llvm-cov] Prevent llvm-cov from using too many threads

As reported here: https://reviews.llvm.org/D75153#1987272

Before, each instance of llvm-cov was creating one thread per hardware core, which wasn't needed probably because the number of inputs were small. This was probably causing a thread rlimit issue on large core count systems.

After this patch, the previous behavior is restored (to what was before rG8404aeb5):

If --num-threads is not specified, we create one thread per input, up to num.cores.
When specified, --num-threads indicates any number of threads, with no upper limit.

Differential Revision: https://reviews.llvm.org/D78408

show more ...

After 09158252f777c2e2f06a86b154c44abcbcf9bb74, fix build when -DLLVM_ENABLE_THREADS=OFF

Tested on Linux with Clang 9, and on Windows with Visual Studio 2019 16.5.1 with -DLLVM_ENABLE_THREADS=ON and

After 09158252f777c2e2f06a86b154c44abcbcf9bb74, fix build when -DLLVM_ENABLE_THREADS=OFF

Tested on Linux with Clang 9, and on Windows with Visual Studio 2019 16.5.1 with -DLLVM_ENABLE_THREADS=ON and OFF.

show more ...

[ThinLTO] Allow usage of all hardware threads in the system

Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was all

[ThinLTO] Allow usage of all hardware threads in the system

Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was allowed, instructed by usage of llvm::heavyweight_hardware_concurrency() in the ThinLTO code. Any number passed to the LLD flag /opt:lldltojobs=..., or any other ThinLTO-specific flag, was previously interpreted in the context of llvm::heavyweight_hardware_concurrency(), which means SMT disabled.

One can now say in LLD:
/opt:lldltojobs=0 -- Use one std::thread / hardware core in the system (no SMT). Default value if flag not specified.
/opt:lldltojobs=N -- Limit usage to N threads, regardless of usage of heavyweight_hardware_concurrency().
/opt:lldltojobs=all -- Use all hardware threads in the system. Equivalent to /opt:lldltojobs=$(nproc) on Linux and /opt:lldltojobs=%NUMBER_OF_PROCESSORS% on Windows. When an affinity mask is set for the process, threads will be created only for the cores selected by the mask.

When N > number-of-hardware-threads-in-the-system, the threads in the thread pool will be dispatched equally on all CPU sockets (tested only on Windows).
When N <= number-of-hardware-threads-on-a-CPU-socket, the threads will remain on the CPU socket where the process started (only on Windows).

Differential Revision: https://reviews.llvm.org/D75153

show more ...

[Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups

The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, s

[Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups

The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.

== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.

By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.

This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.

== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std::thread::hardware_concurrency() -- which can only return processors from the current "processor group".

== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).

When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.

When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.

Differential Revision: https://reviews.llvm.org/D71775

show more ...

Reland "[Support] Add a way to run a function on a detached thread""

This reverts commit 7bc7fe6b789d25d48d6dc71d533a411e9e981237.
The immediate callers have been fixed to pass nullopt where appropr

Reland "[Support] Add a way to run a function on a detached thread""

This reverts commit 7bc7fe6b789d25d48d6dc71d533a411e9e981237.
The immediate callers have been fixed to pass nullopt where appropriate.

show more ...

Revert "[Support] Add a way to run a function on a detached thread"

This reverts commit 40668abca4d307e02b33345cfdb7271549ff48d0.
This causes clang tests to fail, as stacksize=0 is being explicitly

Revert "[Support] Add a way to run a function on a detached thread"

This reverts commit 40668abca4d307e02b33345cfdb7271549ff48d0.
This causes clang tests to fail, as stacksize=0 is being explicitly passed and
is no longer a no-op.

show more ...