[NFC] Add explicit #include llvm-config.h where its macros are used. (#106621)

Without these explicit includes, removing other headers, who implicitly
include llvm-config.h, may have non-trivial si

[NFC] Add explicit #include llvm-config.h where its macros are used. (#106621)

Without these explicit includes, removing other headers, who implicitly
include llvm-config.h, may have non-trivial side effects.

show more ...

[AIX][PowerPC] Teach the Threading Library About the Number of Physical Cores on AIX (#67683)

The threading library does not recognize AIX and always returns `-1` for
number of physical cores on AI

[AIX][PowerPC] Teach the Threading Library About the Number of Physical Cores on AIX (#67683)

The threading library does not recognize AIX and always returns `-1` for
number of physical cores on AIX. This PR teaches the library to
recognize AIX and obtain the correct value for the number of physical
cores.

show more ...

[NFC][TargetParser] Replace uses of llvm/Support/Host.h

The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC w

[NFC][TargetParser] Replace uses of llvm/Support/Host.h

The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC warning about the fact it is deprecated, because it is used
in `isl` from where it is included by Polly.

show more ...

[NFC][TargetParser] Remove llvm/ADT/Triple.h

I also ran `git clang-format` to get the headers in the right order for
the new location, which has changed the order of other headers in two
files.

[Threading] Ensure Tests Reflect Disabled Threads

D137836 changed what llvm::get_physical_cores returns when threads are
disabled, to bring it inline with the other parts of Threading. It now
return

[Threading] Ensure Tests Reflect Disabled Threads

D137836 changed what llvm::get_physical_cores returns when threads are
disabled, to bring it inline with the other parts of Threading. It now
returns the value for "unknown" when threading is disabled.

This commit updates the tests (which are failing on some platforms), to
also reflect this change.

Differential Revision: https://reviews.llvm.org/D139015

show more ...

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing f

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing functions in `Support/Threading.h`, so I moved it into there.
The function is also renamed to `llvm::get_physical_cores()` to match
the style of threading's functions.

The big change here is that now if you have threading disabled,
`llvm::get_physical_cores()` will return -1, as if it had not been able
to work out the right info. This is due to how Threading.cpp includes
OS-specific code/headers. This seems ok, as if threading is disabled,
LLVM should not need to know the number of physical cores.

Differential Revision: https://reviews.llvm.org/D137836

show more ...

Revert "[Support] Move getHostNumPhysicalCores to Threading.h"

This reverts commit 5577207d6d3e0642ea047a8dfbfcf3ad372a7f25.

This breaks building LLVM on recent macOS. Error messages below:

llvm/l

Revert "[Support] Move getHostNumPhysicalCores to Threading.h"

This reverts commit 5577207d6d3e0642ea047a8dfbfcf3ad372a7f25.

This breaks building LLVM on recent macOS. Error messages below:

llvm/lib/Support/Threading.cpp:190:3: error: use of undeclared
identifier 'sysctlbyname'
sysctlbyname("hw.physicalcpu", &count, &len, NULL, 0);
^

llvm/lib/Support/Threading.cpp:193:13: error: use of undeclared
identifier 'CTL_HW'
nm[0] = CTL_HW;
^

llvm/lib/Support/Threading.cpp:194:13: error: use of undeclared identifier 'HW_AVAILCPU'
nm[1] = HW_AVAILCPU;
^

llvm/lib/Support/Threading.cpp:195:5: error: use of undeclared identifier 'sysctl'
sysctl(nm, 2, &count, &len, NULL, 0);
^

show more ...

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing f

[Support] Move getHostNumPhysicalCores to Threading.h

This change is focussed on simplifying `Support/Host.h` to only do
target detection. In this case, this function is close in usage to
existing functions in `Support/Threading.h`, so I moved it into there.
The function is also renamed to `llvm::get_physical_cores()` to match
the style of threading's functions.

Differential Revision: https://reviews.llvm.org/D137836

show more ...

Support: reduce stack used in default size test.

When the sanitizers aren't enabled they can use more than 1KB of stack, causing
an overflow where there shouldn't be.

Should fix Green Dragon test.

Recommit: Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that all

Recommit: Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to set the new thread's
stack size. On Darwin even the default size is boosted to 8MB to match the main
thread.

It also switches all users of the older C-style `llvm_execute_on_thread` API
family over to `llvm::thread` followed by either a `detach` or `join` call and
removes the old API.

Moved definition of DefaultStackSize into the .cpp file to hopefully
fix the build on some (GCC-6?) machines.

show more ...

Revert "Support: add llvm::thread class that supports specifying stack size."

It's causing build failures because DefaultStackSize isn't defined everywhere
it should be and I need time to investigat

Revert "Support: add llvm::thread class that supports specifying stack size."

It's causing build failures because DefaultStackSize isn't defined everywhere
it should be and I need time to investigate.

show more ...

Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to

Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to set the new thread's
stack size. On Darwin even the default size is boosted to 8MB to match the main
thread.

It also switches all users of the older C-style `llvm_execute_on_thread` API
family over to `llvm::thread` followed by either a `detach` or `join` call and
removes the old API.

show more ...

[Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups

The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, s

[Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups

The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.

== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.

By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.

This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.

== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std::thread::hardware_concurrency() -- which can only return processors from the current "processor group".

== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).

When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.

When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.

Differential Revision: https://reviews.llvm.org/D71775

show more ...

[Support] Fix race in threading test, found by TSan

Reland "[Support] Add a way to run a function on a detached thread""

This reverts commit 7bc7fe6b789d25d48d6dc71d533a411e9e981237.
The immediate callers have been fixed to pass nullopt where appropr

Reland "[Support] Add a way to run a function on a detached thread""

This reverts commit 7bc7fe6b789d25d48d6dc71d533a411e9e981237.
The immediate callers have been fixed to pass nullopt where appropriate.

show more ...

Revert "[Support] Add a way to run a function on a detached thread"

This reverts commit 40668abca4d307e02b33345cfdb7271549ff48d0.
This causes clang tests to fail, as stacksize=0 is being explicitly

Revert "[Support] Add a way to run a function on a detached thread"

This reverts commit 40668abca4d307e02b33345cfdb7271549ff48d0.
This causes clang tests to fail, as stacksize=0 is being explicitly passed and
is no longer a no-op.

show more ...

[Support] Add a way to run a function on a detached thread

This roughly mimics `std::thread(...).detach()` except it allows to
customize the stack size. Required for https://reviews.llvm.org/D50993.

[Support] Add a way to run a function on a detached thread

This roughly mimics `std::thread(...).detach()` except it allows to
customize the stack size. Required for https://reviews.llvm.org/D50993.

I've decided against reusing the existing `llvm_execute_on_thread` because
it's not obvious what to do with the ownership of the passed
function/arguments:

1. If we pass possibly owning functions data to `llvm_execute_on_thread`,
we'll lose the ability to pass small non-owning non-allocating functions
for the joining case (as it's used now). Is it important enough?
2. If we use the non-owning interface in the new use case, we'll force
clients to transfer ownership to the spawned thread manually, but
similar code would still have to exist inside
`llvm_execute_on_thread(_async)` anyway (as we can't just pass the same
non-owning pointer to pthreads and Windows implementations, and would be
forced to wrap it in some structure, and deal with its ownership.

Patch by Dmitry Kozhevnikov!

Differential Revision: https://reviews.llvm.org/D51103

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Rename interface for querying physical hardware concurrency

Based on post-commit review for D25585/r284180, rename
hardware_physical_concurrency to heavyweight_hardware_concurrency,
to better reflec

Rename interface for querying physical hardware concurrency

Based on post-commit review for D25585/r284180, rename
hardware_physical_concurrency to heavyweight_hardware_concurrency,
to better reflect what type of tasks it should be used for and
to enable other systems to map this to something other than the
number of physical cores.

llvm-svn: 284390

show more ...

Add interface for querying physical hardware concurrency

Summary:
This will be used by ThinLTO to set the amount of backend
parallelism, which performs better when restricted to the number
of physic

Add interface for querying physical hardware concurrency

Summary:
This will be used by ThinLTO to set the amount of backend
parallelism, which performs better when restricted to the number
of physical cores (on X86 at least, where getHostNumPhysicalCores is
currently defined). If not available this falls back to
thread::hardware_concurrency.

Note I didn't add to the thread class since that is a typedef to
std::thread where available.

Reviewers: mehdi_amini

Subscribers: beanz, llvm-commits, mgorny

Differential Revision: https://reviews.llvm.org/D25585

llvm-svn: 284180

show more ...