Reapply "lldb: Cache string hash during ConstString pool queries/insertions"

Reverted due to an internally discovered lld crash due to the underlying
StringMap changes, which turned out to be an exi

Reapply "lldb: Cache string hash during ConstString pool queries/insertions"

Reverted due to an internally discovered lld crash due to the underlying
StringMap changes, which turned out to be an existing lld bug that got
tickled by the StringMap changes. That's addressed in
dee8786f70a3d62b639113343fa36ef55bdbad63 so let's have another go with
this change.

Original commit message:
lldb was rehashing the string 3 times (once to determine which StringMap
to use, once to query the StringMap, once to insert) on insertion (twice
on successful lookup).

This patch allows the lldb to benefit from hash improvements in LLVM
(from djbHash to xxh3).

Though further changes would be needed to cache this value to disk - we
shouldn't rely on the StringMap::hash remaining the same in the
future/this value should not be serialized to disk. If we want cache
this value StringMap should take a hashing template parameter to allow
for a fixed hash to be requested.

This reverts commit 5bc1adff69315dcef670e9fcbe04067b5d5963fb.
Effectively reapplying the original 2e197602305be18b963928e6ae024a004a95af6d.

show more ...

Reapply "[ADT][StringMap] Add ability to precompute and reuse the string hash"

Reverted due to an internally discovered lld crash, which turned out to
be an existing lld bug that got tickled by this

Reapply "[ADT][StringMap] Add ability to precompute and reuse the string hash"

Reverted due to an internally discovered lld crash, which turned out to
be an existing lld bug that got tickled by this changes. That's
addressed in dee8786f70a3d62b639113343fa36ef55bdbad63 so let's have
another go with this change.

Original commit message:
Useful for lldb's const string pool, using the hash to determine which
string map to lock and query/insert.

Derived from https://reviews.llvm.org/D122974 by Luboš Luňák

This reverts commit f976719fb2cb23364957e5993f7fc3684ee15391.
Effectively reapplying 67c631d283fc96d652304199cd625be426b98f8e.

show more ...

Revert "[ADT][StringMap] Add ability to precompute and reuse the string hash"

Crash identified internally in lld's use of StringMap in
`compareSections`. Will investigate offline before recommitting

Revert "[ADT][StringMap] Add ability to precompute and reuse the string hash"

Crash identified internally in lld's use of StringMap in
`compareSections`. Will investigate offline before recommitting.

This reverts commit 67c631d283fc96d652304199cd625be426b98f8e.

show more ...

Revert "lldb: Cache string hash during ConstString pool queries/insertions"

Underlying StringMap API for providing a hash has caused some problems
(observed a crash in lld) - so reverting this until

Revert "lldb: Cache string hash during ConstString pool queries/insertions"

Underlying StringMap API for providing a hash has caused some problems
(observed a crash in lld) - so reverting this until I can figure out/fix
what's going on there.

This reverts commit 52ba075571958e2fec8d871ddfa1ef56486b86d3.
This reverts commit 2e197602305be18b963928e6ae024a004a95af6d.

show more ...

Add missing paren

[ADT][StringMap] Add ability to precompute and reuse the string hash

Useful for lldb's const string pool, using the hash to determine which
string map to lock and query/insert.

Derived from https:/

[ADT][StringMap] Add ability to precompute and reuse the string hash

Useful for lldb's const string pool, using the hash to determine which
string map to lock and query/insert.

Derived from https://reviews.llvm.org/D122974 by Luboš Luňák

show more ...

[Support] Change StringMap hash function from xxHash64 to xxh3_64bits

Similar to D142862.

xxh3 is significantly faster than xxh64. Switch to xxh3, as we did for
for lld and llvm-dwarfutil to increa

[Support] Change StringMap hash function from xxHash64 to xxh3_64bits

Similar to D142862.

xxh3 is significantly faster than xxh64. Switch to xxh3, as we did for
for lld and llvm-dwarfutil to increase performance (D154813 D155675).
While I think StringMap is not a bottleneck for most applications, it
seems good to eliminate the slower xxh64.
In addition, according to Erik Desjardins, an artificial benchmark of
Rust with very large constant strings improves by ~3% locally.

I have fixed all found issues (~20) separately, but one is remaining:

* ExecutionEngine/RuntimeDyld/ARM/MachO_ARM_PIC_relocations.s has a
failure due to StringMap iteration order. It now passes
with LLVM_ENABLE_REVERSE_ITERATION=on while failing with
LLVM_ENABLE_REVERSE_ITERATION=off.

Reviewed By: erikdesjardins

Differential Revision: https://reviews.llvm.org/D155781

show more ...

[Support] Implement LLVM_ENABLE_REVERSE_ITERATION for StringMap

ProgrammersManual.html says

> StringMap iteration order, however, is not guaranteed to be deterministic, so any uses which require th

[Support] Implement LLVM_ENABLE_REVERSE_ITERATION for StringMap

ProgrammersManual.html says

> StringMap iteration order, however, is not guaranteed to be deterministic, so any uses which require that should instead use a std::map.

This patch makes -DLLVM_REVERSE_ITERATION=on (currently
-DLLVM_ENABLE_REVERSE_ITERATION=on works as well) shuffle StringMap
iteration order (actually flipping the hash so that elements not in the
same bucket are reversed) to catch violations, similar to D35043 for
DenseMap. This should help change the hash function (e.g., D142862,
D155781).

With a lot of fixes, there are still some violations. This patch
implements the "reverse_iteration" lit feature to skip such tests.
Eventually we should remove this feature.

`ninja check-{llvm,clang,clang-tools}` are clean with
`#define LLVM_ENABLE_REVERSE_ITERATION 1`.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D155789

show more ...

Recommit "[Support] change StringMap hash function from djbHash to xxHash"

This reverts commit 37eb9d13f891f7656f811516e765b929b169afe0.

Test failures have been fixed:

- ubsan failure fixed by 72e

Recommit "[Support] change StringMap hash function from djbHash to xxHash"

This reverts commit 37eb9d13f891f7656f811516e765b929b169afe0.

Test failures have been fixed:

- ubsan failure fixed by 72eac42f21c0f45a27f3eaaff9364cbb5189b9e4
- warn-unsafe-buffer-usage-fixits-local-var-span.cpp fixed by
03cc52dfd1dbb4a59b479da55e87838fb93d2067 (wasn't related)
- test-output-format.ll failure was spurious, build failed at
https://lab.llvm.org/buildbot/#/builders/54/builds/3545 (b4431b2d945b6fc19b1a55ac6ce969a8e06e1e93)
but passed at
https://lab.llvm.org/buildbot/#/builders/54/builds/3546 (5ae99be0377248c74346096dc475af254a3fc799)
which is before my revert
https://github.com/llvm/llvm-project/compare/b4431b2d945b6fc19b1a55ac6ce969a8e06e1e93...5ae99be0377248c74346096dc475af254a3fc799

Original commit message:

Depends on https://reviews.llvm.org/D142861.

Alternative to https://reviews.llvm.org/D137601.

xxHash is much faster than djbHash. This makes a simple Rust test case with a large constant string 10% faster to compile.

Previous attempts at changing this hash function (e.g. https://reviews.llvm.org/D97396) had to be reverted due to breaking tests that depended on iteration order.
No additional tests fail with this patch compared to `main` when running `check-all` with `-DLLVM_ENABLE_PROJECTS="all"` (on a Linux host), so I hope I found everything that needs to be changed.

Differential Revision: https://reviews.llvm.org/D142862

show more ...

Revert "[Support] change StringMap hash function from djbHash to xxHash"

This reverts commit d768b97424f9e1a0aae45440a18b99f21c4027ce.

Causes sanitizer failure: https://lab.llvm.org/buildbot/#/buil

Revert "[Support] change StringMap hash function from djbHash to xxHash"

This reverts commit d768b97424f9e1a0aae45440a18b99f21c4027ce.

Causes sanitizer failure: https://lab.llvm.org/buildbot/#/builders/238/builds/1114

```
/b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/llvm/lib/Support/xxhash.cpp:107:12: runtime error: applying non-zero offset 8 to null pointer
#0 0xaaaab28ec6c8 in llvm::xxHash64(llvm::StringRef) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/llvm/lib/Support/xxhash.cpp:107:12
#1 0xaaaab28cbd38 in llvm::StringMapImpl::LookupBucketFor(llvm::StringRef) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/llvm/lib/Support/StringMap.cpp:87:28
```

Probably causes test failure in `warn-unsafe-buffer-usage-fixits-local-var-span.cpp`: https://lab.llvm.org/buildbot/#/builders/60/builds/10619

Probably causes reverse-iteration test failure in `test-output-format.ll`: https://lab.llvm.org/buildbot/#/builders/54/builds/3545

show more ...

[Support] change StringMap hash function from djbHash to xxHash

Depends on https://reviews.llvm.org/D142861.

Alternative to https://reviews.llvm.org/D137601.

xxHash is much faster than djbHash. Th

[Support] change StringMap hash function from djbHash to xxHash

Depends on https://reviews.llvm.org/D142861.

Alternative to https://reviews.llvm.org/D137601.

xxHash is much faster than djbHash. This makes a simple Rust test case with a large constant string 10% faster to compile.

Previous attempts at changing this hash function (e.g. https://reviews.llvm.org/D97396) had to be reverted due to breaking tests that depended on iteration order.
No additional tests fail with this patch compared to `main` when running `check-all` with `-DLLVM_ENABLE_PROJECTS="all"` (on a Linux host), so I hope I found everything that needs to be changed.

Differential Revision: https://reviews.llvm.org/D142862

show more ...

[NFC][llvm][StringMap]Extract createTable and getHashTable functions and add the inline attribute to the getMinBucketToReserveForEntries function.

1. Extract createTable and getHashTable functions.

[NFC][llvm][StringMap]Extract createTable and getHashTable functions and add the inline attribute to the getMinBucketToReserveForEntries function.

1. Extract createTable and getHashTable functions.
2. Add the inline attribute to the getMinBucketToReserveForEntries function.
3. Remove unnecessary local variable HTSize.

Statements in the following order appear in llvm::StringMapImpl::init and llvm::StringMapImpl::RehashTable, so I extracted this code into a function. getHashTable is for the same reason, it appears in llvm::StringMapImpl::FindKey, llvm::StringMapImpl::LookupBucketFor and llvm::StringMapImpl::RehashTable.

```
auto **Table = static_cast<StringMapEntryBase **>(safe_calloc(
NewNumBuckets + 1, sizeof(StringMapEntryBase **) + sizeof(unsigned)));

// Allocate one extra bucket, set it to look filled so the iterators stop at
// end.
Table[NewNumBuckets] = (StringMapEntryBase *)2;
```

```
unsigned *HashTable = (unsigned *)(TheTable + NumBuckets + 1);
```

Reviewed By: skan, sepavloff

Differential Revision: https://reviews.llvm.org/D121934

show more ...

[llvm][ADT] Remove duplicate code in llvm::StringMapImpl::RehashTable

Remove duplicate code in llvm::StringMapImpl::RehashTable, near StringMap.cpp:229

```
if (!NewTableArray[NewBucket]) {
NewTab

[llvm][ADT] Remove duplicate code in llvm::StringMapImpl::RehashTable

Remove duplicate code in llvm::StringMapImpl::RehashTable, near StringMap.cpp:229

```
if (!NewTableArray[NewBucket]) {
NewTableArray[FullHash & (NewSize - 1)] = Bucket;
NewHashArray[FullHash & (NewSize - 1)] = FullHash;
if (I == BucketNo)
NewBucketNo = NewBucket;
continue;
}
```

Reviewed By: MaskRay, dexonsmith

Differential Revision: https://reviews.llvm.org/D121726

show more ...

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. R

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. Removing unused headers in .cpp files. No impact expected.
3. Removing unused headers in .h files. This removes implicit dependencies and
is generally considered a good thing, but this may break downstream builds.
I've updated llvm, clang, lld, lldb and mlir deps, and included a list of the
modification in the second part of the commit.
4. Replacing header inclusion by forward declaration. This has the same impact
as 3.

Notable changes:

- llvm/Support/TargetParser.h no longer includes llvm/Support/AArch64TargetParser.h nor llvm/Support/ARMTargetParser.h
- llvm/Support/TypeSize.h no longer includes llvm/Support/WithColor.h
- llvm/Support/YAMLTraits.h no longer includes llvm/Support/Regex.h
- llvm/ADT/SmallVector.h no longer includes llvm/Support/MemAlloc.h nor llvm/Support/ErrorHandling.h

You may need to add some of these headers in your compilation units, if needs be.

As an hint to the impact of the cleanup, running

clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Support/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l

before: 8000919 lines
after: 7917500 lines

Reduced dependencies also helps incremental rebuilds and is more ccache
friendly, something not shown by the above metric :-)

Discourse thread on the topic: https://llvm.discourse.group/t/include-what-you-use-include-cleanup/5831

show more ...

Revert "Use the default seed value for djb hash for StringMap"

This reverts commit d84440ec919019ac446241db72cfd905c6ac9dfa.

It breaks (at least) lldb and lld validation

https://lab.llvm.org/build

Revert "Use the default seed value for djb hash for StringMap"

This reverts commit d84440ec919019ac446241db72cfd905c6ac9dfa.

It breaks (at least) lldb and lld validation

https://lab.llvm.org/buildbot/#/builders/68/builds/7837
https://lab.llvm.org/buildbot/#/builders/36/builds/5495

show more ...

Use the default seed value for djb hash for StringMap

See original comment in 560ce2c70fb1fe8e4b9b5e39c54e494a50373ba8
Baiscally the default seed value results in less collision, but changes the
ite

Use the default seed value for djb hash for StringMap

See original comment in 560ce2c70fb1fe8e4b9b5e39c54e494a50373ba8
Baiscally the default seed value results in less collision, but changes the
iteration order, which matters for a few test cases.

Differential Revision: https://reviews.llvm.org/D97396

show more ...

Refactor StringMap.h, splitting StringMapEntry out to its own header.

Summary:
StringMapEntry.h can have lower dependencies, than StringMap.h, which
is useful for public headers that want to expose

Refactor StringMap.h, splitting StringMapEntry out to its own header.

Summary:
StringMapEntry.h can have lower dependencies, than StringMap.h, which
is useful for public headers that want to expose inline methods on
StringMapEntry<> but don't need to expose all of StringMap.h. One
example of this is mlir's Identifier.h, another example is the existing
LLVM StringPool.h.

StringPool also could use a cleanup, I'll deal with that in a follow-on
patch.

Reviewers: rriddle

Subscribers: hiraditya, dexonsmith, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D77963

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Use uniform mechanism for OOM errors handling

This is a recommit of r333506, which was reverted in r333518.
The original commit message is below.

In r325551 many calls of malloc/calloc/realloc were

Use uniform mechanism for OOM errors handling

This is a recommit of r333506, which was reverted in r333518.
The original commit message is below.

In r325551 many calls of malloc/calloc/realloc were replaces with calls of
their safe counterparts defined in the namespace llvm. There functions
generate crash if memory cannot be allocated, such behavior facilitates
handling of out of memory errors on Windows.

If the result of *alloc function were checked for success, the function was
not replaced with the safe variant. In these cases the calling function made
the error handling, like:

T *NewElts = static_cast<T*>(malloc(NewCapacity*sizeof(T)));
if (NewElts == nullptr)
report_bad_alloc_error("Allocation of SmallVector element failed.");

Actually knowledge about the function where OOM occurred is useless. Moreover
having a single entry point for OOM handling is convenient for investigation
of memory problems. This change removes custom OOM errors handling and
replaces them with calls to functions `llvm::safe_*alloc`.

Declarations of `safe_*alloc` are moved to a separate include file, to avoid
cyclic dependency in SmallVector.h

Differential Revision: https://reviews.llvm.org/D47440

llvm-svn: 334344

show more ...

Revert commit 333506

It looks like this commit is responsible for the fail:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-autoconf/builds/24382.

llvm-svn: 333518

Use uniform mechanism for OOM errors handling

This is a recommit of r333390, which was reverted in r333395, because it
caused cyclic dependency when building shared library `LLVMDemangle.so`.
In thi

Use uniform mechanism for OOM errors handling

This is a recommit of r333390, which was reverted in r333395, because it
caused cyclic dependency when building shared library `LLVMDemangle.so`.
In this commit `ItaniumDemangler.cpp` was not changed.

The original commit message is below.

In r325551 many calls of malloc/calloc/realloc were replaces with calls of
their safe counterparts defined in the namespace llvm. There functions
generate crash if memory cannot be allocated, such behavior facilitates
handling of out of memory errors on Windows.

If the result of *alloc function were checked for success, the function was
not replaced with the safe variant. In these cases the calling function made
the error handling, like:

T *NewElts = static_cast<T*>(malloc(NewCapacity*sizeof(T)));
if (NewElts == nullptr)
report_bad_alloc_error("Allocation of SmallVector element failed.");

Actually knowledge about the function where OOM occurred is useless. Moreover
having a single entry point for OOM handling is convenient for investigation
of memory problems. This change removes custom OOM errors handling and
replaces them with calls to functions `llvm::safe_*alloc`.

Declarations of `safe_*alloc` are moved to a separate include file, to avoid
cyclic dependency in SmallVector.h

Differential Revision: https://reviews.llvm.org/D47440

llvm-svn: 333506

show more ...

Reverted commits 333390, 333391 and 333394

Build of shared library LLVMDemangle.so fails due to dependency problem.

llvm-svn: 333395

Use uniform mechanism for OOM errors handling

In r325551 many calls of malloc/calloc/realloc were replaces with calls of
their safe counterparts defined in the namespace llvm. There functions
genera

Use uniform mechanism for OOM errors handling

In r325551 many calls of malloc/calloc/realloc were replaces with calls of
their safe counterparts defined in the namespace llvm. There functions
generate crash if memory cannot be allocated, such behavior facilitates
handling of out of memory errors on Windows.

If the result of *alloc function were checked for success, the function was
not replaced with the safe variant. In these cases the calling function made
the error handling, like:

T *NewElts = static_cast<T*>(malloc(NewCapacity*sizeof(T)));
if (NewElts == nullptr)
report_bad_alloc_error("Allocation of SmallVector element failed.");

Actually knowledge about the function where OOM occurred is useless. Moreover
having a single entry point for OOM handling is convenient for investigation
of memory problems. This change removes custom OOM errors handling and
replaces them with calls to functions `llvm::safe_*alloc`.

Declarations of `safe_*alloc` are moved to a separate include file, to avoid
cyclic dependency in SmallVector.h

Differential Revision: https://reviews.llvm.org/D47440

llvm-svn: 333390

show more ...

Re-land: "[Support] Replace HashString with djbHash."

This patch removes the HashString function from StringExtraces and
replaces its uses with calls to djbHash from DJB.h.

This change is *almost*

Re-land: "[Support] Replace HashString with djbHash."

This patch removes the HashString function from StringExtraces and
replaces its uses with calls to djbHash from DJB.h.

This change is *almost* NFC. While the algorithm is identical, the
djbHash implementation in StringExtras used 0 as its default seed while
the implementation in DJB uses 5381. The latter has been shown to result
in less collisions and improved avalanching and is used by the DWARF
accelerator tables.

Because some test were implicitly relying on the hash order, I've
reverted to using zero as a seed for the following two files:

lld/include/lld/Core/SymbolTable.h
llvm/lib/Support/StringMap.cpp

Differential revision: https://reviews.llvm.org/D43615

llvm-svn: 326091

show more ...

Revert "[Support] Replace HashString with djbHash."

It looks like some of our tests depend on the ordering of hashed values.
I'm reverting my changes while I try to reproduce and fix this locally.

Revert "[Support] Replace HashString with djbHash."

It looks like some of our tests depend on the ordering of hashed values.
I'm reverting my changes while I try to reproduce and fix this locally.

Failing builds:

lab.llvm.org:8011/builders/lld-x86_64-darwin13/builds/18388
lab.llvm.org:8011/builders/clang-cmake-x86_64-sde-avx512-linux/builds/6743
lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/15607

llvm-svn: 326082

show more ...