[SpecialCaseList] Use glob by default (#74809)

https://reviews.llvm.org/D154014 addes glob support and enables it when
`#!special-case-list-v2` is the first line. This patch makes the glob
support

[SpecialCaseList] Use glob by default (#74809)

https://reviews.llvm.org/D154014 addes glob support and enables it when
`#!special-case-list-v2` is the first line. This patch makes the glob
support the default (faster than regex after
https://reviews.llvm.org/D156046) and switches to the deprecated regex
support if `#!special-case-list-v1` is the first line.

I have surveyed many ignore lists. All ignore lists I find only use
basic `*` `.` and don't use regex metacharacters such as `(` and `)`.
(As neither `src:` nor `fun:` benefits from using regex.)
They are unaffected by the transition (with a caution that regex
`src:x/a.pb.*` matches `x/axpbx` but glob `src:x/a.pb.*` doesn't).

There is no deprecating warning. If a user finds
`#!special-case-list-v1`, they shall read that the old syntax is
deprecated.

Link:
https://discourse.llvm.org/t/use-glob-instead-of-regex-for-specialcaselists/71666

show more ...

[Support] Use StringRef::starts_with/ends_with instead of startswith/endswith. NFC.

startswith/endswith wrap starts_with/ends_with and will eventually go away (to more closely match string_view)

[SpecialCaseList] Add option to use Globs instead of Regex to match patterns

Add an option in `SpecialCaseList` to use Globs instead of Regex to match patterns. `GlobPattern` was extended in https:/

[SpecialCaseList] Add option to use Globs instead of Regex to match patterns

Add an option in `SpecialCaseList` to use Globs instead of Regex to match patterns. `GlobPattern` was extended in https://reviews.llvm.org/D153587 to support brace expansions which allows us to use patterns like `*/src/foo.{c,cpp}`. It turns out that most patterns only take advantage of `*` so using Regex was overkill and required lots of escaping in practice. This often led to bugs due to forgetting to escape special characters.

Since this would be a breaking change, we temporarily support Regex by default and use Globs when `#!special-case-list-v2` is the first line in the file. Users should switch to the glob format described in https://llvm.org/doxygen/classllvm_1_1GlobPattern.html. For example, `(abc|def)` should become `{abc,def}`.

See discussion in https://reviews.llvm.org/D152762 and https://discourse.llvm.org/t/use-glob-instead-of-regex-for-specialcaselists/71666.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D154014

show more ...

[SpecialCaseList] Remove TrigramIndex

`TrigramIndex` was added back in https://reviews.llvm.org/D27188 as an optimization to make `SpecialCaseList::match()` faster. I've found that `TrigramIndex` ac

[SpecialCaseList] Remove TrigramIndex

`TrigramIndex` was added back in https://reviews.llvm.org/D27188 as an optimization to make `SpecialCaseList::match()` faster. I've found that `TrigramIndex` actually makes the function slower and it has no functional use, so we can remove it.

I grabbed the list of queries passed to `SpecialCaseList::match()` on a random very large file (`AArch64ISelLowering.cpp`) and measured the runtime to call `match()` on all of them with [this line](https://github.com/llvm/llvm-project/blob/8e1f820bb4eadf5c0704818f6063e0db1006e32d/llvm/lib/Support/SpecialCaseList.cpp#L64) disabled and then enabled.

```
$ hyperfine --warmup 3 'GTEST_FILTER="SpecialCaseListTest.Large" USE_TRIGRAMS=1 build/unittests/Support/SupportTests' 'GTEST_FILTER="SpecialCaseListTest.Large" USE_TRIGRAMS=0 build/unittests/Support/SupportTests'
Benchmark 1: GTEST_FILTER="SpecialCaseListTest.Large" USE_TRIGRAMS=1 build/unittests/Support/SupportTests
Time (mean ± σ): 575.9 ms ± 20.3 ms [User: 573.1 ms, System: 2.7 ms]
Range (min … max): 555.5 ms … 620.0 ms 10 runs

Benchmark 2: GTEST_FILTER="SpecialCaseListTest.Large" USE_TRIGRAMS=0 build/unittests/Support/SupportTests
Time (mean ± σ): 283.4 ms ± 6.7 ms [User: 280.3 ms, System: 3.0 ms]
Range (min … max): 277.0 ms … 294.9 ms 10 runs

Summary
'GTEST_FILTER="SpecialCaseListTest.Large" USE_TRIGRAMS=0 build/unittests/Support/SupportTests' ran
2.03 ± 0.09 times faster than 'GTEST_FILTER="SpecialCaseListTest.Large" USE_TRIGRAMS=1 build/unittests/Support/SupportTests'
```

Using `perf` I found that most of the runtime in `TrigramIndex::isDefinitelyOut()` comes from a division operation that seems to come from `std::unordered_map`: https://github.com/llvm/llvm-project/blob/8e1f820bb4eadf5c0704818f6063e0db1006e32d/llvm/include/llvm/Support/TrigramIndex.h#L62

Removing `TrigramIndex` will make it easier to potentially switch to using `GlobPattern` instead of a full regex for `SpecialCaseList`. See discussion in https://reviews.llvm.org/D152762 for details.

Reviewed By: MaskRay, #sanitizers, vitalybuka

Differential Revision: https://reviews.llvm.org/D153171

show more ...

[llvm] Use *{Set,Map}::contains (NFC)

[llvm] Use = default (NFC)

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. R

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. Removing unused headers in .cpp files. No impact expected.
3. Removing unused headers in .h files. This removes implicit dependencies and
is generally considered a good thing, but this may break downstream builds.
I've updated llvm, clang, lld, lldb and mlir deps, and included a list of the
modification in the second part of the commit.
4. Replacing header inclusion by forward declaration. This has the same impact
as 3.

Notable changes:

- llvm/Support/TargetParser.h no longer includes llvm/Support/AArch64TargetParser.h nor llvm/Support/ARMTargetParser.h
- llvm/Support/TypeSize.h no longer includes llvm/Support/WithColor.h
- llvm/Support/YAMLTraits.h no longer includes llvm/Support/Regex.h
- llvm/ADT/SmallVector.h no longer includes llvm/Support/MemAlloc.h nor llvm/Support/ErrorHandling.h

You may need to add some of these headers in your compilation units, if needs be.

As an hint to the impact of the cleanup, running

clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Support/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l

before: 8000919 lines
after: 7917500 lines

Reduced dependencies also helps incremental rebuilds and is more ccache
friendly, something not shown by the above metric :-)

Discourse thread on the topic: https://llvm.discourse.group/t/include-what-you-use-include-cleanup/5831

show more ...

[llvm] Replace report_fatal_error(std::string) uses with report_fatal_error(Twine)

As described on D111049, we're trying to remove the <string> dependency from error handling and replace uses of rep

[llvm] Replace report_fatal_error(std::string) uses with report_fatal_error(Twine)

As described on D111049, we're trying to remove the <string> dependency from error handling and replace uses of report_fatal_error(const std::string&) with the Twine() variant which can be forward declared.

show more ...

[Support] Qualify auto (NFC)

Identified with readability-qualified-auto.

As part of using inclusive language within the llvm project,
migrate away from the use of blacklist and whitelist.

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

Reland 9f3fdb0d7fab: [Driver] Use VFS to check if sanitizer blacklists exist

With updates to various LLVM tools that use SpecialCastList.

It was tempting to use RealFileSystem as the default, but t

Reland 9f3fdb0d7fab: [Driver] Use VFS to check if sanitizer blacklists exist

With updates to various LLVM tools that use SpecialCastList.

It was tempting to use RealFileSystem as the default, but that makes it
too easy to accidentally forget passing VFS in clang code.

show more ...

Revert "[Driver] Use VFS to check if sanitizer blacklists exist"

This reverts commit ba6f906854263375cff3257d22d241a8a259cf77.
Commit caused compilation errors on llvm tests. Will fix and re-land.

[Driver] Use VFS to check if sanitizer blacklists exist

Summary:
This is a follow-up to 590f279c456bbde632eca8ef89a85c478f15a249, which
moved some of the callers to use VFS.

It turned out more code

[Driver] Use VFS to check if sanitizer blacklists exist

Summary:
This is a follow-up to 590f279c456bbde632eca8ef89a85c478f15a249, which
moved some of the callers to use VFS.

It turned out more code in Driver calls into real filesystem APIs and
also needs an update.

Reviewers: gribozavr2, kadircet

Reviewed By: kadircet

Subscribers: ormris, mgorny, hiraditya, llvm-commits, jkorous, cfe-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D70440

show more ...

[clang] Add VFS support for sanitizers' blacklists

Differential Revision: https://reviews.llvm.org/D69648

[llvm] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of

[llvm] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

llvm-svn: 369013

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Extend SpecialCaseList to allow users to blame matches on entries in the file.

Summary:
Extends SCL functionality to allow users to find the line number in the file the SCL is built from through Spe

Extend SpecialCaseList to allow users to blame matches on entries in the file.

Summary:
Extends SCL functionality to allow users to find the line number in the file the SCL is built from through SpecialCaseList::inSectionBlame(...).

Also removes the need to compile the SCL before use. As the matcher now contains a list of regexes to test against instead of a single regex, the regexes can be individually built on each insertion rather than one large compilation at the end of construction.

This change also fixes a bug where blank lines would cause the parser to become out-of-sync with the line number. An error on line `k` was being reported as being on line `k - num_blank_lines_before_k`.

Note: This change has a cyclical dependency on D39486. Both these changes must be submitted at the same time to avoid a build breakage.

Reviewers: vlad.tsyrklevich

Reviewed By: vlad.tsyrklevich

Subscribers: kcc, pcc, llvm-commits

Differential Revision: https://reviews.llvm.org/D39485

llvm-svn: 317617

show more ...

Check special-case-list regex before insertion.

Summary:
Checks that the supplied regex to SpecialCaseList::Matcher::insert(..) is non-empty.

Reported by OSS-fuzz: https://bugs.chromium.org/p/oss-f

Check special-case-list regex before insertion.

Summary:
Checks that the supplied regex to SpecialCaseList::Matcher::insert(..) is non-empty.

Reported by OSS-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3688

Verified that this fixes the provided assertion failure (built with {asan, fuzzer}):
```
mitchp@mitchp2:~/llvm-build/git-fuzz$ ninja llvm-special-case-list-fuzzer[12/12] Linking CXX executable bin/llvm-special-case-list-fuzzer
mitchp@mitchp2:~/llvm-build/git-fuzz$ bin/llvm-special-case-list-fuzzer ~/Downloads/clusterfuzz-testcase-6748633157337088
INFO: Seed: 1697404507
INFO: Loaded 1 modules (18581 inline 8-bit counters): 18581 [0x9e9f60, 0x9ee7f5),
INFO: Loaded 1 PC tables (18581 PCs): 18581 [0x9ee7f8,0xa37148),
bin/llvm-special-case-list-fuzzer: Running 1 inputs 1 time(s) each.
Running: /usr/local/google/home/mitchp/Downloads/clusterfuzz-testcase-6748633157337088
Executed /usr/local/google/home/mitchp/Downloads/clusterfuzz-testcase-6748633157337088 in 0 ms
***
*** NOTE: fuzzing was not performed, you have only
*** executed the target code on a fixed set of inputs.
***
mitchp@mitchp2:~/llvm-build/git-fuzz$

```

Reviewers: kcc, vsk

Reviewed By: vsk

Subscribers: vsk, llvm-commits, vlad.tsyrklevich

Differential Revision: https://reviews.llvm.org/D39212

llvm-svn: 316537

show more ...

Add section headers to SpecialCaseLists

Summary:
Sanitizer blacklist entries currently apply to all sanitizers--there
is no way to specify that an entry should only apply to a specific
sanitizer. Th

Add section headers to SpecialCaseLists

Summary:
Sanitizer blacklist entries currently apply to all sanitizers--there
is no way to specify that an entry should only apply to a specific
sanitizer. This is important for Control Flow Integrity since there are
several different CFI modes that can be enabled at once. For maximum
security, CFI blacklist entries should be scoped to only the specific
CFI mode(s) that entry applies to.

Adding section headers to SpecialCaseLists allows users to specify more
information about list entries, like sanitizer names or other metadata,
like so:

[section1]
fun:*fun1*
[section2|section3]
fun:*fun23*

The section headers are regular expressions. For backwards compatbility,
blacklist entries entered before a section header are put into the '[*]'
section so that blacklists without sections retain the same behavior.

SpecialCaseList has been modified to also accept a section name when
matching against the blacklist. It has also been modified so the
follow-up change to clang can define a derived class that allows
matching sections by SectionMask instead of by string.

Reviewers: pcc, kcc, eugenis, vsk

Reviewed By: eugenis, vsk

Subscribers: vitalybuka, llvm-commits

Differential Revision: https://reviews.llvm.org/D37924

llvm-svn: 314170

show more ...

Sort the remaining #include lines in include/... and lib/....

I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line

Sort the remaining #include lines in include/... and lib/....

I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line with a #include and let it re-sort things according to the precise
LLVM rules for include ordering baked into clang-format these days.

I've reverted a number of files where the results of sorting includes
isn't healthy. Either places where we have legacy code relying on
particular include ordering (where possible, I'll fix these separately)
or where we have particular formatting around #include lines that
I didn't want to disturb in this patch.

This patch is *entirely* mechanical. If you get merge conflicts or
anything, just ignore the changes in this patch and run clang-format
over your #include lines in the files.

Sorry for any noise here, but it is important to keep these things
stable. I was seeing an increasing number of patches with irrelevant
re-ordering of #include lines because clang-format was used. This patch
at least isolates that churn, makes it easy to skip when resolving
conflicts, and gets us to a clean baseline (again).

llvm-svn: 304787

show more ...

Use trigrams to speed up SpecialCaseList.

Summary:
it's often the case when the rules in the SpecialCaseList
are of the form hel.o*bar. That gives us a chance to build
trigram index to quickly disca

Use trigrams to speed up SpecialCaseList.

Summary:
it's often the case when the rules in the SpecialCaseList
are of the form hel.o*bar. That gives us a chance to build
trigram index to quickly discard 99% of inputs without
running a full regex. A similar idea was used in Google Code Search
as described in the blog post:
https://swtch.com/~rsc/regexp/regexp4.html

The check is defeated, if there's at least one regex
more complicated than that. In this case, all inputs
will go through the regex. That said, the real-world
rules are often simple or can be simplied. That considerably
speeds up compiling Chromium with CFI and UBSan.

As measured on Chromium's content_message_generator.cc:

before, CFI: 44 s
after, CFI: 23 s
after, CFI, no blacklist: 23 s (~1% slower, but 3 runs were unable to show the difference)
after, regular compilation to bitcode: 23 s

Reviewers: pcc

Subscribers: mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D27188

llvm-svn: 288303

show more ...

Apply clang-tidy's 'performance-faster-string-find' check to LLVM.

No functionality change intended.

llvm-svn: 288235

Do a sweep over move ctors and remove those that are identical to the default.

All of these existed because MSVC 2013 was unable to synthesize default
move ctors. We recently dropped support for it

Do a sweep over move ctors and remove those that are identical to the default.

All of these existed because MSVC 2013 was unable to synthesize default
move ctors. We recently dropped support for it so all that error-prone
boilerplate can go.

No functionality change intended.

llvm-svn: 284721

show more ...

Avoid some copies by using const references.

clang-tidy's performance-unnecessary-copy-initialization with some manual
fixes. No functional changes intended.

llvm-svn: 270988