[ASan][libc++] std::basic_string annotations (#72677)

This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, on

[ASan][libc++] std::basic_string annotations (#72677)

This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, only long strings with the default allocator will be
annotated. Short strings (_SSO - short string optimization_) and strings
with non-default allocators will be annotated in the near future, with
separate commits dedicated to enabling them. The process will be similar
to the workflow employed for enabling annotations in `std::deque`.

**Please note**: these annotations function effectively only when libc++
and libc++abi dylibs are instrumented (with ASan). This aligns with the
prevailing behavior of Memory Sanitizer.

To avoid breaking everything, this commit also appends
`_LIBCPP_INSTRUMENTED_WITH_ASAN` to `__config_site` whenever libc++ is
compiled with ASan. If this macro is not defined, string annotations are
not enabled. However, linking a binary that does **not** annotate
strings with a dynamic library that annotates strings, is not permitted.

Originally proposed here: https://reviews.llvm.org/D132769

Related patches on Phabricator:
- Turning on annotations for short strings:
https://reviews.llvm.org/D147680
- Turning on annotations for all allocators:
https://reviews.llvm.org/D146214

This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.

The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.

This Pull Request introduces basic annotations for `std::basic_string`.
Long strings exhibit structural similarities to `std::vector` and will
be annotated accordingly. Short strings are already implemented, but
will be turned on separately in a forthcoming commit. Look at [a
comment](https://github.com/llvm/llvm-project/pull/72677#issuecomment-1850554465)
below to read about SSO issues at current moment.

Due to the functionality introduced in
[D132522](https://github.com/llvm/llvm-project/commit/dd1b7b797a116eed588fd752fbe61d34deeb24e4),
the `__sanitizer_annotate_contiguous_container` function now offers
compatibility with all allocators. However, enabling this support will
be done in a subsequent commit. For the time being, only strings with
the default allocator will be annotated.

If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com

show more ...

[libc++] Refactor string unit tests to ease addition of new allocators

While doing this, I also found a few tests that were either clearly
incorrect (e.g. testing the wrong function) or that lacked

[libc++] Refactor string unit tests to ease addition of new allocators

While doing this, I also found a few tests that were either clearly
incorrect (e.g. testing the wrong function) or that lacked basic test
coverage like testing std::string itself (e.g. the test was only checking
std::basic_string with a custom allocator). In these cases, I did a few
conservative drive-by changes.

Differential Revision: https://reviews.llvm.org/D140550
Co-authored-by: Brendan Emery <brendan.emery@esrlabs.com>

show more ...

[libc++] Apply clang formatting to all string unit tests

This applies clang-format to the std::string unit tests in preparation
for landing https://reviews.llvm.org/D140550.

Differential Revision:

[libc++] Apply clang formatting to all string unit tests

This applies clang-format to the std::string unit tests in preparation
for landing https://reviews.llvm.org/D140550.

Differential Revision: https://reviews.llvm.org/D140612

show more ...

[libcxx][AIX] Switch build compiler to clang

This patch switches the build compiler for AIX from ibm-clang to clang. ibm-clang++_r has `-pthread` by default, but clang for AIX doesn't, so `-pthread`

[libcxx][AIX] Switch build compiler to clang

This patch switches the build compiler for AIX from ibm-clang to clang. ibm-clang++_r has `-pthread` by default, but clang for AIX doesn't, so `-pthread` had to be added to the test config. A bunch of tests now pass, so the `XFAIL` was removed. This patch also switch the build to use the visibility support available in clang-15 to control symbols exported by the shared library (AIX traditionally uses explicit export lists for this purpose).

Reviewed By: #libc, #libc_abi, daltenty, #libunwind, ldionne

Differential Revision: https://reviews.llvm.org/D127470

show more ...

[libc++] Implement P0980R1 (constexpr std::string)

Reviewed By: #libc, ldionne

Spies: daltenty, sdasgup3, ldionne, arichardson, MTC, ChuanqiXu, mehdi_amini, shauheen, antiagainst, nicolasvasilache,

[libc++] Implement P0980R1 (constexpr std::string)

Reviewed By: #libc, ldionne

Spies: daltenty, sdasgup3, ldionne, arichardson, MTC, ChuanqiXu, mehdi_amini, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, aartbik, liufengdb, stephenneuendorffer, Joonsoo, grosul1, Kayjukh, jurahul, msifontes, tatianashp, rdzhabarov, teijeong, cota, dcaballe, Chia-hungDuan, wrengr, wenzhicui, arphaman, Mordante, miscco, Quuxplusone, smeenai, libcxx-commits

Differential Revision: https://reviews.llvm.org/D110598

show more ...

[libc++] Prepare string.{access, capacity, cons} tests for constexpr

Reviewed By: ldionne, #libc

Spies: libcxx-commits, arphaman

Differential Revision: https://reviews.llvm.org/D119123

Support tests in freestanding

Summary:
Freestanding is *weird*. The standard allows it to differ in a bunch of odd
manners from regular C++, and the committee would like to improve that
situation. I

Support tests in freestanding

Summary:
Freestanding is *weird*. The standard allows it to differ in a bunch of odd
manners from regular C++, and the committee would like to improve that
situation. I'd like to make libc++ behave better with what freestanding should
be, so that it can be a tool we use in improving the standard. To do that we
need to try stuff out, both with "freestanding the language mode" and
"freestanding the library subset".

Let's start with the super basic: run the libc++ tests in freestanding, using
clang as the compiler, and see what works. The easiest hack to do this:

In utils/libcxx/test/config.py add:

self.cxx.compile_flags += ['-ffreestanding']

Run the tests and they all fail.

Why? Because in freestanding `main` isn't special. This "not special" property
has two effects: main doesn't get mangled, and main isn't allowed to omit its
`return` statement. The first means main gets mangled and the linker can't
create a valid executable for us to test. The second means we spew out warnings
(ew) and the compiler doesn't insert the `return` we omitted, and main just
falls of the end and does whatever undefined behavior (if you're luck, ud2
leading to non-zero return code).

Let's start my work with the basics. This patch changes all libc++ tests to
declare `main` as `int main(int, char**` so it mangles consistently (enabling us
to declare another `extern "C"` main for freestanding which calls the mangled
one), and adds `return 0;` to all places where it was missing. This touches 6124
files, and I apologize.

The former was done with The Magic Of Sed.

The later was done with a (not quite correct but decent) clang tool:

https://gist.github.com/jfbastien/793819ff360baa845483dde81170feed

This works for most tests, though I did have to adjust a few places when e.g.
the test runs with `-x c`, macros are used for main (such as for the filesystem
tests), etc.

Once this is in we can create a freestanding bot which will prevent further
regressions. After that, we can start the real work of supporting C++
freestanding fairly well in libc++.

<rdar://problem/47754795>

Reviewers: ldionne, mclow.lists, EricWF

Subscribers: christof, jkorous, dexonsmith, arphaman, miyuki, libcxx-commits

Differential Revision: https://reviews.llvm.org/D57624

llvm-svn: 353086

show more ...

Update more file headers across all of the LLVM projects in the monorepo
to reflect the new license. These used slightly different spellings that
defeated my regular expressions.

We understand that

Update more file headers across all of the LLVM projects in the monorepo
to reflect the new license. These used slightly different spellings that
defeated my regular expressions.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351648

show more ...

[libcxx] [test] Fix Clang -Wunused-local-typedef, part 2/3.

These typedefs were completely unused.

Fixes D29136.

llvm-svn: 294155

[libcxx] [test] Fix MSVC warning C4244 "conversion from 'X' to 'Y', possible loss of data", part 7/7.

test/std/input.output/iostream.format/input.streams/istream.unformatted/get.pass.cpp
Add static_

[libcxx] [test] Fix MSVC warning C4244 "conversion from 'X' to 'Y', possible loss of data", part 7/7.

test/std/input.output/iostream.format/input.streams/istream.unformatted/get.pass.cpp
Add static_cast<char> because basic_istream::get() returns int_type (N4606 27.7.2.3 [istream.unformatted]/4).

test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minus1.pass.cpp
Add static_cast<char> because toupper() returns int (C11 7.4.2.2/1).

test/std/iterators/stream.iterators/ostream.iterator/ostream.iterator.ops/assign_t.pass.cpp
This test is intentionally writing doubles to ostream_iterator<int>.
It's silencing -Wliteral-conversion for Clang, so I'm adding C4244 silencing for MSVC.

test/std/language.support/support.limits/limits/numeric.limits.members/infinity.pass.cpp
Given `extern float zero;`, the expression `1./zero` has type double, which emits a truncation warning
when being passed to test<float>() taking float. The fix is to say `1.f/zero` which has type float.

test/std/numerics/complex.number/cmplx.over/arg.pass.cpp
test/std/numerics/complex.number/cmplx.over/norm.pass.cpp
These tests were constructing std::complex<double>(x, 0), emitting truncation warnings when x is long long.
Saying static_cast<double>(x) avoids this.

test/std/numerics/rand/rand.eng/rand.eng.lcong/seed_result_type.pass.cpp
This was using `int s` to construct and seed a linear_congruential_engine<T, stuff>, where T is
unsigned short/unsigned int/unsigned long/unsigned long long. That emits a truncation warning in the
unsigned short case. Because the range [0, 20) is tiny and we aren't doing anything else with the index,
we can just iterate with `T s`.

test/std/re/re.traits/value.pass.cpp
regex_traits<wchar_t>::value()'s first parameter is wchar_t (N4606 28.7 [re.traits]/13). This loop is
using int to iterate through ['g', 0xFFFF), emitting a truncation warning from int to wchar_t
(which is 16-bit for some of us). Because the bound is exclusive, we can just iterate with wchar_t.

test/std/strings/basic.string/string.cons/size_char_alloc.pass.cpp
This test is a little strange. It's trying to verify that basic_string's (InIt, InIt) range constructor
isn't confused by "N copies of C" when N and C have the same integral type. To do this, it was
testing (100, 65), but that eventually emits truncation warnings from int to char. There's a simple way
to avoid this - passing (static_cast<char>(100), static_cast<char>(65)) also exercises the disambiguation.
(And 100 is representable even when char has a signed range.)

test/std/strings/string.view/string.view.hash/string_view.pass.cpp
Add static_cast<char_type> because `'0' + i` has type int.

test/std/utilities/function.objects/bind/func.bind/func.bind.bind/nested.pass.cpp
What's more horrible than nested bind()? pow() overloads! This operator()(T a, T b) was assuming that
std::pow(a, b) can be returned as T. (In this case, T is int.) However, N4606 26.9.1 [cmath.syn]/2
says that pow(int, int) returns double, so this was truncating double to int.
Adding static_cast<T> silences this.

test/std/utilities/function.objects/unord.hash/integral.pass.cpp
This was iterating `for (int i = 0; i <= 5; ++i)` and constructing `T t(i);` but that's truncating
when T is short. (And super truncating when T is bool.) Adding static_cast<T> silences this.

test/std/utilities/utility/exchange/exchange.pass.cpp
First, this was exchanging 67.2 into an int, but that's inherently truncating.
Changing this to static_cast<short>(67) avoids the truncation while preserving the
"what if T and U are different" test coverage.
Second, this was exchanging {} with the explicit type float into an int, and that's also
inherently truncating. Specifying short is just as good.

test/std/utilities/utility/pairs/pairs.spec/make_pair.pass.cpp
Add static_cast<short>. Note that this affects template argument deduction for make_pair(),
better fulfilling the test's intent. For example, this was saying
`typedef std::pair<int, short> P1; P1 p1 = std::make_pair(3, 4);` but that was asking
make_pair() to return pair<int, int>, which was then being converted to pair<int, short>.
(pair's converting constructors are tested elsewhere.)
Now, std::make_pair(3, static_cast<short>(4)) actually returns pair<int, short>.
(There's still a conversion from pair<nullptr_t, short> to pair<unique_ptr<int>, short>.)

Fixes D27544.

llvm-svn: 289111

show more ...

[libcxx] [test] D27022: Fix MSVC warning C4389 "signed/unsigned mismatch", part 9/12.

Add static_cast<std::size_t> to more comparisons. (Performed manually, unlike part 8/12.)

Also, include <cstdde

[libcxx] [test] D27022: Fix MSVC warning C4389 "signed/unsigned mismatch", part 9/12.

Add static_cast<std::size_t> to more comparisons. (Performed manually, unlike part 8/12.)

Also, include <cstddef> when it wasn't already being included.

llvm-svn: 288746

show more ...

[libcxx] [test] D27018: Fix MSVC warning C4018 "signed/unsigned mismatch", part 5/12.

Various changes:

test/std/algorithms/alg.sorting/alg.binary.search/binary.search/binary_search.pass.cpp
Change

[libcxx] [test] D27018: Fix MSVC warning C4018 "signed/unsigned mismatch", part 5/12.

Various changes:

test/std/algorithms/alg.sorting/alg.binary.search/binary.search/binary_search.pass.cpp
Change M from unsigned to int. It's compared against "int x",
and we binary_search() for it within a vector<int>.

test/std/numerics/rand/rand.dis/rand.dist.norm/rand.dist.norm.f/eval.pass.cpp
test/std/numerics/rand/rand.dis/rand.dist.norm/rand.dist.norm.f/eval_param.pass.cpp
Add static_cast<unsigned> when comparing int to unsigned.

test/std/strings/basic.string/string.cons/size_char_alloc.pass.cpp
Change unsigned indices to int when we're being given int as a bound.

llvm-svn: 287825

show more ...

Guard libc++ specific c.__invariants() tests in LIBCPP_ASSERT macros

llvm-svn: 267947

Move test into test/std subdirectory.

llvm-svn: 224658