[libc++][hardening] Use bounded iterators in std::vector and std::string (#78929)

~~NB: This PR depends on #78876. Ignore the first commit when reviewing,
and don't merge it until #78876 is resolve

[libc++][hardening] Use bounded iterators in std::vector and std::string (#78929)

~~NB: This PR depends on #78876. Ignore the first commit when reviewing,
and don't merge it until #78876 is resolved. When/if #78876 lands, I'll
clean this up.~~

This partially restores parity with the old, since removed debug build.
We now can re-enable a bunch of the disabled tests. Some things of note:

- `bounded_iter`'s converting constructor has never worked. It needs a
friend declaration to access the other `bound_iter` instantiation's
private fields.

- The old debug iterators also checked that callers did not try to
compare iterators from different objects. `bounded_iter` does not
currently do this, so I've left those disabled. However, I think we
probably should add those. See
https://github.com/llvm/llvm-project/issues/78771#issuecomment-1902999181

- The `std::vector` iterators are bounded up to capacity, not size. This
makes for a weaker safety check. This is because the STL promises not to
invalidate iterators when appending up to the capacity. Since we cannot
retroactively update all the iterators on `push_back()`, I've instead
sized it to the capacity. This is not as good, but at least will stop
the iterator from going off the end of the buffer.

There was also no test for this, so I've added one in the `std`
directory.

- `std::string` has two ambiguities to deal with. First, I opted not to
size it against the capacity. https://eel.is/c++draft/string.require#4
says iterators are invalidated on an non-const operation. Second,
whether the iterator can reach the NUL terminator. The previous debug
tests and the special-case in https://eel.is/c++draft/string.access#2
suggest no. If either of these causes widespread problems, I figure we
can revisit.

- `resize_and_overwrite.pass.cpp` assumed `std::string`'s iterator
supported `s.begin().base()`, but I see no promise of this in the
standard. GCC also doesn't support this. I fixed the test to use
`std::to_address`.

- `alignof.compile.pass.cpp`'s pointer isn't enough of a real pointer.
(It needs to satisfy `NullablePointer`, `LegacyRandomAccessIterator`,
and `LegacyContiguousIterator`.) `__bounded_iter` seems to instantiate
enough to notice. I've added a few more bits to satisfy it.

Fixes #78805

show more ...

[ASan][libc++] std::basic_string annotations (#72677)

This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, on

[ASan][libc++] std::basic_string annotations (#72677)

This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, only long strings with the default allocator will be
annotated. Short strings (_SSO - short string optimization_) and strings
with non-default allocators will be annotated in the near future, with
separate commits dedicated to enabling them. The process will be similar
to the workflow employed for enabling annotations in `std::deque`.

**Please note**: these annotations function effectively only when libc++
and libc++abi dylibs are instrumented (with ASan). This aligns with the
prevailing behavior of Memory Sanitizer.

To avoid breaking everything, this commit also appends
`_LIBCPP_INSTRUMENTED_WITH_ASAN` to `__config_site` whenever libc++ is
compiled with ASan. If this macro is not defined, string annotations are
not enabled. However, linking a binary that does **not** annotate
strings with a dynamic library that annotates strings, is not permitted.

Originally proposed here: https://reviews.llvm.org/D132769

Related patches on Phabricator:
- Turning on annotations for short strings:
https://reviews.llvm.org/D147680
- Turning on annotations for all allocators:
https://reviews.llvm.org/D146214

This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.

The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.

This Pull Request introduces basic annotations for `std::basic_string`.
Long strings exhibit structural similarities to `std::vector` and will
be annotated accordingly. Short strings are already implemented, but
will be turned on separately in a forthcoming commit. Look at [a
comment](https://github.com/llvm/llvm-project/pull/72677#issuecomment-1850554465)
below to read about SSO issues at current moment.

Due to the functionality introduced in
[D132522](https://github.com/llvm/llvm-project/commit/dd1b7b797a116eed588fd752fbe61d34deeb24e4),
the `__sanitizer_annotate_contiguous_container` function now offers
compatibility with all allocators. However, enabling this support will
be done in a subsequent commit. For the time being, only strings with
the default allocator will be annotated.

If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com

show more ...

[libc++] Refactor string unit tests to ease addition of new allocators

While doing this, I also found a few tests that were either clearly
incorrect (e.g. testing the wrong function) or that lacked

[libc++] Refactor string unit tests to ease addition of new allocators

While doing this, I also found a few tests that were either clearly
incorrect (e.g. testing the wrong function) or that lacked basic test
coverage like testing std::string itself (e.g. the test was only checking
std::basic_string with a custom allocator). In these cases, I did a few
conservative drive-by changes.

Differential Revision: https://reviews.llvm.org/D140550
Co-authored-by: Brendan Emery <brendan.emery@esrlabs.com>

show more ...

[libc++] Apply clang formatting to all string unit tests

This applies clang-format to the std::string unit tests in preparation
for landing https://reviews.llvm.org/D140550.

Differential Revision:

[libc++] Apply clang formatting to all string unit tests

This applies clang-format to the std::string unit tests in preparation
for landing https://reviews.llvm.org/D140550.

Differential Revision: https://reviews.llvm.org/D140612

show more ...

[libc++] Qualifies size_t.

This has been done using the following command

find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)size_t)|\1std::\2|' \{} \;

And manually removed some fals

[libc++] Qualifies size_t.

This has been done using the following command

find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)size_t)|\1std::\2|' \{} \;

And manually removed some false positives in std/depr/depr.c.headers.

The `std` module doesn't export `::size_t`, this is a preparation for that module.

Reviewed By: ldionne, #libc, EricWF, philnik

Differential Revision: https://reviews.llvm.org/D146088

show more ...

[libc++] Implement P0980R1 (constexpr std::string)

Reviewed By: #libc, ldionne

Spies: daltenty, sdasgup3, ldionne, arichardson, MTC, ChuanqiXu, mehdi_amini, shauheen, antiagainst, nicolasvasilache,

[libc++] Implement P0980R1 (constexpr std::string)

Reviewed By: #libc, ldionne

Spies: daltenty, sdasgup3, ldionne, arichardson, MTC, ChuanqiXu, mehdi_amini, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, aartbik, liufengdb, stephenneuendorffer, Joonsoo, grosul1, Kayjukh, jurahul, msifontes, tatianashp, rdzhabarov, teijeong, cota, dcaballe, Chia-hungDuan, wrengr, wenzhicui, arphaman, Mordante, miscco, Quuxplusone, smeenai, libcxx-commits

Differential Revision: https://reviews.llvm.org/D110598

show more ...

[libc++] basic_string::resize_and_overwrite: Adopt LWG3645 (Not voted in yet)

Adopt LWG3645, which fixes the value categories of basic_string::resize_and_overwrite
https://timsong-cpp.github.io/lwg-

[libc++] basic_string::resize_and_overwrite: Adopt LWG3645 (Not voted in yet)

Adopt LWG3645, which fixes the value categories of basic_string::resize_and_overwrite
https://timsong-cpp.github.io/lwg-issues/3645

Reviewed By: ldionne, #libc

Spies: libcxx-commits

Differential Revision: https://reviews.llvm.org/D116815

show more ...

[libcxx][test] Add missing includes and suppress warnings

... from testing with MSVC's STL. Mostly truncation warnings and variables that are only used in `LIBCPP_ASSERT`.

Differential Revision: ht

[libcxx][test] Add missing includes and suppress warnings

... from testing with MSVC's STL. Mostly truncation warnings and variables that are only used in `LIBCPP_ASSERT`.

Differential Revision: https://reviews.llvm.org/D116878

show more ...

[libc++] Implement P1072R10 (std::basic_string::resize_and_overwrite)

Reviewed By: Quuxplusone, #libc, Mordante

Spies: mzeren-vmw, ckennelly, arichardson, ldionne, Mordante, libcxx-commits, Quuxplu

[libc++] Implement P1072R10 (std::basic_string::resize_and_overwrite)

Reviewed By: Quuxplusone, #libc, Mordante

Spies: mzeren-vmw, ckennelly, arichardson, ldionne, Mordante, libcxx-commits, Quuxplusone

Differential Revision: https://reviews.llvm.org/D113013

show more ...