[ASan][libc++] std::basic_string annotations (#72677)

This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, on

[ASan][libc++] std::basic_string annotations (#72677)

This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, only long strings with the default allocator will be
annotated. Short strings (_SSO - short string optimization_) and strings
with non-default allocators will be annotated in the near future, with
separate commits dedicated to enabling them. The process will be similar
to the workflow employed for enabling annotations in `std::deque`.

**Please note**: these annotations function effectively only when libc++
and libc++abi dylibs are instrumented (with ASan). This aligns with the
prevailing behavior of Memory Sanitizer.

To avoid breaking everything, this commit also appends
`_LIBCPP_INSTRUMENTED_WITH_ASAN` to `__config_site` whenever libc++ is
compiled with ASan. If this macro is not defined, string annotations are
not enabled. However, linking a binary that does **not** annotate
strings with a dynamic library that annotates strings, is not permitted.

Originally proposed here: https://reviews.llvm.org/D132769

Related patches on Phabricator:
- Turning on annotations for short strings:
https://reviews.llvm.org/D147680
- Turning on annotations for all allocators:
https://reviews.llvm.org/D146214

This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.

The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.

This Pull Request introduces basic annotations for `std::basic_string`.
Long strings exhibit structural similarities to `std::vector` and will
be annotated accordingly. Short strings are already implemented, but
will be turned on separately in a forthcoming commit. Look at [a
comment](https://github.com/llvm/llvm-project/pull/72677#issuecomment-1850554465)
below to read about SSO issues at current moment.

Due to the functionality introduced in
[D132522](https://github.com/llvm/llvm-project/commit/dd1b7b797a116eed588fd752fbe61d34deeb24e4),
the `__sanitizer_annotate_contiguous_container` function now offers
compatibility with all allocators. However, enabling this support will
be done in a subsequent commit. For the time being, only strings with
the default allocator will be annotated.

If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com

show more ...

[libc++] Apply clang formatting to all string unit tests

This applies clang-format to the std::string unit tests in preparation
for landing https://reviews.llvm.org/D140550.

Differential Revision:

[libc++] Apply clang formatting to all string unit tests

This applies clang-format to the std::string unit tests in preparation
for landing https://reviews.llvm.org/D140550.

Differential Revision: https://reviews.llvm.org/D140612

show more ...

[libc++] Qualifies uint32_t and friends.

This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)(?<!::u)u?int(_[a-z]+)?[0-9]{1,2}_t)|\1std::\

[libc++] Qualifies uint32_t and friends.

This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)(?<!::u)u?int(_[a-z]+)?[0-9]{1,2}_t)|\1std::\2|' \{} \;

And manually removed some false positives in std/depr/depr.c.headers.

Reviewed By: ldionne, #libc

Differential Revision: https://reviews.llvm.org/D145880

show more ...

[libc++][NFC] Remove some of the code duplication in the string tests

Reviewed By: ldionne, #libc, huixie90

Spies: huixie90, libcxx-commits, arphaman

Differential Revision: https://reviews.llvm.or

[libc++][NFC] Remove some of the code duplication in the string tests

Reviewed By: ldionne, #libc, huixie90

Spies: huixie90, libcxx-commits, arphaman

Differential Revision: https://reviews.llvm.org/D131856

show more ...

[libcxx][AIX] Switch build compiler to clang

This patch switches the build compiler for AIX from ibm-clang to clang. ibm-clang++_r has `-pthread` by default, but clang for AIX doesn't, so `-pthread`

[libcxx][AIX] Switch build compiler to clang

This patch switches the build compiler for AIX from ibm-clang to clang. ibm-clang++_r has `-pthread` by default, but clang for AIX doesn't, so `-pthread` had to be added to the test config. A bunch of tests now pass, so the `XFAIL` was removed. This patch also switch the build to use the visibility support available in clang-15 to control symbols exported by the shared library (AIX traditionally uses explicit export lists for this purpose).

Reviewed By: #libc, #libc_abi, daltenty, #libunwind, ldionne

Differential Revision: https://reviews.llvm.org/D127470

show more ...

[libc++] Implement P0980R1 (constexpr std::string)

Reviewed By: #libc, ldionne

Spies: daltenty, sdasgup3, ldionne, arichardson, MTC, ChuanqiXu, mehdi_amini, shauheen, antiagainst, nicolasvasilache,

[libc++] Implement P0980R1 (constexpr std::string)

Reviewed By: #libc, ldionne

Spies: daltenty, sdasgup3, ldionne, arichardson, MTC, ChuanqiXu, mehdi_amini, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, aartbik, liufengdb, stephenneuendorffer, Joonsoo, grosul1, Kayjukh, jurahul, msifontes, tatianashp, rdzhabarov, teijeong, cota, dcaballe, Chia-hungDuan, wrengr, wenzhicui, arphaman, Mordante, miscco, Quuxplusone, smeenai, libcxx-commits

Differential Revision: https://reviews.llvm.org/D110598

show more ...

[libc++] Prepare string.modifiers tests for constexpr

Reviewed By: ldionne, #libc

Spies: libcxx-commits

Differential Revision: https://reviews.llvm.org/D119329

[libc++] Remove incorrect default constructor in cpp17_input_iterator

AFAICT, Cpp17InputIterators are not required to be default constructible,
since that requirement is added in Cpp17ForwardIterato

[libc++] Remove incorrect default constructor in cpp17_input_iterator

AFAICT, Cpp17InputIterators are not required to be default constructible,
since that requirement is added in Cpp17ForwardIterator. Hence, our
archetype for Cpp17InputIterator should not be default constructible.
Removing that constructor has a ripple effect on a couple of tests that
were making incorrect assumptions. Notably:

- Some tests were using cpp17_input_iterator as a sentinel for itself.
That is not valid, because a cpp17_input_iterator is not semiregular
anymore after the change (and hence it doesn't satisfy sentinel_for).

- Some tests were using a stride-counted cpp17_input_iterator as the
sentinel for a range. This doesn't work anymore because of the problem
above, so these tests were changed not to check stride counts for
input iterators.

- Some tests were default constructing cpp17_input_iterator when a simple
alternative was available -- those have been changed to use that alternative.

Differential Revision: https://reviews.llvm.org/D115806

show more ...

[libcxx][nfc] prefixes test type `input_iterator` with `cpp17_`

C++20 revised the definition of what it means to be an iterator. While
all _Cpp17InputIterators_ satisfy `std::input_iterator`, the re

[libcxx][nfc] prefixes test type `input_iterator` with `cpp17_`

C++20 revised the definition of what it means to be an iterator. While
all _Cpp17InputIterators_ satisfy `std::input_iterator`, the reverse
isn't true. D100271 introduces a new test adaptor to accommodate this
new definition (`cpp20_input_iterator`).

In order to help readers immediately distinguish which input iterator
adaptor is _Cpp17InputIterator_, the current `input_iterator` adaptor
has been prefixed with `cpp17_`.

Differential Revision: https://reviews.llvm.org/D101242

show more ...

[libc++] [test] Improve test_exceptions() in each string.modifiers test.

When checking the strong exception guarantee, also check that
iterators haven't been invalidated.

Reviewed as part of https:

[libc++] [test] Improve test_exceptions() in each string.modifiers test.

When checking the strong exception guarantee, also check that
iterators haven't been invalidated.

Reviewed as part of https://reviews.llvm.org/D98573

show more ...

[libc++] Remove the special logic for "noexcept iterators" in basic_string.

This reverts a large chunk of http://reviews.llvm.org/D15862 ,
and also fixes bugs in `insert`, `append`, and `assign`, wh

[libc++] Remove the special logic for "noexcept iterators" in basic_string.

This reverts a large chunk of http://reviews.llvm.org/D15862 ,
and also fixes bugs in `insert`, `append`, and `assign`, which are now regression-tested.
(Thanks to Tim Song for pointing out the bug in `append`!)

Before this patch, we did a special dance in `append`, `assign`, and `insert`
(but not `replace`). All of these require the strong exception guarantee,
even when the user-provided InputIterator might have throwing operations.

The naive way to accomplish this is to construct a temporary string and
then append/assign/insert from the temporary; i.e., finish all the potentially
throwing and self-inspecting InputIterator operations *before* starting to
modify self. But this is slow, so we'd like to skip it when possible.

The old code (D15682) attempted to check that specific iterator operations
were nothrow: it assumed that if the iterator operations didn't throw, then
it was safe to iterate the input range multiple times and therefore it was
safe to use the fast-path non-naive version. This was wrong for two reasons:
(1) the old code checked the wrong operations (e.g. checked noexceptness of `==`,
but the code that ran used `!=`), and (2) the conversion of value_type to char
could still throw, or inspect the contents of self.

The new code is much simpler, although still much more complicated than it
really could be. We'll likely revisit this codepath at some point, but for now
this patch suffices to get it passing all the new regression tests.

The added tests all fail before this patch, and succeed afterward.
See https://quuxplusone.github.io/blog/2021/04/17/pathological-string-appends/

Differential Revision: https://reviews.llvm.org/D98573

show more ...

[libc++] Fix several debug mode tests

These tests were only being run when _LIBCPP_DEBUG was defined, which
isn't the case by default when we run the test suite. In other words,
all these debug mode

[libc++] Fix several debug mode tests

These tests were only being run when _LIBCPP_DEBUG was defined, which
isn't the case by default when we run the test suite. In other words,
all these debug mode tests were never being run. This commit makes sure
they are run, and in some cases, extracts them into a file under test/libcxx
to separate them from the Standard tests.

Differential Revision: https://reviews.llvm.org/D88836

show more ...

Add include for 'test_macros.h' to all the tests that were missing them. Thanks to Zoe for the (big, but simple) patch. NFC intended.

llvm-svn: 362252

Support tests in freestanding

Summary:
Freestanding is *weird*. The standard allows it to differ in a bunch of odd
manners from regular C++, and the committee would like to improve that
situation. I

Support tests in freestanding

Summary:
Freestanding is *weird*. The standard allows it to differ in a bunch of odd
manners from regular C++, and the committee would like to improve that
situation. I'd like to make libc++ behave better with what freestanding should
be, so that it can be a tool we use in improving the standard. To do that we
need to try stuff out, both with "freestanding the language mode" and
"freestanding the library subset".

Let's start with the super basic: run the libc++ tests in freestanding, using
clang as the compiler, and see what works. The easiest hack to do this:

In utils/libcxx/test/config.py add:

self.cxx.compile_flags += ['-ffreestanding']

Run the tests and they all fail.

Why? Because in freestanding `main` isn't special. This "not special" property
has two effects: main doesn't get mangled, and main isn't allowed to omit its
`return` statement. The first means main gets mangled and the linker can't
create a valid executable for us to test. The second means we spew out warnings
(ew) and the compiler doesn't insert the `return` we omitted, and main just
falls of the end and does whatever undefined behavior (if you're luck, ud2
leading to non-zero return code).

Let's start my work with the basics. This patch changes all libc++ tests to
declare `main` as `int main(int, char**` so it mangles consistently (enabling us
to declare another `extern "C"` main for freestanding which calls the mangled
one), and adds `return 0;` to all places where it was missing. This touches 6124
files, and I apologize.

The former was done with The Magic Of Sed.

The later was done with a (not quite correct but decent) clang tool:

https://gist.github.com/jfbastien/793819ff360baa845483dde81170feed

This works for most tests, though I did have to adjust a few places when e.g.
the test runs with `-x c`, macros are used for main (such as for the filesystem
tests), etc.

Once this is in we can create a freestanding bot which will prevent further
regressions. After that, we can start the real work of supporting C++
freestanding fairly well in libc++.

<rdar://problem/47754795>

Reviewers: ldionne, mclow.lists, EricWF

Subscribers: christof, jkorous, dexonsmith, arphaman, miyuki, libcxx-commits

Differential Revision: https://reviews.llvm.org/D57624

llvm-svn: 353086

show more ...

Update more file headers across all of the LLVM projects in the monorepo
to reflect the new license. These used slightly different spellings that
defeated my regular expressions.

We understand that

Update more file headers across all of the LLVM projects in the monorepo
to reflect the new license. These used slightly different spellings that
defeated my regular expressions.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351648

show more ...

[libcxx] [test] Untabify, NFC.

llvm-svn: 309464

Fix PR32642 - string::insert and string::append don't work with move_iterator.

llvm-svn: 300397

[libcxx] [test] D27027: Strip trailing whitespace.

llvm-svn: 287829

Fix PR#30303 - no matching function for call to '__ptr_in_range'

llvm-svn: 280779

Fix Bug 30240 - std::string: append(first, last) error when aliasing. Add test cases for append/insert/assign/replace while we're at it, and fix a similar bug in insert.

llvm-svn: 280643

Guard libc++ specific c.__invariants() tests in LIBCPP_ASSERT macros

llvm-svn: 267947

More string fixes for noexcept cases. Apparently I didn't get them all in r258281.

llvm-svn: 258291

Fix up the tests I added for string exceptions to be skipped when exceptions are disabled

llvm-svn: 258279

Fix PR#25973 : 'basic_string::assign(InputIt, InputIt) doesn't provide the strong exception safety guarantee'. This turned out to be a pervasive problem in <string>, which required a fair amount of r

Fix PR#25973 : 'basic_string::assign(InputIt, InputIt) doesn't provide the strong exception safety guarantee'. This turned out to be a pervasive problem in <string>, which required a fair amount of rework. Add in an optimization for when iterators provide noexcept increment/comparison/assignment/dereference (which covers many of the iterators in libc++). Reviewed as http://reviews.llvm.org/D15862

llvm-svn: 257682

show more ...

Move test into test/std subdirectory.

llvm-svn: 224658