[Clang] Update Unicode version to 15.1 (#77147)

This update all of our Unicode tables to Unicode 15.1. This is a minor
version so only a relatively small numbers of characters are added,
mainly ideo

[Clang] Update Unicode version to 15.1 (#77147)

This update all of our Unicode tables to Unicode 15.1. This is a minor
version so only a relatively small numbers of characters are added,
mainly ideographs

https://www.unicode.org/versions/Unicode15.1.0/#Appendices_nb

show more ...

Fix handling of medial hyphens in Unicode Names.

In a Unicode name was stored in a way that caused
a medial hyphen to be at the end of a a chunk, it would not
be properly ignored by the loose matchi

Fix handling of medial hyphens in Unicode Names.

In a Unicode name was stored in a way that caused
a medial hyphen to be at the end of a a chunk, it would not
be properly ignored by the loose matching algorithm.

For example if `LEFT-TO-RIGHT OVERRIDE` was stored as
`LEFT-` [...], the `-` would not be ignored.

The generators now ensures nodes are not cut accross
medial hyphen boundaries.

Fixes #64161

Differential Revision: https://reviews.llvm.org/D156518

show more ...

Use *{Map,Set}::contains (NFC)

Differential Revision: https://reviews.llvm.org/D146104

[llvm] Include <optional> instead of "llvm/ADT/Optional.h" (NFC)

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-

[llvm] Include <optional> instead of "llvm/ADT/Optional.h" (NFC)

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[llvm] Use std::optional instead of Optional

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-ge

[llvm] Use std::optional instead of Optional

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[Clang][NFC] Fix UnicodeData.txt parsing.

Fix the UnicodeNameToCodepoint tool misshandling
the case of a line containing a single semi-colon
(which does not happens in practice)

Update Unicode to 15.0

Unicode 15.0 adds 4,489 characters, for a total of 149,186 characters.
These additions include 2 new scripts along with 20 new emoji characters,
and 4,193 CJK ideographs.

Thi

Update Unicode to 15.0

Unicode 15.0 adds 4,489 characters, for a total of 149,186 characters.
These additions include 2 new scripts along with 20 new emoji characters,
and 4,193 CJK ideographs.

This changes modify most existing tables including
- XID_Start/XID_Continue in Clang
- The character name database (used by \N{} in Clang)
- The list of formattable/printable codepoints
- The case folding algorithm (which we had not updated since Unicode 9)
- The list of nonspacing/enclosing marks used by the column width
computation algorithm. The rest of the column width algorithm
is not updated.

Reviewed By: tahonermann

Differential Revision: https://reviews.llvm.org/D133807

show more ...

[llvm] Use llvm::find_if (NFC)

Use llvm::all_of (NFC)

[llvm] Fix comment typos (NFC)

Use llvm::sort instead of std::sort where possible

llvm::sort is beneficial even when we use the iterator-based overload,
since it can optionally shuffle the elements (to detect
non-determinism). Ho

Use llvm::sort instead of std::sort where possible

llvm::sort is beneficial even when we use the iterator-based overload,
since it can optionally shuffle the elements (to detect
non-determinism). However llvm::sort is not usable everywhere, for
example, in compiler-rt.

Reviewed By: nhaehnle

Differential Revision: https://reviews.llvm.org/D130406

show more ...

Use callables directly in any_of, count_if, etc (NFC)

Silence some format specifier warnings

This solves a format specifier warning for char32_t being converted to
an unsigned integer type, and multiple format specifier warnings for
size_t being conver

Silence some format specifier warnings

This solves a format specifier warning for char32_t being converted to
an unsigned integer type, and multiple format specifier warnings for
size_t being converted to long.

show more ...

[Clang][C++23] P2071 Named universal character escapes

Implements [[ https://wg21.link/p2071r1 | P2071 Named Universal Character Escapes ]] - as an extension in all language mode, the patch not wa

[Clang][C++23] P2071 Named universal character escapes

Implements [[ https://wg21.link/p2071r1 | P2071 Named Universal Character Escapes ]] - as an extension in all language mode, the patch not warn in c++23 mode will be done later once this paper is plenary approved (in July).

We add

* A code generator that transforms `UnicodeData.txt` and `NameAliases.txt` to a space efficient data structure that can be queried in `O(NameLength)`
* A set of functions in `Unicode.h` to query that data, including

* A function to find an exact match of a given Unicode character name
* A function to perform a loose (ignoring case, space, underscore, medial hyphen) matching
* A function returning the best matching codepoint for a given string per edit distance

* Support of `\N{}` escape sequences in String and character Literals, with loose and typos diagnostics/fixits
* Support of `\N{}` as UCN with loose matching diagnostics/fixits.

Loose matching is considered an error to match closely the semantics of P2071.

The generated data contributes to 280kB of data to the binaries.

`UnicodeData.txt` and `NameAliases.txt` are not committed to the repository in this patch, and regenerating the data is a manual process.

Reviewed By: tahonermann

Differential Revision: https://reviews.llvm.org/D123064

show more ...