[SystemZ][z/OS] Update autoconversion functions to improve support for UTF-8 (#98652)

This fixes the following error when reading source and header files on
z/OS: error: source file is not valid UT

[SystemZ][z/OS] Update autoconversion functions to improve support for UTF-8 (#98652)

This fixes the following error when reading source and header files on
z/OS: error: source file is not valid UTF-8

show more ...

[Support] Remove unused includes (NFC) (#116752)

Identified with misc-include-cleaner.

[NFC][SystemZ][z/OS] Rename autoconversion-related functions to be less generic (#107399)

This patch renames the functions in AutoConvert.h/cpp to have a less
generic name because they are z/OS spe

[NFC][SystemZ][z/OS] Rename autoconversion-related functions to be less generic (#107399)

This patch renames the functions in AutoConvert.h/cpp to have a less
generic name because they are z/OS specific.

show more ...

Correct mismatched allocation/deallocation calls

This amends dceaa0f4491ebe30c0b0f1bc7fa5ec365b60ced6 because ASAN
caught an issue where the allocation and deallocation were not properly
paired: htt

Correct mismatched allocation/deallocation calls

This amends dceaa0f4491ebe30c0b0f1bc7fa5ec365b60ced6 because ASAN
caught an issue where the allocation and deallocation were not properly
paired: https://lab.llvm.org/buildbot/#/builders/239/builds/7001

Use malloc and free throughout this file to ensure that all kinds of
memory buffers use the proper pairing.

show more ...

[Support] Use malloc instead of non-throwing new (#92157)

When allocating a memory buffer, we use a non-throwing new so that we
can explicitly handle memory buffers that are too large to fit into

[Support] Use malloc instead of non-throwing new (#92157)

When allocating a memory buffer, we use a non-throwing new so that we
can explicitly handle memory buffers that are too large to fit into
memory. However, when exceptions are disabled, LLVM installs a custom
new handler

(https://github.com/llvm/llvm-project/blob/90109d444839683b09f0aafdc50b749cb4b3203b/llvm/lib/Support/InitLLVM.cpp#L61)
that explicitly crashes when we run out of memory

(https://github.com/llvm/llvm-project/blob/de14b749fee41d4ded711e771e43043ae3100cb3/llvm/lib/Support/ErrorHandling.cpp#L188)
and that means this particular out-of-memory situation cannot be
gracefully handled.

This was discovered while working on #embed
(https://github.com/llvm/llvm-project/pull/68620) on Windows and
resulted in a crash rather than the preprocessor issuing a diagnostic as
expected.

This patch switches away from the non-throwing new to a call to malloc
(and free), which will return a null pointer without calling a custom
new handler. It is the only instance in Clang or LLVM that I could find
which used a non-throwing new, so I did not think we would need anything
more involved than this change.

Testing this would be highly platform dependent and so it does not come
with test coverage. And because it doesn't change behavior that users
are likely to be able to observe, it does not come with a release note.

show more ...

[Support][MemBuffer] Prevent UB on empty StringRefs

Empty StringRefs are usually identified by their length being zero, and
sometimes they'll have Data==nullptr (e.g. default constructed, or derived

[Support][MemBuffer] Prevent UB on empty StringRefs

Empty StringRefs are usually identified by their length being zero, and
sometimes they'll have Data==nullptr (e.g. default constructed, or derived from
an operation like split/copy and result turned out to be empty).

If such StringRef objects are passed to llvm::MemoryBuffer::getMemBufferCopy,
it'll result in UB as neither src nor dst can be null, even if size is zero.

This patch prevents that UB by not issuing a copy whenever StringRef is empty.

Differential Revision: https://reviews.llvm.org/D144706

show more ...

[Support] llvm::Optional => std::optional

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

Cache memory buffer's name length

This avoids repeated calls to strlen while we already know its value.

When preprocessing sqlite3.c, this gives a surprising 2% speedup.

Full benchmark available h

Cache memory buffer's name length

This avoids repeated calls to strlen while we already know its value.

When preprocessing sqlite3.c, this gives a surprising 2% speedup.

Full benchmark available here:

https://llvm-compile-time-tracker.com/compare.php?from=d14c2d408dccd8c6defa7d151e9a96be3cac8cc3&to=04f0641c1cbdcd0bdbd11cd910ca6091420bf52e&stat=instructions:u

Recommit 1824432174b3166b40bce59477beb5821170748e, with restored '\0' at the end of buffer name.

Differential Revision: https://reviews.llvm.org/D138555

show more ...

Revert "Cache memory buffer's name length"

This reverts commit 1824432174b3166b40bce59477beb5821170748e.

The change triggers an assertion when building clang on macOS:

FAILED: tools/clang/include/

Revert "Cache memory buffer's name length"

This reverts commit 1824432174b3166b40bce59477beb5821170748e.

The change triggers an assertion when building clang on macOS:

FAILED: tools/clang/include/clang/Tooling/Syntax/Nodes.inc
cd /Users/florianhahn/projects/llvm-project/builds/release-with-assertions/ccache-stage1 && /Users/florianhahn/projects/llvm-project/builds/release-with-assertions/ccache-stage1/bin/clang-tblgen -gen-clang-syntax-node-list -I /Users/florianhahn/projects/llvm-project/clang/include/clang/Tooling/Syntax -I/Users/florianhahn/projects/llvm-project/clang/include -I/Users/florianhahn/projects/llvm-project/builds/release-with-assertions/ccache-stage1/tools/clang/include -I/Users/florianhahn/projects/llvm-project/builds/release-with-assertions/ccache-stage1/include -I/Users/florianhahn/projects/llvm-project/llvm/include /Users/florianhahn/projects/llvm-project/clang/include/clang/Tooling/Syntax/Nodes.td --write-if-changed -o tools/clang/include/clang/Tooling/Syntax/Nodes.inc -d tools/clang/include/clang/Tooling/Syntax/Nodes.inc.d
Assertion failed: ((!RequiresNullTerminator || BufEnd[0] == 0) && "Buffer is not null terminated!"), function init, file MemoryBuffer.cpp, line 52.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0 clang-tblgen 0x000000010466b68c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1 clang-tblgen 0x000000010466a808 llvm::sys::RunSignalHandlers() + 64
2 clang-tblgen 0x000000010466bd60 SignalHandler(int) + 344
3 libsystem_platform.dylib 0x00000001877a92a4 _sigtramp + 56
4 libsystem_pthread.dylib 0x000000018777acec pthread_kill + 288
5 libsystem_c.dylib 0x00000001876b42c8 abort + 180
6 libsystem_c.dylib 0x00000001876b3620 err + 0
7 clang-tblgen 0x00000001046ce150 llvm::ErrorOr<std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>> getOpenFileImpl<llvm::MemoryBuffer>(int, llvm::Twine const&, unsigned long long, unsigned long long, long long, bool, bool, llvm::Optional<llvm::Align>) (.cold.1) + 0
8 clang-tblgen 0x000000010464a548 llvm::WritableMemoryBuffer::getNewUninitMemBuffer(unsigned long, llvm::Twine const&, llvm::Optional<llvm::Align>) + 496
9 clang-tblgen 0x000000010464a7a0 llvm::ErrorOr<std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>> getOpenFileImpl<llvm::MemoryBuffer>(int, llvm::Twine const&, unsigned long long, unsigned long long, long long, bool, bool, llvm::Optional<llvm::Align>) + 528
10 clang-tblgen 0x000000010464a254 llvm::ErrorOr<std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>> getFileAux<llvm::MemoryBuffer>(llvm::Twine const&, unsigned long long, unsigned long long, bool, bool, bool, llvm::Optional<llvm::Align>) + 176
11 clang-tblgen 0x000000010466fbbc llvm::TableGenMain(char const*, bool (*)(llvm::raw_ostream&, llvm::RecordKeeper&)) + 1428
12 clang-tblgen 0x0000000104626114 main + 156
13 dyld 0x0000000187453e50 start + 2544

show more ...

Cache memory buffer's name length

This avoids repeated calls to strlen while we already know its value.

When preprocessing sqlite3.c, this gives a surprising 2% speedup.

Full benchmark available h

Cache memory buffer's name length

This avoids repeated calls to strlen while we already know its value.

When preprocessing sqlite3.c, this gives a surprising 2% speedup.

Full benchmark available here: https://llvm-compile-time-tracker.com/compare.php?from=5279e6a7d677cdf4488883b77aacab911318100c&to=389601b0dbdf23cf25167ddfc49b3af5742ebd9a&stat=instructions:u

Differential Revision: https://reviews.llvm.org/D138555

show more ...

[MemoryBuffer] Allow optionally specifying desired buffer alignment

Underlying data may have requirements/expectations/etc. about
the run-time alignment. WritableMemoryBuffer currently uses
a 16 byt

[MemoryBuffer] Allow optionally specifying desired buffer alignment

Underlying data may have requirements/expectations/etc. about
the run-time alignment. WritableMemoryBuffer currently uses
a 16 byte alignment, which works for many situations but not all.
Allowing a desired alignment makes it easier to reuse WritableMemoryBuffer
in situations of special alignment, and also removes a problem when
opening files with special alignment constraints. Large files generally
get mmaped, which has ~page alignment, but small files go through
WritableMemoryBuffer which has the much smaller alignment guarantee.

Differential Revision: https://reviews.llvm.org/D137820

show more ...

Fix issing header on z/OS

Bug introduced in fbbc41f8dd23

Cleanup include: TableGen

This also includes a few cleanup from Support.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.l

Cleanup include: TableGen

This also includes a few cleanup from Support.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121331

show more ...

Fixed a roll-over on size_t in getNewUninitMemBuffer()

Reviewed By: serge-sans-paille

Differential Revision: https://reviews.llvm.org/D121399

[llvm] Use = default (NFC)

[SystemZ][z/OS] Add AutoConvert.h header to MemoryBuffer.cpp

This commit https://github.com/llvm/llvm-project/commit/75e164f61d391979b4829bf2746a5d74b94e95f2 removed the AutoConvert.h header causing

[SystemZ][z/OS] Add AutoConvert.h header to MemoryBuffer.cpp

This commit https://github.com/llvm/llvm-project/commit/75e164f61d391979b4829bf2746a5d74b94e95f2 removed the AutoConvert.h header causing a build break on z/OS. This patch adds it back to fix it.

Reviewed By: zibi

Differential Revision: https://reviews.llvm.org/D118129

show more ...

[NFC] Additional header dependency cleanup LLVMSupport

A few more forward-declarations, a few less headers. the impact on number of
preprocessed lines for LLVMSupport is negligible (-3K lines) but i

[NFC] Additional header dependency cleanup LLVMSupport

A few more forward-declarations, a few less headers. the impact on number of
preprocessed lines for LLVMSupport is negligible (-3K lines) but it's always
good to remove dependencies.

Related discourse thread: https://llvm.discourse.group/t/include-what-you-use-include-cleanup

show more ...

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. R

[llvm] Cleanup header dependencies in ADT and Support

The cleanup was manual, but assisted by "include-what-you-use". It consists in

1. Removing unused forward declaration. No impact expected.
2. Removing unused headers in .cpp files. No impact expected.
3. Removing unused headers in .h files. This removes implicit dependencies and
is generally considered a good thing, but this may break downstream builds.
I've updated llvm, clang, lld, lldb and mlir deps, and included a list of the
modification in the second part of the commit.
4. Replacing header inclusion by forward declaration. This has the same impact
as 3.

Notable changes:

- llvm/Support/TargetParser.h no longer includes llvm/Support/AArch64TargetParser.h nor llvm/Support/ARMTargetParser.h
- llvm/Support/TypeSize.h no longer includes llvm/Support/WithColor.h
- llvm/Support/YAMLTraits.h no longer includes llvm/Support/Regex.h
- llvm/ADT/SmallVector.h no longer includes llvm/Support/MemAlloc.h nor llvm/Support/ErrorHandling.h

You may need to add some of these headers in your compilation units, if needs be.

As an hint to the impact of the cleanup, running

clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Support/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l

before: 8000919 lines
after: 7917500 lines

Reduced dependencies also helps incremental rebuilds and is more ccache
friendly, something not shown by the above metric :-)

Discourse thread on the topic: https://llvm.discourse.group/t/include-what-you-use-include-cleanup/5831

show more ...

Support: Extract sys::fs::readNativeFileToEOF() from MemoryBuffer

Extract the `readNativeFile()` loop from
`MemoryBuffer::getMemoryBufferForStream()` into `readNativeFileToEOF()`
to allow reuse. The

Support: Extract sys::fs::readNativeFileToEOF() from MemoryBuffer

Extract the `readNativeFile()` loop from
`MemoryBuffer::getMemoryBufferForStream()` into `readNativeFileToEOF()`
to allow reuse. The chunk size is configurable; the default of `4*4096`
is exposed as `sys::fs::DefaultReadChunkSize` to allow sizing of
SmallVectors.

There's somewhere I'd like to read a usually-small file without overhead
of a MemoryBuffer; extracting existing logic rather than duplicating it.

Differential Revision: https://reviews.llvm.org/D115397

show more ...

[Support] Add MemoryBuffer::dontNeedIfMmap

On *NIX systems, this API calls madvise(MADV_DONTNEED) on read-only file mappings.
It should not be used on a writable buffer.
The API is used to implement

[Support] Add MemoryBuffer::dontNeedIfMmap

On *NIX systems, this API calls madvise(MADV_DONTNEED) on read-only file mappings.
It should not be used on a writable buffer.
The API is used to implement ld.lld LTO memory saving trick (D116367).

Note: on read-only file mappings, Linux's MADV_DONTNEED semantics match POSIX
POSIX_MADV_DONTNEED and BSD systems' MADV_DONTNEED.

On Windows, VirtualAllocEx MEM_COMMIT/MEM_RESET have similar semantics
but are unfortunately not drop-in replacements. dontNeedIfMmap is currently a no-op.

Reviewed By: aganea

Differential Revision: https://reviews.llvm.org/D116366

show more ...

Support: Avoid using SmallVector::set_size() in MemoryBuffer

Update getMemoryBufferForStream() to use `resize_for_overwrite()` and
`truncate()` instead of `reserve()` and `set_size()`.

Differential

Support: Avoid using SmallVector::set_size() in MemoryBuffer

Update getMemoryBufferForStream() to use `resize_for_overwrite()` and
`truncate()` instead of `reserve()` and `set_size()`.

Differential Revision: https://reviews.llvm.org/D115384

show more ...

[SystemZ][z/OS] Add the functions needed for handling EBCDIC I/O

This patch adds the basic functions needed for controlling auto conversion on z/OS.
Auto conversion is enabled on untagged input file

[SystemZ][z/OS] Add the functions needed for handling EBCDIC I/O

This patch adds the basic functions needed for controlling auto conversion on z/OS.
Auto conversion is enabled on untagged input file to ASCII by making the assumption that all untagged files are EBCDIC encoded. Output files are auto converted to EBCDIC IBM-1047.
This change also enables conversion for stdin/stdout/stderr.

For more information on how fcntl controls codepage https://www.ibm.com/docs/en/zos/2.4.0?topic=descriptions-fcntl-bpx1fct-bpx4fct-control-open-file-descriptors

Reviewed By: anirudhp

Differential Revision: https://reviews.llvm.org/D100483

show more ...

[SystemZ][z/OS][Windows] Add new functions that set Text/Binary mode for Stdin and Stdout based on OpenFlags

On Windows, we want to open a file in Binary mode if OF_CRLF bit is not set. On z/OS, we

[SystemZ][z/OS][Windows] Add new functions that set Text/Binary mode for Stdin and Stdout based on OpenFlags

On Windows, we want to open a file in Binary mode if OF_CRLF bit is not set. On z/OS, we want to open a file in Binary mode if the OF_Text bit is not set.

This patch creates two new functions called ChangeStdinMode and ChangeStdoutMode which will take OpenFlags as an arg to determine which mode to set stdin and stdout to. This will enable patches like https://reviews.llvm.org/D100056 to not affect Windows when setting the OF_Text flag for raw_fd_streams.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D100130

show more ...

[SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text

Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a C

[SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text

Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a CRLF '\r\n' which may not be desirable.

Solution:
This patch adds two new flags

- OF_CRLF which indicates that CRLF translation is used.
- OF_TextWithCRLF = OF_Text | OF_CRLF indicates that the file is text and uses CRLF translation.

Developers should now use either the OF_Text or OF_TextWithCRLF for text files and OF_None for binary files. If the developer doesn't want carriage returns on Windows, they should use OF_Text, if they do want carriage returns on Windows, they should use OF_TextWithCRLF.

So this is the behaviour per platform with my patch:

z/OS:
OF_None: open in binary mode
OF_Text : open in text mode
OF_TextWithCRLF: open in text mode

Windows:
OF_None: open file with no carriage return
OF_Text: open file with no carriage return
OF_TextWithCRLF: open file with carriage return

The Major change is in llvm/lib/Support/Windows/Path.inc to only set text mode if the OF_CRLF is set.
```
if (Flags & OF_CRLF)
CrtOpenFlags |= _O_TEXT;
```

These following files are the ones that still use OF_Text which I left unchanged. I modified all these except raw_ostream.cpp in recent patches so I know these were previously in Binary mode on Windows.
./llvm/lib/Support/raw_ostream.cpp
./llvm/lib/TableGen/Main.cpp
./llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
./llvm/unittests/Support/Path.cpp
./clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp
./clang/lib/Frontend/CompilerInstance.cpp
./clang/lib/Driver/Driver.cpp
./clang/lib/Driver/ToolChains/Clang.cpp

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D99426

show more ...

[NFC] Reordering parameters in getFile and getFileOrSTDIN

In future patches I will be setting the IsText parameter frequently so I will refactor the args to be in the following order. I have removed

[NFC] Reordering parameters in getFile and getFileOrSTDIN

In future patches I will be setting the IsText parameter frequently so I will refactor the args to be in the following order. I have removed the FileSize parameter because it is never used.

```
static ErrorOr<std::unique_ptr<MemoryBuffer>>
getFile(const Twine &Filename, bool IsText = false,
bool RequiresNullTerminator = true, bool IsVolatile = false);

static ErrorOr<std::unique_ptr<MemoryBuffer>>
getFileOrSTDIN(const Twine &Filename, bool IsText = false,
bool RequiresNullTerminator = true);

static ErrorOr<std::unique_ptr<MB>>
getFileAux(const Twine &Filename, uint64_t MapSize, uint64_t Offset,
bool IsText, bool RequiresNullTerminator, bool IsVolatile);

static ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
getFile(const Twine &Filename, bool IsVolatile = false);
```

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D99182

show more ...