[SystemZ][z/OS] Enable lit testing for z/OS (#107631)

This patch fixes various errors to enable llvm-lit to run on z/OS

fix(llvm/**.py): fix comparison to None (#94018)

from PEP8
(https://peps.python.org/pep-0008/#programming-recommendations):

> Comparisons to singletons like None should always be done with is or

fix(llvm/**.py): fix comparison to None (#94018)

from PEP8
(https://peps.python.org/pep-0008/#programming-recommendations):

> Comparisons to singletons like None should always be done with is or
is not, never the equality operators.

Co-authored-by: Eisuke Kawashima <e-kwsm@users.noreply.github.com>

show more ...

[libc++][AIX] Add OS version to target triple

This will allow for configuring tests according to AIX version.

Reviewed By: daltenty, #libc, Mordante

Differential Revision: https://reviews.llvm.org

[libc++][AIX] Add OS version to target triple

This will allow for configuring tests according to AIX version.

Reviewed By: daltenty, #libc, Mordante

Differential Revision: https://reviews.llvm.org/D149660

show more ...

[lit] abs_path_preserve_drive - don't normalize path, leave this to the caller

As noted on D154130, this was preventing path matching between normalized/unnormalized paths on some windows builds.

[lit][clang] Avoid realpath on Windows due to MAX_PATH limitations

Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths

[lit][clang] Avoid realpath on Windows due to MAX_PATH limitations

Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths short and avoid hitting MAX_PATH limitations.

Changes lit logic to:

Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we
can work around using substitute drives, which `os.path.realpath` would
resolve.

Use `os.path.realpath` on Unix, where the current directory always has
symlinks resolved, so it is impossible to preserve symlinks in the
presence of relative paths, and so we must make sure that all code paths
use real paths.

Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI`
code to avoid resolving substitute drives (i.e. resolving to a path
under a different root).

How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows

Differential Revision: https://reviews.llvm.org/D154130
Reviewed By: @benlangmuir

Patch by Tristan Labelle <tristan@thebrowser.company>!

show more ...

[NFC][Py Reformat] Reformat python files in llvm

This is the first commit in a series that will reformat
all the python files in the LLVM repository.

Reformatting is done with `black`.

See more in

[NFC][Py Reformat] Reformat python files in llvm

This is the first commit in a series that will reformat
all the python files in the LLVM repository.

Reformatting is done with `black`.

See more information here:

https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style

Reviewed By: jhenderson, JDevlieghere, MatzeB

Differential Revision: https://reviews.llvm.org/D150545

show more ...

[test] Fix macOS triple check

While the inferred host triple for macOS is something like
<arch>-apple-darwin, it's also valid to have <arch>-apple-macos.
Currently that globally changes whether an S

[test] Fix macOS triple check

While the inferred host triple for macOS is something like
<arch>-apple-darwin, it's also valid to have <arch>-apple-macos.
Currently that globally changes whether an SDKROOT is provided in tests,
so make this check more portable.

Differential Revision: https://reviews.llvm.org/D129684

show more ...

[lit] Keep stdout/stderr when using GoogleTest format

When a unit test crashes or timeout, print the shard's stdout and
stderr. When a unit test fails, attaches the test's output to the LIT
output t

[lit] Keep stdout/stderr when using GoogleTest format

When a unit test crashes or timeout, print the shard's stdout and
stderr. When a unit test fails, attaches the test's output to the LIT
output to help debugging.

While at it, concatenating shard's environment variables using space
instead of newline to make the reproducer script user friendly.

Based on D123797. (Thanks to @lenary)

show more ...

[lit] Support relative path arguments

Currently the behavior with relative paths is pretty broken. It differs
between external shell and internal shell because the path resolution
is done with a dif

[lit] Support relative path arguments

Currently the behavior with relative paths is pretty broken. It differs
between external shell and internal shell because the path resolution
is done with a different working directory. With the internal shell,
it's resolved relative to the directory from which lit is executed,
whereas with the external shell it's resolved relative to where the
test case is executed. To make matters worse, using the internal shell
the filepath to binaries looked up with `which` is returned relative
to the directory from which lit is executed, but then executed from
the test execution directory. That means that relative paths with the
internal shell give a `[Errno 2] No such file or directory` error
instead of the expected `command not found`.

To address these issues this patch makes lit interpret relative paths
as relative to the directory from which lit was invoked and modifies
`which` to return absolute paths, matching the behavior of its
namesake unix function.

See https://groups.google.com/g/llvm-dev/c/KzMWlOXR98Y/m/QJoqn0U5HAAJ

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D115486

show more ...

[lit] Bump up the Windows process cap from 32 to 60

At 61 or over, I see messages like

File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winap

[lit] Bump up the Windows process cap from 32 to 60

At 61 or over, I see messages like

File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)

ValueError: need at most 63 handles, got a sequence of length 64

60 seems to work for me.

If this causes issues for anybody else, feel free to revert.

show more ...

Revert "lit: revert 134b103fc0f3a995d76398bf4b029d72bebe8162"

This reverts commit d319005a3746a7661c8c9a3302266b6ff7cf61be.

Causing messages like:

File "...\Python\Python39\lib\multiprocessing\c

Revert "lit: revert 134b103fc0f3a995d76398bf4b029d72bebe8162"

This reverts commit d319005a3746a7661c8c9a3302266b6ff7cf61be.

Causing messages like:

File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 74

show more ...

lit: revert 134b103fc0f3a995d76398bf4b029d72bebe8162

Revert the 32-process cap on Windows. When testing with Swift, we found
that there was a time reduction for testing with the higher load. This

lit: revert 134b103fc0f3a995d76398bf4b029d72bebe8162

Revert the 32-process cap on Windows. When testing with Swift, we found
that there was a time reduction for testing with the higher load. This
should hopefully not matter much in practice. In the case that the
original problem with python remains with a high subprocess count, we
can easily revert this change.

show more ...

Reland "[lit] Use os.cpu_count() to cleanup TODO"

The initial problem with the remaining bot config was resolved.

We can now use Python3. Let's use `os.cpu_count()` to cleanup this
helper.

Differ

Reland "[lit] Use os.cpu_count() to cleanup TODO"

The initial problem with the remaining bot config was resolved.

We can now use Python3. Let's use `os.cpu_count()` to cleanup this
helper.

Differential Revision: https://reviews.llvm.org/D94734

show more ...

Revert "[lit] Use os.cpu_count() to cleanup TODO"

A bot owner contacted me. I will re-land after confirming that this
doesn't break anyone (since it's low priority).

This reverts commit 9946b169c3

Revert "[lit] Use os.cpu_count() to cleanup TODO"

A bot owner contacted me. I will re-land after confirming that this
doesn't break anyone (since it's low priority).

This reverts commit 9946b169c379daee603436a4753acfef8be373dd.

show more ...

[lit] Use os.cpu_count() to cleanup TODO

We can now use Python3. Let's use `os.cpu_count()` to cleanup this
helper.

Differential Revision: https://reviews.llvm.org/D94734

Remove a function that has been dead since r313889.

[lit] Print slowest tests and time histogram before result groups

[lit] Print slowest test first when timing tests

lit supports `--time-tests` which will report the 20 slowest tests and
print a nice histogram for test times. This change prints this list and
the h

[lit] Print slowest test first when timing tests

lit supports `--time-tests` which will report the 20 slowest tests and
print a nice histogram for test times. This change prints this list and
the histogram rows by decreasing test times. After all, we are most
interested in the slowest tests.

show more ...

[lit] Cleanly exit on user keyboard interrupt

Graceful lit shutdown on user keyboard interrupt [Ctrl+C] was a
longstanding goal of mine. After a few refactorings this revision
finally enables it.

[lit] Cleanly exit on user keyboard interrupt

Graceful lit shutdown on user keyboard interrupt [Ctrl+C] was a
longstanding goal of mine. After a few refactorings this revision
finally enables it. We use the following strategy to deal with
KeyboardInterrupt:
https://noswap.com/blog/python-multiprocessing-keyboardinterrupt

Printing of a helpful summary for interrupted runs (just as the one for
completed runs) will be tackled in future revisions.

Reviewed By: serge-sans-paille, rnk

Differential Revision: https://reviews.llvm.org/D77365

show more ...

[lit] [windows] Make sure to convert all path separators to backslashes in NT style \\?\... paths

E.g. the mingw python distributed in msys2 (the mingw one, which is a
normal win32 application and d

[lit] [windows] Make sure to convert all path separators to backslashes in NT style \\?\... paths

E.g. the mingw python distributed in msys2 (the mingw one, which is a
normal win32 application and doesn't use the msys2 runtime itself),
despite being a normal win32 python, still uses forward slashes. This
works fine for other cases (many, but not all), but when constructing a
raw NT path, all path separators must be backslashes.

Differential Revision: https://reviews.llvm.org/D71490

show more ...

[lit] Move sharding logic into separate function

[AIX][lit] Don't depend on psutil on AIX

Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off

[AIX][lit] Don't depend on psutil on AIX

Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off
a test and it's children.

This patch adds a workaround by invoking shell via subprocess and using a
platform specific option to ps to list all the descendant processes so we can
kill them. We add some checks so lit can tell whether timeout tests are
supported with out exposing whether we are utilizing the psutil
implementation or the alternative.

Reviewers: hubert.reinterpretcast, andusy, davide, delcypher

Reviewed By: delcypher

Subscribers: davide, delcypher, christof, lldb-commits, libcxx-commits, llvm-commits

Tags: #lldb, #libc, #llvm

Differential Revision: https://reviews.llvm.org/D64251

llvm-svn: 366912

show more ...

NFC: lit: python3 compatibility for functions
usePlatformSdkOnDarwin/findPlatformSdkVersionOnMacOS

These functions should decode subprocess output

llvm-svn: 365852

lit: make rm python 3 friendly (NFC)

Add some alterations for python 3 compatibility.

llvm-svn: 357789

lit: support long paths on Windows

Use ctypes to call into SHFileOperationW with the extended NT path to allow us
to remove paths which exceed 261 characters on Windows. This functionality is
exerci

lit: support long paths on Windows

Use ctypes to call into SHFileOperationW with the extended NT path to allow us
to remove paths which exceed 261 characters on Windows. This functionality is
exercised by swift's test suite.

llvm-svn: 357778

show more ...