[lit][NFC] not check stdout in googletest-timeout.py

This is flicky for buildbots (for example, https://lab.llvm.org/buildbot/#/builders/188/builds/21560)
because of the short timeout. The test cove

[lit][NFC] not check stdout in googletest-timeout.py

This is flicky for buildbots (for example, https://lab.llvm.org/buildbot/#/builders/188/builds/21560)
because of the short timeout. The test coverage is not reduced because
the stdout check is performed in other googtest-* unit tests.

show more ...

[lit] Keep stdout/stderr when using GoogleTest format

When a unit test crashes or timeout, print the shard's stdout and
stderr. When a unit test fails, attaches the test's output to the LIT
output t

[lit] Keep stdout/stderr when using GoogleTest format

When a unit test crashes or timeout, print the shard's stdout and
stderr. When a unit test fails, attaches the test's output to the LIT
output to help debugging.

While at it, concatenating shard's environment variables using space
instead of newline to make the reproducer script user friendly.

Based on D123797. (Thanks to @lenary)

show more ...

[Reland][lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtest

[Reland][lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instead of one (this is the current situation).

The shards are executed by the test runner and the results are stored in the
json format supported by the GoogleTest. Later in the test reporting stage,
all test results in the json file are retrieved to continue the test results
summary etc.

On my Win10 desktop, before this patch: `check-clang-unit`: 177s, `check-llvm-unit`: 38s; after this patch: `check-clang-unit`: 37s, `check-llvm-unit`: 11s.
On my Linux machine, before this patch: `check-clang-unit`: 46s, `check-llvm-unit`: 8s; after this patch: `check-clang-unit`: 7s, `check-llvm-unit`: 4s.

Reviewed By: yln, rnk, abrachet

Differential Revision: https://reviews.llvm.org/D122251

show more ...

Revert "Reland "[lit] Use sharding for GoogleTest format""

This reverts commit 948f3deca91a66caf4a618f826ff6de8277bed9c.

Reland "[lit] Use sharding for GoogleTest format"

This relands commit a87ba5c86d5d72defdbcdb278baad6515ec99463.

Adjust llvm/utils/lit/tests/googletest-timeout.py for new test output.

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default t

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default to using lexical order and one job in the lit test
suite.

Differential Revision: https://reviews.llvm.org/D107695

show more ...

[lit] Print full googletest commad line

Similar to regular output of LIT tests:
https://github.com/llvm/llvm-project/blob/c162f086ba632ffaedfe92d63bf21571bc8ae4da/llvm/utils/lit/lit/TestRunner.py#L1

[lit] Print full googletest commad line

Similar to regular output of LIT tests:
https://github.com/llvm/llvm-project/blob/c162f086ba632ffaedfe92d63bf21571bc8ae4da/llvm/utils/lit/lit/TestRunner.py#L1569

Differential Revision: https://reviews.llvm.org/D102899

show more ...

[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`.

The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine withi

[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`.

The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine within a short time window (1 or 2
seconds). While this "seems to work" it can fail on resource constrained
machines. We could bump the timeout a little bit (bumping it too
much would mean the test would take a long time to execute) but it wouldn't
really solve the problem of the test being prone to failures.

This patch tries to remove this flakeyness by separating testing into
two separate parts:

1. Testing if a test can hit a timeout.
2. Testing if a test can run to completion in the presence of a
timeout.

This way we can give (1.) a really short timeout (to make the test run
as fast as possible) and (2.) a really long timeout. This means for (2.)
we are no longer trying to rely on the "short" test executing within
some short time window. Instead the window is now 3600 seconds which
should be long enough even for a heavily resource constrained machine to
execute the "short" test.

Thanks to Julian Lettner for suggesting this approach. This superseeds
my original approach in https://reviews.llvm.org/D88807.

This patch also changes the command line override test to run the quick
test rather than the slow one to make the test run faster.

Differential Revision: https://reviews.llvm.org/D89020

show more ...

[lit] Improve naming of test result categories

Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing te

[lit] Improve naming of test result categories

Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines. For
example, "Failing Tests"/"Unexpected Failures". This commit changes lit
to label things consistently.

Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous. Some labels contain the word "Tests" and some don't.
Let's simplify the names.

Before:
```
Failing Tests (1):
...

Expected Passes : 3
Unexpected Failures: 1
```

After:
```
Failed Tests (1):
...

Passed: 3
Failed: 1
```

Reviewed By: ldionne

Differential Revision: https://reviews.llvm.org/D77708

show more ...

[AIX][lit] Don't depend on psutil on AIX

Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off

[AIX][lit] Don't depend on psutil on AIX

Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off
a test and it's children.

This patch adds a workaround by invoking shell via subprocess and using a
platform specific option to ps to list all the descendant processes so we can
kill them. We add some checks so lit can tell whether timeout tests are
supported with out exposing whether we are utilizing the psutil
implementation or the alternative.

Reviewers: hubert.reinterpretcast, andusy, davide, delcypher

Reviewed By: delcypher

Subscribers: davide, delcypher, christof, lldb-commits, libcxx-commits, llvm-commits

Tags: #lldb, #libc, #llvm

Differential Revision: https://reviews.llvm.org/D64251

llvm-svn: 366912

show more ...

[lit] Port googletest lit tests to Windows

Summary:
The technique of directly calling subprocess.Popen on a python script
doesn't work on Windows. The executable path of the command must refer
to a

[lit] Port googletest lit tests to Windows

Summary:
The technique of directly calling subprocess.Popen on a python script
doesn't work on Windows. The executable path of the command must refer
to a valid win32 executable.

Instead, rename all the python scripts masquerading as gtest executables
to have .py extensions, so we can easily detect then and call the python
executable for them. Do this on Linux as well as Windows for
consistency.

The test suite directory names also come out in lower-case on Windows.
We can consider removing that in a later patch. This change just updates
the FileCheck lines to match on Windows.

Fixes PR33933

Reviewers: modocache, mgorny

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35909

llvm-svn: 309347

show more ...

[lit] Mark several of lit's tests XFAIL on Windows

Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Window

[lit] Mark several of lit's tests XFAIL on Windows

Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.

In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.

Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:

```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```

Verify that the test suite is run and a successful exit code is
returned.

Reviewers: mgorny, rnk, delcypher, beanz

Reviewed By: rnk

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35879

llvm-svn: 309123

show more ...

[lit] Implement support of per test timeout in lit.

This should work with ShTest (executed externally or internally) and GTest
test formats.

To set the timeout a new option ``--timeout=`` has
been

[lit] Implement support of per test timeout in lit.

This should work with ShTest (executed externally or internally) and GTest
test formats.

To set the timeout a new option ``--timeout=`` has
been added which specifies the maximum run time of an individual test
in seconds. By default this 0 which causes no timeout to be enforced.

The timeout can also be set from a lit configuration file by modifying
the ``lit_config.maxIndividualTestTime`` property.

To implement a timeout we now require the psutil Python module if a
timeout is requested. This dependency is confined to the newly added
``lit.util.killProcessAndChildren()``. A note has been added into the
TODO document describing how we can remove the dependency on the
``pustil`` module in the future. It would be nice to remove this
immediately but that is a lot more work and Daniel Dunbar believes it is
better that we get a working implementation first and then improve it.

To avoid breaking the existing behaviour the psutil module will not be
imported if no timeout is requested.

The included testcases are derived from test cases provided by
Jonathan Roelofs which were in an previous attempt to add a per test
timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan!

Reviewers: ddunbar, jroelofs, cmatthews, MatzeB

Subscribers: cmatthews, llvm-commits

Differential Revision: http://reviews.llvm.org/D14706

llvm-svn: 256471

show more ...