|
Revision tags: llvmorg-18.1.8, llvmorg-18.1.7, llvmorg-18.1.6, llvmorg-18.1.5, llvmorg-18.1.4, llvmorg-18.1.3, llvmorg-18.1.2, llvmorg-18.1.1, llvmorg-18.1.0, llvmorg-18.1.0-rc4, llvmorg-18.1.0-rc3, llvmorg-18.1.0-rc2, llvmorg-18.1.0-rc1, llvmorg-19-init, llvmorg-17.0.6, llvmorg-17.0.5, llvmorg-17.0.4, llvmorg-17.0.3, llvmorg-17.0.2, llvmorg-17.0.1, llvmorg-17.0.0, llvmorg-17.0.0-rc4, llvmorg-17.0.0-rc3, llvmorg-17.0.0-rc2, llvmorg-17.0.0-rc1, llvmorg-18-init, llvmorg-16.0.6, llvmorg-16.0.5, llvmorg-16.0.4, llvmorg-16.0.3, llvmorg-16.0.2, llvmorg-16.0.1, llvmorg-16.0.0, llvmorg-16.0.0-rc4, llvmorg-16.0.0-rc3, llvmorg-16.0.0-rc2, llvmorg-16.0.0-rc1, llvmorg-17-init, llvmorg-15.0.7, llvmorg-15.0.6, llvmorg-15.0.5, llvmorg-15.0.4, llvmorg-15.0.3, working, llvmorg-15.0.2, llvmorg-15.0.1, llvmorg-15.0.0, llvmorg-15.0.0-rc3, llvmorg-15.0.0-rc2, llvmorg-15.0.0-rc1, llvmorg-16-init, llvmorg-14.0.6, llvmorg-14.0.5, llvmorg-14.0.4, llvmorg-14.0.3, llvmorg-14.0.2, llvmorg-14.0.1, llvmorg-14.0.0, llvmorg-14.0.0-rc4, llvmorg-14.0.0-rc3, llvmorg-14.0.0-rc2, llvmorg-14.0.0-rc1, llvmorg-15-init, llvmorg-13.0.1, llvmorg-13.0.1-rc3, llvmorg-13.0.1-rc2, llvmorg-13.0.1-rc1, llvmorg-13.0.0, llvmorg-13.0.0-rc4, llvmorg-13.0.0-rc3, llvmorg-13.0.0-rc2, llvmorg-13.0.0-rc1, llvmorg-14-init, llvmorg-12.0.1, llvmorg-12.0.1-rc4, llvmorg-12.0.1-rc3, llvmorg-12.0.1-rc2, llvmorg-12.0.1-rc1, llvmorg-12.0.0, llvmorg-12.0.0-rc5, llvmorg-12.0.0-rc4, llvmorg-12.0.0-rc3, llvmorg-12.0.0-rc2, llvmorg-11.1.0, llvmorg-11.1.0-rc3, llvmorg-12.0.0-rc1, llvmorg-13-init, llvmorg-11.1.0-rc2, llvmorg-11.1.0-rc1, llvmorg-11.0.1, llvmorg-11.0.1-rc2 |
|
| #
44156787 |
| 08-Dec-2020 |
David Blaikie <dblaikie@gmail.com> |
Fix shtest-timeout test harder
Don't produce or expect any output from the infinite looping test -
doing so is a recipe for racey flakyness without a longer timeout to
ensure the output is received
Fix shtest-timeout test harder
Don't produce or expect any output from the infinite looping test -
doing so is a recipe for racey flakyness without a longer timeout to
ensure the output is received first, even though that doesn't seem
integral/important to the test. Instead have a plain, no output infinite
loop and check that that is caught and handled.
If for some reason the output is valuable for test coverage - the
timeout should be increased from 1 second to give the process time to
output the text, flush, and for that text to be received and buffered
before the test is timed out.
show more ...
|
| #
7d04e706 |
| 07-Dec-2020 |
Oliver Stannard <oliver.stannard@linaro.org> |
[Lit] Fix flaky test on heavily loaded bots
On some of the slow or heavily-loaded bots, this test was failing
intermittently because the infinite_loop.py script might not emit
anything to stdout bef
[Lit] Fix flaky test on heavily loaded bots
On some of the slow or heavily-loaded bots, this test was failing
intermittently because the infinite_loop.py script might not emit
anything to stdout before the 1 second timeout, so the "Command Output"
line isn't present in the output. That output isn't really important to
this test, we just care that the process is killed, so we can just rmove
that check line from the test.
Differential revision: https://reviews.llvm.org/D92563
show more ...
|
|
Revision tags: llvmorg-11.0.1-rc1 |
|
| #
295d4e42 |
| 07-Oct-2020 |
Dan Liew <dliew@apple.com> |
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`.
The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine withi
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`.
The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine within a short time window (1 or 2
seconds). While this "seems to work" it can fail on resource constrained
machines. We could bump the timeout a little bit (bumping it too
much would mean the test would take a long time to execute) but it wouldn't
really solve the problem of the test being prone to failures.
This patch tries to remove this flakeyness by separating testing into
two separate parts:
1. Testing if a test can hit a timeout.
2. Testing if a test can run to completion in the presence of a
timeout.
This way we can give (1.) a really short timeout (to make the test run
as fast as possible) and (2.) a really long timeout. This means for (2.)
we are no longer trying to rely on the "short" test executing within
some short time window. Instead the window is now 3600 seconds which
should be long enough even for a heavily resource constrained machine to
execute the "short" test.
Thanks to Julian Lettner for suggesting this approach. This superseeds
my original approach in https://reviews.llvm.org/D88807.
This patch also changes the command line override test to run the quick
test rather than the slow one to make the test run faster.
Differential Revision: https://reviews.llvm.org/D89020
show more ...
|
|
Revision tags: llvmorg-11.0.0, llvmorg-11.0.0-rc6, llvmorg-11.0.0-rc5, llvmorg-11.0.0-rc4, llvmorg-11.0.0-rc3, llvmorg-11.0.0-rc2, llvmorg-11.0.0-rc1, llvmorg-12-init, llvmorg-10.0.1, llvmorg-10.0.1-rc4, llvmorg-10.0.1-rc3, llvmorg-10.0.1-rc2, llvmorg-10.0.1-rc1 |
|
| #
99d6e05e |
| 08-Apr-2020 |
Julian Lettner <julian.lettner@apple.com> |
[lit] Improve naming of test result categories
Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing te
[lit] Improve naming of test result categories
Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines. For
example, "Failing Tests"/"Unexpected Failures". This commit changes lit
to label things consistently.
Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous. Some labels contain the word "Tests" and some don't.
Let's simplify the names.
Before:
```
Failing Tests (1):
...
Expected Passes : 3
Unexpected Failures: 1
```
After:
```
Failed Tests (1):
...
Passed: 3
Failed: 1
```
Reviewed By: ldionne
Differential Revision: https://reviews.llvm.org/D77708
show more ...
|
|
Revision tags: llvmorg-10.0.0, llvmorg-10.0.0-rc6, llvmorg-10.0.0-rc5, llvmorg-10.0.0-rc4, llvmorg-10.0.0-rc3, llvmorg-10.0.0-rc2, llvmorg-10.0.0-rc1, llvmorg-11-init, llvmorg-9.0.1, llvmorg-9.0.1-rc3, llvmorg-9.0.1-rc2, llvmorg-9.0.1-rc1, llvmorg-9.0.0, llvmorg-9.0.0-rc6, llvmorg-9.0.0-rc5, llvmorg-9.0.0-rc4, llvmorg-9.0.0-rc3, llvmorg-9.0.0-rc2, llvmorg-9.0.0-rc1 |
|
| #
b9179ff8 |
| 24-Jul-2019 |
David Tenty <daltenty@ibm.com> |
[AIX][lit] Don't depend on psutil on AIX
Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off
[AIX][lit] Don't depend on psutil on AIX
Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off
a test and it's children.
This patch adds a workaround by invoking shell via subprocess and using a
platform specific option to ps to list all the descendant processes so we can
kill them. We add some checks so lit can tell whether timeout tests are
supported with out exposing whether we are utilizing the psutil
implementation or the alternative.
Reviewers: hubert.reinterpretcast, andusy, davide, delcypher
Reviewed By: delcypher
Subscribers: davide, delcypher, christof, lldb-commits, libcxx-commits, llvm-commits
Tags: #lldb, #libc, #llvm
Differential Revision: https://reviews.llvm.org/D64251
llvm-svn: 366912
show more ...
|
|
Revision tags: llvmorg-10-init, llvmorg-8.0.1, llvmorg-8.0.1-rc4, llvmorg-8.0.1-rc3, llvmorg-8.0.1-rc2 |
|
| #
83c28abd |
| 29-May-2019 |
Stella Stamenova <stilis@microsoft.com> |
lit: modernize the lit configuration for the lit tests
Summary: This also normalizes the config feature that represents the windows platform to "system-windows" as opposed to having both "windows" a
lit: modernize the lit configuration for the lit tests
Summary: This also normalizes the config feature that represents the windows platform to "system-windows" as opposed to having both "windows" and "system-windows"
Reviewers: asmith, probinson
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61798
llvm-svn: 361998
show more ...
|
|
Revision tags: llvmorg-8.0.1-rc1 |
|
| #
46b9d19c |
| 09-May-2019 |
Stella Stamenova <stilis@microsoft.com> |
Use UNSUPPORTED: windows in shtest-timeout.py. Apparently system-windows does not cover all cases either and the case it doesn't cover affects one of the buildbots.
llvm-svn: 360373
|
| #
bc9e0866 |
| 09-May-2019 |
Stella Stamenova <stilis@microsoft.com> |
Use UNSUPPORTED: system-windows instead of REQUIRES: nowindows or UNSUPPORTED: windows. nowindows is not currently defined and windows does not cover all cases. system-windows is also consistent with
Use UNSUPPORTED: system-windows instead of REQUIRES: nowindows or UNSUPPORTED: windows. nowindows is not currently defined and windows does not cover all cases. system-windows is also consistent with how other platforms are used.
llvm-svn: 360368
show more ...
|
| #
0b68fc3f |
| 09-May-2019 |
Paul Robinson <paul.robinson@sony.com> |
Re-enable lit test shtest-timeout.py on non-Windows.
It was disabled incorrectly, which meant it wasn't running anywhere.
llvm-svn: 360356
|
|
Revision tags: llvmorg-8.0.0, llvmorg-8.0.0-rc5, llvmorg-8.0.0-rc4, llvmorg-8.0.0-rc3, llvmorg-7.1.0, llvmorg-7.1.0-rc1, llvmorg-8.0.0-rc2, llvmorg-8.0.0-rc1, llvmorg-7.0.1, llvmorg-7.0.1-rc3, llvmorg-7.0.1-rc2, llvmorg-7.0.1-rc1, llvmorg-7.0.0 |
|
| #
4f780ea1 |
| 10-Sep-2018 |
Stella Stamenova <stilis@microsoft.com> |
[lit] Disable shtest-timeout on Windows
Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparin
[lit] Disable shtest-timeout on Windows
Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparing causing a failure, so I'm going to disable it. Since the test has rarely, if ever, passed on the bots, this should have the same effect and it will unblock the creation of the new bot.
Reviewers: asmith, delcypher, zturner
Subscribers: stella.stamenova, llvm-commits
Differential Revision: https://reviews.llvm.org/D51871
llvm-svn: 341856
show more ...
|
|
Revision tags: llvmorg-7.0.0-rc3, llvmorg-7.0.0-rc2 |
|
| #
4f03fb1c |
| 07-Aug-2018 |
Stella Stamenova <stilis@microsoft.com> |
[lit] Disable shtest-timeout on Windows
This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python
[lit] Disable shtest-timeout on Windows
This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python 2.
llvm-svn: 339184
show more ...
|
| #
8efc95ac |
| 07-Aug-2018 |
Stella Stamenova <stilis@microsoft.com> |
[lit, python3] Update lit error logging to work correctly in python3 and other test fixes
Summary:
In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and ins
[lit, python3] Update lit error logging to work correctly in python3 and other test fixes
Summary:
In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and instead all 'str' objects are Unicode string. This change updates the logic in the test logging for lit to correctly process each of the types, and more importantly, to not just fail in Python3.
This change also reverses the use of quotes in several of the cfg files. By using '""' we are guaranteeing that the resulting path will work correctly on Windows while "''" only works correctly sometimes. This also fixes one of the failing tests.
Reviewers: asmith, zturner
Subscribers: stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50397
llvm-svn: 339179
show more ...
|
|
Revision tags: llvmorg-7.0.0-rc1, llvmorg-6.0.1, llvmorg-6.0.1-rc3, llvmorg-6.0.1-rc2 |
|
| #
6dfcc783 |
| 22-May-2018 |
Dan Liew <dan@su-root.co.uk> |
[lit] Try to make `shtest-timeout.py` test more reliable by using a
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.
While we are here a
[lit] Try to make `shtest-timeout.py` test more reliable by using a
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.
While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.
rdar://problem/40221572
llvm-svn: 332987
show more ...
|
| #
9f778685 |
| 22-May-2018 |
Dan Liew <dan@su-root.co.uk> |
[lit] Don't run `slow.py` in `shtest-timeout.py` test.
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't
[lit] Don't run `slow.py` in `shtest-timeout.py` test.
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.
rdar://problem/40221572
llvm-svn: 332986
show more ...
|
| #
8b4d36a5 |
| 22-May-2018 |
Dan Liew <dan@su-root.co.uk> |
[lit] Don't check output of commands used in `shtest-timeout.py` test.
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test
[lit] Don't check output of commands used in `shtest-timeout.py` test.
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.
rdar://problem/40221572
llvm-svn: 332985
show more ...
|
|
Revision tags: llvmorg-6.0.1-rc1, llvmorg-5.0.2, llvmorg-5.0.2-rc2 |
|
| #
8ade9e75 |
| 28-Mar-2018 |
Dan Liew <dan@su-root.co.uk> |
Revert "[lit] Temporarily disable shtest-timeout.py on darwin"
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)
Hopefully the test will now pass on the bots.
rdar://problem/
Revert "[lit] Temporarily disable shtest-timeout.py on darwin"
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)
Hopefully the test will now pass on the bots.
rdar://problem/38774530
llvm-svn: 328703
show more ...
|
| #
7efde3c4 |
| 28-Mar-2018 |
Dan Liew <dan@su-root.co.uk> |
[lit] Remove a timing senstive part of `shtest-timeout.py`
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to
[lit] Remove a timing senstive part of `shtest-timeout.py`
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.
The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.
We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.
rdar://problem/38774530
llvm-svn: 328702
show more ...
|
| #
1e0e0b07 |
| 27-Mar-2018 |
Jan Korous <jkorous@apple.com> |
[lit] Temporarily disable shtest-timeout.py on darwin
Disabled until fixed in order to avoid random failures on green dragon.
rdar://problem/38774530
llvm-svn: 328598
|
|
Revision tags: llvmorg-5.0.2-rc1, llvmorg-6.0.0, llvmorg-6.0.0-rc3, llvmorg-6.0.0-rc2, llvmorg-6.0.0-rc1, llvmorg-5.0.1, llvmorg-5.0.1-rc3, llvmorg-5.0.1-rc2, llvmorg-5.0.1-rc1, llvmorg-5.0.0, llvmorg-5.0.0-rc5, llvmorg-5.0.0-rc4, llvmorg-5.0.0-rc3, llvmorg-5.0.0-rc2, llvmorg-5.0.0-rc1 |
|
| #
0787253c |
| 26-Jul-2017 |
Brian Gesiak <modocache@gmail.com> |
[lit] Mark several of lit's tests XFAIL on Windows
Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Window
[lit] Mark several of lit's tests XFAIL on Windows
Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.
In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.
Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:
```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```
Verify that the test suite is run and a successful exit code is
returned.
Reviewers: mgorny, rnk, delcypher, beanz
Reviewed By: rnk
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35879
llvm-svn: 309123
show more ...
|
|
Revision tags: llvmorg-4.0.1, llvmorg-4.0.1-rc3, llvmorg-4.0.1-rc2, llvmorg-4.0.1-rc1, llvmorg-4.0.0, llvmorg-4.0.0-rc4, llvmorg-4.0.0-rc3, llvmorg-4.0.0-rc2, llvmorg-4.0.0-rc1, llvmorg-3.9.1, llvmorg-3.9.1-rc3, llvmorg-3.9.1-rc2, llvmorg-3.9.1-rc1 |
|
| #
1d62f4ab |
| 12-Oct-2016 |
Michal Gorny <mgorny@gentoo.org> |
[lit] Fix test shtest-timeout.py for modern output
Update the CHECK lines in the shtest-timeout.py lit test to account for
the current output. The output has been changed in r271610 without
adjustin
[lit] Fix test shtest-timeout.py for modern output
Update the CHECK lines in the shtest-timeout.py lit test to account for
the current output. The output has been changed in r271610 without
adjusting the tests.
Differential Revision: https://reviews.llvm.org/D25236
llvm-svn: 284057
show more ...
|
|
Revision tags: llvmorg-3.9.0, llvmorg-3.9.0-rc3, llvmorg-3.9.0-rc2, llvmorg-3.9.0-rc1, llvmorg-3.8.1, llvmorg-3.8.1-rc1, llvmorg-3.8.0, llvmorg-3.8.0-rc3, llvmorg-3.8.0-rc2, llvmorg-3.8.0-rc1 |
|
| #
75742410 |
| 27-Dec-2015 |
Dan Liew <dan@su-root.co.uk> |
[lit] Implement support of per test timeout in lit.
This should work with ShTest (executed externally or internally) and GTest
test formats.
To set the timeout a new option ``--timeout=`` has
been
[lit] Implement support of per test timeout in lit.
This should work with ShTest (executed externally or internally) and GTest
test formats.
To set the timeout a new option ``--timeout=`` has
been added which specifies the maximum run time of an individual test
in seconds. By default this 0 which causes no timeout to be enforced.
The timeout can also be set from a lit configuration file by modifying
the ``lit_config.maxIndividualTestTime`` property.
To implement a timeout we now require the psutil Python module if a
timeout is requested. This dependency is confined to the newly added
``lit.util.killProcessAndChildren()``. A note has been added into the
TODO document describing how we can remove the dependency on the
``pustil`` module in the future. It would be nice to remove this
immediately but that is a lot more work and Daniel Dunbar believes it is
better that we get a working implementation first and then improve it.
To avoid breaking the existing behaviour the psutil module will not be
imported if no timeout is requested.
The included testcases are derived from test cases provided by
Jonathan Roelofs which were in an previous attempt to add a per test
timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan!
Reviewers: ddunbar, jroelofs, cmatthews, MatzeB
Subscribers: cmatthews, llvm-commits
Differential Revision: http://reviews.llvm.org/D14706
llvm-svn: 256471
show more ...
|