[lldb][test] Add `pexpect` category for tests that `import pexpect` (#84860)

Instead of directly annotating pexpect-based tests with
`@skipIfWindows`, we can tag them with a new `pexpect` category.

[lldb][test] Add `pexpect` category for tests that `import pexpect` (#84860)

Instead of directly annotating pexpect-based tests with
`@skipIfWindows`, we can tag them with a new `pexpect` category. We
still automatically skip windows behavior by adding `pexpect` to the
skip category list if the platform is windows, but also allow
non-Windows users to skip them by configuring cmake with
`-DLLDB_TEST_USER_ARGS=--skip-category=pexpect`

As a prerequisite, remove the restriction that `@add_test_categories`
can only apply to test cases, and we make the test runner look for
categories on both the class and the test method.

show more ...

[lldb][test] Switch LLDB API tests from vendored unittest2 to unittest (#79945)

This removes the dependency LLDB API tests have on
lldb/third_party/Python/module/unittest2, and instead uses the sta

[lldb][test] Switch LLDB API tests from vendored unittest2 to unittest (#79945)

This removes the dependency LLDB API tests have on
lldb/third_party/Python/module/unittest2, and instead uses the standard
one provided by Python.

This does not actually remove the vendored dep yet, nor update the docs.
I'll do both those once this sticks.

Non-trivial changes to call out:
- expected failures (i.e. "bugnumber") don't have a reason anymore, so
those params were removed
- `assertItemsEqual` is now called `assertCountEqual`
- When a test is marked xfail, our copy of unittest2 considers failures
during teardown to be OK, but modern unittest does not. See
TestThreadLocal.py. (Very likely could be a real bug/leak).
- Our copy of unittest2 was patched to print all test results, even ones
that don't happen, e.g. `(5 passes, 0 failures, 1 errors, 0 skipped,
...)`, but standard unittest prints a terser message that omits test
result types that didn't happen, e.g. `OK (skipped=1)`. Our lit
integration parses this stderr and needs to be updated w/ that
expectation.

I tested this w/ `ninja check-lldb-api` on Linux. There's a good chance
non-Linux tests have similar quirks, but I'm not able to uncover those.

show more ...

[NFC][Py Reformat] Reformat python files in lldb

This is an ongoing series of commits that are reformatting our Python
code. Reformatting is done with `black` (23.1.0).

If you end up having problem

[NFC][Py Reformat] Reformat python files in lldb

This is an ongoing series of commits that are reformatting our Python
code. Reformatting is done with `black` (23.1.0).

If you end up having problems merging this commit because you have made
changes to a python file, the best way to handle that is to run `git
checkout --ours <yourfile>` and then reformat it with black.

RFC: https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style

Differential revision: https://reviews.llvm.org/D151460

show more ...

[lldb] [test] Rename '.categories' to 'categories'

Make category-specifying files visible. There is really no good reason
to keep them hidden, and having them visible increases the chances
that som

[lldb] [test] Rename '.categories' to 'categories'

Make category-specifying files visible. There is really no good reason
to keep them hidden, and having them visible increases the chances
that someone will actually spot them.

Differential Revision: https://reviews.llvm.org/D91065

show more ...

[lldb] Print the exception traceback when hitting cleanup errors

Right now if the test suite encounters a cleanup error it just prints "CLEANUP
ERROR:" but not any additional information.

This patc

[lldb] Print the exception traceback when hitting cleanup errors

Right now if the test suite encounters a cleanup error it just prints "CLEANUP
ERROR:" but not any additional information.

This patch just prints the exception that caused the cleanup error. This should
make debugging the failing tests for D83865 easier (and seems in general nice to
have).

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D83874

show more ...

[lldb/test] Delete result formatter machinery entirely

After more investigation, I realised this part of the code is totally
unused. It was used for communicating the test results from the
"inferior

[lldb/test] Delete result formatter machinery entirely

After more investigation, I realised this part of the code is totally
unused. It was used for communicating the test results from the
"inferior" dotest process to the main "dosep" process running
everything. Now that everything is being orchestrated through lit, this
is not used for anything.

show more ...

[lldb][tests] Make it possible to expect failure for a whole category

There already are decorators and "--excluded" option to mark test-cases/files
as expected to fail. However, when a new test file

[lldb][tests] Make it possible to expect failure for a whole category

There already are decorators and "--excluded" option to mark test-cases/files
as expected to fail. However, when a new test file is added and it which relates
to a feature that a target doesn't support, this requires either adding decorators
to that file or modifying the file provided as "--excluded" option value.

The purpose of this patch is to avoid any modifications in such cases.
E.g. if a target doesn't support "watchpoints" and passes "--xfail-category watchpoint"
to dotest, a testing job will not fail after a new watchpoint-related test file is added.

Differential Revision: https://reviews.llvm.org/D71906

show more ...

[lldb][tests][NFC] Unify variable naming convention

[lldb][test] NFC, re-use _getTestPath() function

[lldb][tests] Take into account all parent's categories when traverse folders upwards

This is needed to not re-write parent's categories by categories of a nested folder,
e.g. commands/expression/co

[lldb][tests] Take into account all parent's categories when traverse folders upwards

This is needed to not re-write parent's categories by categories of a nested folder,
e.g. commands/expression/completion specify "cmdline" category, however it still belongs
to parent's "expression" category.

The sentinel ".categories" in the test-suite root directory is no longer needed.

Differential Revision: https://reviews.llvm.org/D71905

show more ...

[lldb][NFC] Remove 'from __future__ import print_function' from all tests that don't actually call 'print()'

Summary:
A lot of tests do this trick but the vast majority of them don't even call `prin

[lldb][NFC] Remove 'from __future__ import print_function' from all tests that don't actually call 'print()'

Summary:
A lot of tests do this trick but the vast majority of them don't even call `print()`.

Most of this patch was generated by a script that just looks at all the files and deletes the line if there is no `print (` or `print(` anywhere else in the file.
I checked the remaining tests manually and deleted the import if we never call print (but instead do stuff like `expr print(...)` and similar false-positives).
I also corrected the additional empty lines after the import in the files that I manually edited.

Reviewers: JDevlieghere, labath, jfb

Reviewed By: labath

Subscribers: dexonsmith, wuzish, nemanjai, kbarton, christof, arphaman, abidh, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D71452

show more ...

[lldb][test] Handle .categories lookup for inline tests.

Summary:
When creating a test with `lldbinline.MakeInlineTest()`, the reported `inspect.getfile(test.__class__)` is `lldbtest.pyc`, meaning a

[lldb][test] Handle .categories lookup for inline tests.

Summary:
When creating a test with `lldbinline.MakeInlineTest()`, the reported `inspect.getfile(test.__class__)` is `lldbtest.pyc`, meaning any `.categories` file will be ineffective for those tests. Check for the test_filename first, which inline tests will set.

Additionally, raise an error with the starting dir if `.categories` is not found. This makes the problem more obvious when it occurs: when the test is separated from the test framework tree.

Reviewers: labath, JDevlieghere

Subscribers: lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D71099

show more ...

[dotest] Remove -q (quiet) flag.

This patch removes the -q (quiet) flag and changing the default
behavior. Currently the flag serves two purposes that are somewhat
contradictory, as illustrated by t

[dotest] Remove -q (quiet) flag.

This patch removes the -q (quiet) flag and changing the default
behavior. Currently the flag serves two purposes that are somewhat
contradictory, as illustrated by the difference between the argument
name (quiet) and the configuration flag (parsable). On the one hand it
reduces output, but on the other hand it prints more output, like the
result of individual tests. My proposal is to guard the extra output
behind the verbose flag and always print the individual test results.

Differential revision: https://reviews.llvm.org/D66837

llvm-svn: 370226

show more ...

[lldb][NFC] Remove unused imports in python tests

llvm-svn: 367663

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Remove vestigial remnants of the test crash info hook

llvm-svn: 325137

[dotest] Remove crashinfo hook

Summary:
This used to be important when all tests were run in a single process,
but that has no longer been the case for a while. Furthermore, this hook fails
to build

[dotest] Remove crashinfo hook

Summary:
This used to be important when all tests were run in a single process,
but that has no longer been the case for a while. Furthermore, this hook fails
to build on new mac versions for several people, and it's not clear
whether fixing it is worth the effort.

Reviewers: jingham, clayborg, davide

Subscribers: lldb-commits

Differential Revision: https://reviews.llvm.org/D41871

llvm-svn: 322167

show more ...

Remove getCategories mechanism of specifying test categories

Summary:
This mechanism was mostly redundant with the file-based .categories
mechanism, and it was interfering with it, as any test which

Remove getCategories mechanism of specifying test categories

Summary:
This mechanism was mostly redundant with the file-based .categories
mechanism, and it was interfering with it, as any test which implemented
a getCategories method would not inherit the filesystem categories.

This patch removes it. The existing categories are preserved either by
adding a .categories file, or using the @add_test_categories decorator.

Reviewers: jingham, clayborg, zturner

Subscribers: lldb-commits

Differential Revision: https://reviews.llvm.org/D39515

llvm-svn: 317277

show more ...

Improvements to testing blacklist

Summary:
This patch is necessary because individual test cases are not required
to have unique names. Therefore, test cases must now
be specified explicitly in the

Improvements to testing blacklist

Summary:
This patch is necessary because individual test cases are not required
to have unique names. Therefore, test cases must now
be specified explicitly in the form <TestCase>.<TestMethod>.
Because it works by regex matching, passing just <TestCase> will
still disable an entire file.

This also allows for multiple exclusion files to be specified.

Reviewers: zturner, labath, jingham, tfiala

Subscribers: lldb-commits, sas

Differential Revision: https://reviews.llvm.org/D24988

llvm-svn: 283238

show more ...

Allow for tests to be disabled at runtime

Summary:
The current implementation of the test suite allows the user to run
a certain subset of tests using '-p', but does not allow the inverse,
where a u

Allow for tests to be disabled at runtime

Summary:
The current implementation of the test suite allows the user to run
a certain subset of tests using '-p', but does not allow the inverse,
where a user wants to run all but some number of known failing tests.
Implement this functionality.

Reviewers: labath, zturner, tfiala

Subscribers: jingham, sas, lldb-commits

Differential Revision: https://reviews.llvm.org/D24629

llvm-svn: 282298

show more ...

*** This commit represents a complete reformatting of the LLDB source code
*** to conform to clang-format’s LLVM style. This kind of mass change has
*** two obvious implications:

Firstly, merging t

*** This commit represents a complete reformatting of the LLDB source code
*** to conform to clang-format’s LLVM style. This kind of mass change has
*** two obvious implications:

Firstly, merging this particular commit into a downstream fork may be a huge
effort. Alternatively, it may be worth merging all changes up to this commit,
performing the same reformatting operation locally, and then discarding the
merge for this particular commit. The commands used to accomplish this
reformatting were as follows (with current working directory as the root of
the repository):

find . \( -iname "*.c" -or -iname "*.cpp" -or -iname "*.h" -or -iname "*.mm" \) -exec clang-format -i {} +
find . -iname "*.py" -exec autopep8 --in-place --aggressive --aggressive {} + ;

The version of clang-format used was 3.9.0, and autopep8 was 1.2.4.

Secondly, “blame” style tools will generally point to this commit instead of
a meaningful prior commit. There are alternatives available that will attempt
to look through this change and find the appropriate prior commit. YMMV.

llvm-svn: 280751

show more ...

[test] Fix category-based skipping

Summary:
LLDBTestResult.hardMarkAsSkipped marked the whole class as skipped when the first class in the
test failed the category check. This meant that subsequent

[test] Fix category-based skipping

Summary:
LLDBTestResult.hardMarkAsSkipped marked the whole class as skipped when the first class in the
test failed the category check. This meant that subsequent tests in the same class did not run
even if they were passing the category filter. Fix that.

Reviewers: tfiala

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D22213

llvm-svn: 275173

show more ...

surface build error content through test event system

Summary:
print build errors nicely in test output

This test infrastructure change adds a new Python exception
for test subject builds that fail

surface build error content through test event system

Summary:
print build errors nicely in test output

This test infrastructure change adds a new Python exception
for test subject builds that fail. The output of the build
command is captured and propagated to both the textual test
output display code and to the test event system.

The ResultsFormatter objects have been modified to do something
more useful with this information. The xUnit formatter
now replaces the non-informative Python build error stacktrace
with the build error content. The curses ResultsFormatter
prints a 'B' for build errors rather than 'E'.

The xUnit output, in particular, makes it much easier for
developers to track down test subject build errors that cause
test failures when reports come in from CI.

Reviewers: granata.enrico

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D20252

llvm-svn: 269525

show more ...

test infra: move test event-related handling into its own package

This change moves all the test event handling and its related
ResultsFormatter classes out of the packages/Python/lldbsuite/test dir

test infra: move test event-related handling into its own package

This change moves all the test event handling and its related
ResultsFormatter classes out of the packages/Python/lldbsuite/test dir
into a packages/Python/lldbsuite/test_event package. Formatters are
moved into a sub-package under that.

I am limiting the scope of this change to just the motion and a few
minor issues caught by a static Python checker (e.g. removing unused
import statements).

This is a pre-step for adding package-level tests to the test event
system. I also intend to simplify test event results formatter selection
after I make sure this doesn't break anybody.

See:
http://reviews.llvm.org/D19288

Reviewed by:
Pavel Labath

llvm-svn: 266885

show more ...

Make test categories composable

Summary:
Previously the add_test_categories would simply overwrite the current set of categories for a
method. This change makes the decorator truly "add" categories,

Make test categories composable

Summary:
Previously the add_test_categories would simply overwrite the current set of categories for a
method. This change makes the decorator truly "add" categories, by extending the current set of
categories instead of replacing it.

To do this, I have:
- replaced the getCategories() property on a method (which was itself a method), with a simple
list property "categories". This makes add_test_categories easier to implement, and test
categories isn't something which should change between calls anyway.
- rewritten the getCategoriesForTest function to merge method categories with the categories of
the test case. Previously, it would just use the method categories if they were present. I have
also greatly simplified this method. Originally, it would use a lot of introspection to enable
it being called on various types of objects. Based on my tests, it was only ever being called
on a test case. The new function uses much less introspection then the preivous one, so we
should easily catch any stray uses, if there are any, as they will generate exceptions now.

Reviewers: zturner, tfiala, tberghammer

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D15451

llvm-svn: 255493

show more ...