[test] Fix expected line number in `TestRunner.py` (#73996)

My #73541 added lines to
`llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt` so what
was previously on line 7 is now on lin

[test] Fix expected line number in `TestRunner.py` (#73996)

My #73541 added lines to
`llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt` so what
was previously on line 7 is now on line 12.

Before:

https://github.com/llvm/llvm-project/blob/28412d1800e391c5ba8e7607bb15c74b106d581b/llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt#L7-L8

After:

https://github.com/llvm/llvm-project/blob/6fb7c2d713587a061cd281eda917746750559380/llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt#L12-L13

This didn't show up in the PR checks, but caused a buildbot failure
after merging, https://lab.llvm.org/buildbot/#/builders/139/builds/54597
:

```
# | ======================================================================
# | FAIL: test_commands (__main__.TestIntegratedTestKeywordParser)
# | ----------------------------------------------------------------------
# | Traceback (most recent call last):
# | File "/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/utils/lit/tests/unit/TestRunner.py", line 135, in test_commands
# | self.assertEqual(value[1].command.strip(), "%dbg(MY_RUN: at line 7) foo bar")
# | AssertionError: '%dbg(MY_RUN: at line 12) foo bar' != '%dbg(MY_RUN: at line 7) foo bar'
# | - %dbg(MY_RUN: at line 12) foo bar
# | ? ^^
# | + %dbg(MY_RUN: at line 7) foo bar
# | ? ^
```

Apologies for the build break :scream_cat:

show more ...

[libc++][test] `ADDITIONAL_COMPILE_FLAGS` should be a space-separated list (#73541)

Found while running libc++'s test suite with MSVC's STL.

`ADDITIONAL_COMPILE_FLAGS` is a `ParserKind.LIST`:

[libc++][test] `ADDITIONAL_COMPILE_FLAGS` should be a space-separated list (#73541)

Found while running libc++'s test suite with MSVC's STL.

`ADDITIONAL_COMPILE_FLAGS` is a `ParserKind.LIST`:

https://github.com/llvm/llvm-project/blob/3c23ed156f0151923b168bdff0c34ec73fb37f38/libcxx/utils/libcxx/test/format.py#L104-L108

With a comma-separated example:

https://github.com/llvm/llvm-project/blob/3c23ed156f0151923b168bdff0c34ec73fb37f38/libcxx/utils/libcxx/test/format.py#L223-L228

And comma-separated test coverage:

https://github.com/llvm/llvm-project/blob/dd3184c30ff531b8aecea280e65233337dd02815/libcxx/test/libcxx/selftest/additional_compile_flags/substitutes-in-run.sh.cpp#L12-L15

Because the machinery splits on commas:

https://github.com/llvm/llvm-project/blob/dd09221a29506031415cad8a1308998358633d48/llvm/utils/lit/lit/TestRunner.py#L1882-L1883

https://github.com/llvm/llvm-project/blob/dd09221a29506031415cad8a1308998358633d48/llvm/utils/lit/lit/TestRunner.py#L1951-L1956

However, most (although not all) usage of `ADDITIONAL_COMPILE_FLAGS` is
treating it as space-separated. That apparently works in the normal
Clang environment, but in my exotic configuration it causes `"-DMEOW
-DWOOF"` to be passed as a single argument to MSVC, which then emits
"warning C5102: ignoring invalid command-line macro definition
`'_LIBCPP_DISABLE_DEPRECATION_WARNINGS
-D_LIBCPP_ENABLE_CXX26_REMOVED_CODECVT'`", causing test failures due to
warnings-as-errors.

This PR changes `ADDITIONAL_COMPILE_FLAGS` to actually be parsed as a
space-separated list, and changes the few uses/examples that had commas.

show more ...

[lit] Remove the --no-indirectly-run-check option

This option had originally been added in D83069 to allow disabling the
check that something is going to get run at all when a specific test name
is

[lit] Remove the --no-indirectly-run-check option

This option had originally been added in D83069 to allow disabling the
check that something is going to get run at all when a specific test name
is used on the command-line. Since we now use getTestsForPath() (from D151664)
to get the tests to run for a specific path, we don't need a specific check
for this anymore -- Lit will produce the same complaint it would produce if
you provided a directory with no tests.

If one needs to run a specific test on the command-line and the Lit
configuration would normally not include that test, the configuration
should be set up as a "standalone" configuration or it should be fixed
to allow for that test to be found (i.e. probably fix the allowed test
suffixes).

Differential Revision: https://reviews.llvm.org/D153967

show more ...

[NFC][Py Reformat] Reformat python files in llvm

This is the first commit in a series that will reformat
all the python files in the LLVM repository.

Reformatting is done with `black`.

See more in

[NFC][Py Reformat] Reformat python files in llvm

This is the first commit in a series that will reformat
all the python files in the LLVM repository.

Reformatting is done with `black`.

See more information here:

https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style

Reviewed By: jhenderson, JDevlieghere, MatzeB

Differential Revision: https://reviews.llvm.org/D150545

show more ...

[lit] Implement DEFINE and REDEFINE directives

These directives define per-test lit substitutions. The concept was
discussed at
<https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>.

For

[lit] Implement DEFINE and REDEFINE directives

These directives define per-test lit substitutions. The concept was
discussed at
<https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>.

For example, the following directives can be inserted into a test file
to define `%{cflags}` and `%{fcflags}` substitutions with empty
initial values, which serve as the parameters of another newly defined
`%{check}` substitution:

```
// DEFINE: %{cflags} =
// DEFINE: %{fcflags} =

// DEFINE: %{check} = %clang_cc1 %{cflags} -emit-llvm -o - %s | \
// DEFINE: FileCheck %{fcflags} %s
```

The following directives then redefine the parameters before each use
of `%{check}`:

```
// REDEFINE: %{cflags} = -foo
// REDEFINE: %{fcflags} = -check-prefix=FOO
// RUN: %{check}

// REDEFINE: %{cflags} = -bar
// REDEFINE: %{fcflags} = -check-prefix=BAR
// RUN: %{check}
```

Of course, `%{check}` would typically be more elaborate, increasing
the benefit of the reuse.

One issue is that the strings `DEFINE:` and `REDEFINE:` already appear
in 5 tests. This patch adjusts those tests not to use those strings.
Our prediction is that, in the vast majority of cases, if a test
author mistakenly uses one of those strings for another purpose, the
text appearing after the string will not happen to have the syntax
required for these directives. Thus, the test author will discover
the mistake immediately when lit reports the syntax error.

This patch also expands the documentation on existing lit substitution
behavior.

Reviewed By: jhenderson, MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D132513

show more ...

[Reland][lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtest

[Reland][lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instead of one (this is the current situation).

The shards are executed by the test runner and the results are stored in the
json format supported by the GoogleTest. Later in the test reporting stage,
all test results in the json file are retrieved to continue the test results
summary etc.

On my Win10 desktop, before this patch: `check-clang-unit`: 177s, `check-llvm-unit`: 38s; after this patch: `check-clang-unit`: 37s, `check-llvm-unit`: 11s.
On my Linux machine, before this patch: `check-clang-unit`: 46s, `check-llvm-unit`: 8s; after this patch: `check-clang-unit`: 7s, `check-llvm-unit`: 4s.

Reviewed By: yln, rnk, abrachet

Differential Revision: https://reviews.llvm.org/D122251

show more ...

[LIT] fixed up accidentally committed EOF problems.

[LIT] error if directly named test won't be run indirectly

Currently, a LIT test named directly (on the command line) will
be run even if the name of the test file does not meet the rules
to be cons

[LIT] error if directly named test won't be run indirectly

Currently, a LIT test named directly (on the command line) will
be run even if the name of the test file does not meet the rules
to be considered a test in the LIT test configuration files for
its test suite. For example, if the test does not have a
recognised file extension.

This makes it relatively easy to write a LIT test that won't
actually be run. I did in: https://reviews.llvm.org/D82567

This patch adds an error to avoid users doing that. There is a
small performance overhead for this check. A command line option
has been added so that users can opt into the old behaviour.

Differential Revision: https://reviews.llvm.org/D83069

show more ...

[lit] Fix handling of various keyword parse errors

In TestRunner.py, D78589 extracts a `_parseKeywords` function from
`parseIntegratedTestScript`, which then expects `_parseKeywords` to
always retur

[lit] Fix handling of various keyword parse errors

In TestRunner.py, D78589 extracts a `_parseKeywords` function from
`parseIntegratedTestScript`, which then expects `_parseKeywords` to
always return a list of keyword/value pairs. However, the extracted
code sometimes returns an unresolved `lit.Test.Result` on a keyword
parsing error, which then produces a stack dump instead of the
expected diagnostic.

This patch fixes that, makes the style of those diagnostics more
consistent, and extends the lit test suite to cover them.

Reviewed By: ldionne

Differential Revision: https://reviews.llvm.org/D81665

show more ...

[lit] Recursively expand substitutions

This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cx

[lit] Recursively expand substitutions

This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.

Differential Revision: https://reviews.llvm.org/D76178

show more ...

[lit] Add builtin support for flaky tests in lit

This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
n

[lit] Add builtin support for flaky tests in lit

This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.

This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.

Differential Revision: https://reviews.llvm.org/D76288

show more ...

[lit] Small cleanups. NFCI

Remove unnecessary (argument same as default), cleanup imports, use
"pythonic" names for variables, and general formatting.

[lit] Remove unnecessary usage of lit.Run

llvm-svn: 375056

[lit] Set --single-process for single tests and --threads=1

Summary:
Automatically upgrade debugging experience (single process, no thread
pool) when:
1) we only run a single test
2) user specif

[lit] Set --single-process for single tests and --threads=1

Summary:
Automatically upgrade debugging experience (single process, no thread
pool) when:
1) we only run a single test
2) user specifies `-j1`

Details:
Fix `--max-failures` in single process mode. Option did not have an
effect in single process mode.

Add display feedback for single process mode. Adapted test.

Improve argument checking (require positive integers).

`--single-process` is now essentially an alias for `-j1`. Should we
remove it?

Reviewers: rnk

Differential Revision: https://reviews.llvm.org/D58249

llvm-svn: 354068

show more ...

make XFAIL, REQUIRES, and UNSUPPORTED support multi-line expressions

llvm-svn: 351668

[lit] Terminate ": RUN at line N" with ";" not "&&"

This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614. The trouble was that "&&" changes the command for
which "$!

[lit] Terminate ": RUN at line N" with ";" not "&&"

This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614. The trouble was that "&&" changes the command for
which "$!" gives the pid.

llvm-svn: 333620

show more ...

[lit] Report line number for failed RUN command

(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locat

[lit] Report line number for failed RUN command

(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333614

show more ...

Revert r333584: [lit] Report line number for failed RUN command

It breaks test-suite.

llvm-svn: 333592

[lit] Report line number for failed RUN command

(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this m

[lit] Report line number for failed RUN command

(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333584

show more ...

Revert r330755 "[lit] Report line number for failed RUN command"

It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211

llvm-svn:

Revert r330755 "[lit] Report line number for failed RUN command"

It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211

llvm-svn: 330848

show more ...

[lit] Report line number for failed RUN command

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example

[lit] Report line number for failed RUN command

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

Reviewed By: asmith, delcypher

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 330755

show more ...

[lit] Fix the lit unit tests.

A few tests were manually constructing a LitConfig object, since
I added a new argument to it this was triggering some failures
I didn't detect. `ninja check-lit` pass

[lit] Fix the lit unit tests.

A few tests were manually constructing a LitConfig object, since
I added a new argument to it this was triggering some failures
I didn't detect. `ninja check-lit` passes now.

llvm-svn: 313461

show more ...

Un-XFAIL some internal lit tests on Windows, they pass for me locally

llvm-svn: 309144

[lit] Mark several of lit's tests XFAIL on Windows

Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Window

[lit] Mark several of lit's tests XFAIL on Windows

Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.

In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.

Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:

```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```

Verify that the test suite is run and a successful exit code is
returned.

Reviewers: mgorny, rnk, delcypher, beanz

Reviewed By: rnk

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35879

llvm-svn: 309123

show more ...

lit: remove python2-isms

Summary:
`assert.assertItemEqual` went away in Python 3. Seeing how lists
are ordered, comparing a list against each other should work just
as well.

Patch by @jbergstroem (

lit: remove python2-isms

Summary:
`assert.assertItemEqual` went away in Python 3. Seeing how lists
are ordered, comparing a list against each other should work just
as well.

Patch by @jbergstroem (Johan Bergström).

Reviewers: modocache, gparker42

Reviewed By: modocache

Differential Revision: https://reviews.llvm.org/D31229

llvm-svn: 298479

show more ...