[llvm-lit][test][NFC] Moved cat command tests into separate lit test file (#102366)

This patch separates the lit tests that check for the functionality of
lit's built-in cat command into its own te

[llvm-lit][test][NFC] Moved cat command tests into separate lit test file (#102366)

This patch separates the lit tests that check for the functionality of
lit's built-in cat command into its own test file and folder. This is a
prerequisite for https://github.com/llvm/llvm-project/pull/101530.

show more ...

[lit] Clean up internal shell parse errors with ScriptFatal (#68496)

Without this patch, the functions `executeScriptInternal` and thus
`runOnce` in `llvm/utils/lit/lit/TestRunner.py` return either

[lit] Clean up internal shell parse errors with ScriptFatal (#68496)

Without this patch, the functions `executeScriptInternal` and thus
`runOnce` in `llvm/utils/lit/lit/TestRunner.py` return either a tuple like
`(out, err, exitCode, timeoutInfo)` or a `lit.Test.Result` object. They
return the latter only when there's a lit internal shell parse error in
a RUN line. In my opinion, a more straight-forward way to handle
exceptional cases like that is to use python exceptions.

For that purpose, this patch introduces `ScriptFatal`. Thus, this patch
changes `executeScriptInternal` to always either return the tuple or
raise the `ScriptFatal` exception. It updates `runOnce` and
`libcxx/utils/libcxx/test/format.py` to catch the exception rather than
check for the special return type.

This patch also changes `runOnce` to convert the exception to a
`Test.UNRESOLVED` result instead of `TEST.FAIL`. The former is the
proper result for such a malformed test, for which a rerun (given an
`ALLOW_RETRIES:`) serves no purpose. There are at least two benefits
from this change. First, `_runShTest` no longer has to specially and
cryptically handle this case to avoid unnecessary reruns. Second, an
`XFAIL:` directive no longer hides such a failure [as we saw
previously](https://reviews.llvm.org/D154987#4501125).

To facilitate the `_runShTest` change, this patch inserts the internal
shell parse error diagnostic into the format of the test's normal debug
output rather than suppressing the latter entirely. That change is also
important for [D154987](https://reviews.llvm.org/D154987), which
proposes to reuse `ScriptFatal` for python compile errors in PYTHON
lines or in `config.prologue`. In that case, the diagnostic might follow
debugging output from the test's previous RUN or PYTHON lines, so
suppressing the normal debug output would lose information.

show more ...

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 rem

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 removes the "Script:" section that lit prints along with a
test's output, and it makes -v and -a imply -vv. For example, after
D154984, the "Script:" section below is never shown, but -v is enough
to produce the execution trace following it:

```
Script:
--
: 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
--
Exit Code: 2

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello"
# command output:
hello

$ "FileCheck" "bogus.txt"
# command stderr:
Could not open check file 'bogus.txt': No such file or directory

error: command failed with exit status: 2

--
```

In the D154984 review, some reviewers point out that they have been
using the "Script:" section for copying and pasting a test's shell
commands to a terminal window. The shell commands as printed in the
execution trace can be harder to copy and paste for the following
reasons:

- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to
copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and
are not clearly delineated.
- They don't always use proper shell quoting. Instead, they blindly
enclose all command-line arguments in double quotes.

Changes
-------

D154984 plus this patch converts the above example into:

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
echo hello | FileCheck bogus-file.txt && echo success
# executed command: echo hello
# .---command stdout------------
# | hello
# `-----------------------------
# executed command: FileCheck bogus-file.txt
# .---command stderr------------
# | Could not open check file 'bogus-file.txt': No such file or directory
# `-----------------------------
# error: command failed with exit status: 2

--
```

Thus, this patch addresses the above issues as follows:

- The entire execution trace can be copied and pasted in bulk to a
terminal for correct execution of the RUN lines, which are printed
intact as they appeared in the original RUN lines except lit
substitutions are expanded. Everything else in the execution trace
appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as
it executes to show (1) that the command actually executed (e.g.,
`echo success` above didn't) and (2) what stdout, stderr, non-zero
exit status, and output files are associated with the command, if
any. Shell quoting in the command is now correct and minimal but is
not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output
file is now delineated clearly in the trace.

To help produce some of the above output, this patch extends lit's
internal shell with a built-in `@echo` command. It's like `echo`
except lit suppresses the normal execution trace for `@echo` and just
prints its stdout directly. For now, `@echo` isn't documented for use
in lit tests.

Without this patch, libcxx's custom lit test format tries to parse the
stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's
internal shell) to extract the stdout and stderr produced by shell
commands, and that parse no longer works after the above changes.
This patch makes a small adjustment to
`lit.TestRunner.executeScriptInternal` so libcxx can just request
stdout and stderr without an execution trace.

(As a minor drive-by fix that came up in testing: lit's internal `not`
command now always produces a numeric exit status and never `True`.)

Caveat
------

This patch only makes the above changes for lit's internal shell. In
most cases, we do not know how to force external shells (e.g., bash,
sh, window's `cmd`) to produce execution traces in the manner we want.

To configure a test suite to use lit's internal shell (which is
usually better for test portability than external shells anyway), add
this to the test suite's `lit.cfg` or other configuration file:

```
config.test_format = lit.formats.ShTest(execute_external=False)
```

Reviewed By: MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D156954

show more ...

Revert "[lit] Improve test output from lit's internal shell"

This reverts commit c981c533055e14302e7bff5d6898c9308065f665.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rf

Revert "[lit] Improve test output from lit's internal shell"

This reverts commit c981c533055e14302e7bff5d6898c9308065f665.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Try to fix c981c533055e test fails under windows"

This reverts commit f254bbf23374190c88a2b1a5f163622fbec9a936.

The reason for the revert is discussed at:
https://discourse.llvm.org/t

Revert "[lit] Try to fix c981c533055e test fails under windows"

This reverts commit f254bbf23374190c88a2b1a5f163622fbec9a936.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Fix f254bbf23374 FileCheck patterns"

This reverts commit 3db5db92d746bad8ba1762ca290a176e25d48565.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improvin

Revert "[lit] Fix f254bbf23374 FileCheck patterns"

This reverts commit 3db5db92d746bad8ba1762ca290a176e25d48565.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Fix c981c533055e's remaining test fails under windows"

This reverts commit 012d844fb856a89368aca95ca994726554b90f22.

The reason for the revert is discussed at:
https://discourse.llvm.

Revert "[lit] Fix c981c533055e's remaining test fails under windows"

This reverts commit 012d844fb856a89368aca95ca994726554b90f22.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

[lit] Fix c981c533055e's remaining test fails under windows

For `llvm/utils/lit/tests/shtest-output-printing.py`, the executable
in `%{python}` wasn't properly shell-quoted for windows. This caused

[lit] Fix c981c533055e's remaining test fails under windows

For `llvm/utils/lit/tests/shtest-output-printing.py`, the executable
in `%{python}` wasn't properly shell-quoted for windows. This caused
the 127 exit code mentioned in f254bbf23374. Fix quoting and expect
exit code 1 again.

Fix shell-quoting issue in a few more file names in
`llvm/utils/lit/tests/shtest-shell.py`, missed in f254bbf23374.

Test failures seen in
<https://lab.llvm.org/buildbot/#/builders/216/builds/26436>.

show more ...

[lit] Fix f254bbf23374 FileCheck patterns

Handle the case when shell quotes are not necessary.

[lit] Try to fix c981c533055e test fails under windows

Failures were seen at
<https://lab.llvm.org/buildbot/#/builders/216/builds/26431>.

All but one failure is due to different shell-quoting of fi

[lit] Try to fix c981c533055e test fails under windows

Failures were seen at
<https://lab.llvm.org/buildbot/#/builders/216/builds/26431>.

All but one failure is due to different shell-quoting of file names
because they contain special characters under windows. Generalize
associated FileCheck patterns.

`llvm/utils/lit/tests/shtest-output-printing.py` fails because the
exit code was 127 instead of the expected 1. Unfortunately, this CI
config doesn't pass `-dump-input-filter=all` to FileCheck, so we
cannot see the rest of the lit execution trace. For now, generalize
the FileCheck pattern to accept any non-zero exit code to get past
this error.

show more ...

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 rem

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 removes the "Script:" section that lit prints along with a
test's output, and it makes -v and -a imply -vv. For example, after
D154984, the "Script:" section below is never shown, but -v is enough
to produce the execution trace following it:

```
Script:
--
: 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
--
Exit Code: 2

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello"
# command output:
hello

$ "FileCheck" "bogus.txt"
# command stderr:
Could not open check file 'bogus.txt': No such file or directory

error: command failed with exit status: 2

--
```

In the D154984 review, some reviewers point out that they have been
using the "Script:" section for copying and pasting a test's shell
commands to a terminal window. The shell commands as printed in the
execution trace can be harder to copy and paste for the following
reasons:

- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to
copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and
are not clearly delineated.
- They don't always use proper shell quoting. Instead, they blindly
enclose all command-line arguments in double quotes.

Changes
-------

D154984 plus this patch converts the above example into:

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
echo hello | FileCheck bogus-file.txt && echo success
# executed command: echo hello
# .---command stdout------------
# | hello
# `-----------------------------
# executed command: FileCheck bogus-file.txt
# .---command stderr------------
# | Could not open check file 'bogus-file.txt': No such file or directory
# `-----------------------------
# error: command failed with exit status: 2

--
```

Thus, this patch addresses the above issues as follows:

- The entire execution trace can be copied and pasted in bulk to a
terminal for correct execution of the RUN lines, which are printed
intact as they appeared in the original RUN lines except lit
substitutions are expanded. Everything else in the execution trace
appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as
it executes to show (1) that the command actually executed (e.g.,
`echo success` above didn't) and (2) what stdout, stderr, non-zero
exit status, and output files are associated with the command, if
any. Shell quoting in the command is now correct and minimal but is
not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output
file is now delineated clearly in the trace.

To help produce some of the above output, this patch extends lit's
internal shell with a built-in `@echo` command. It's like `echo`
except lit suppresses the normal execution trace for `@echo` and just
prints its stdout directly. For now, `@echo` isn't documented for use
in lit tests.

Without this patch, libcxx's custom lit test format tries to parse the
stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's
internal shell) to extract the stdout and stderr produced by shell
commands, and that parse no longer works after the above changes.
This patch makes a small adjustment to
`lit.TestRunner.executeScriptInternal` so libcxx can just request
stdout and stderr without an execution trace.

(As a minor drive-by fix that came up in testing: lit's internal `not`
command now always produces a numeric exit status and never `True`.)

Caveat
------

This patch only makes the above changes for lit's internal shell. In
most cases, we do not know how to force external shells (e.g., bash,
sh, window's `cmd`) to produce execution traces in the manner we want.

To configure a test suite to use lit's internal shell (which is
usually better for test portability than external shells anyway), add
this to the test suite's `lit.cfg` or other configuration file:

```
config.test_format = lit.formats.ShTest(execute_external=False)
```

Reviewed By: MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D156954

show more ...

[lit] Implement DEFINE and REDEFINE directives

These directives define per-test lit substitutions. The concept was
discussed at
<https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>.

For

[lit] Implement DEFINE and REDEFINE directives

These directives define per-test lit substitutions. The concept was
discussed at
<https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>.

For example, the following directives can be inserted into a test file
to define `%{cflags}` and `%{fcflags}` substitutions with empty
initial values, which serve as the parameters of another newly defined
`%{check}` substitution:

```
// DEFINE: %{cflags} =
// DEFINE: %{fcflags} =

// DEFINE: %{check} = %clang_cc1 %{cflags} -emit-llvm -o - %s | \
// DEFINE: FileCheck %{fcflags} %s
```

The following directives then redefine the parameters before each use
of `%{check}`:

```
// REDEFINE: %{cflags} = -foo
// REDEFINE: %{fcflags} = -check-prefix=FOO
// RUN: %{check}

// REDEFINE: %{cflags} = -bar
// REDEFINE: %{fcflags} = -check-prefix=BAR
// RUN: %{check}
```

Of course, `%{check}` would typically be more elaborate, increasing
the benefit of the reuse.

One issue is that the strings `DEFINE:` and `REDEFINE:` already appear
in 5 tests. This patch adjusts those tests not to use those strings.
Our prediction is that, in the vast majority of cases, if a test
author mistakenly uses one of those strings for another purpose, the
text appearing after the string will not happen to have the syntax
required for these directives. Thus, the test author will discover
the mistake immediately when lit reports the syntax error.

This patch also expands the documentation on existing lit substitution
behavior.

Reviewed By: jhenderson, MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D132513

show more ...

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default t

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default to using lexical order and one job in the lit test
suite.

Differential Revision: https://reviews.llvm.org/D107695

show more ...

[NFCI][lit] Unbreak more lit self-tests after D98179

All of these depend on the order of tests, so if one runs them twice,
the tests within them will naturally be reordered
using the previous run ti

[NFCI][lit] Unbreak more lit self-tests after D98179

All of these depend on the order of tests, so if one runs them twice,
the tests within them will naturally be reordered
using the previous run times, which breaks them.

show more ...

[lit] Sort test start times based on prior test timing data

Lit as it exists today has three hacks that allow users to run tests earlier:

1) An entire test suite can set the `is_early` boolean.
2)

[lit] Sort test start times based on prior test timing data

Lit as it exists today has three hacks that allow users to run tests earlier:

1) An entire test suite can set the `is_early` boolean.
2) A very recently introduced "early_tests" feature.
3) The `--incremental` flag forces failing tests to run first.

All of these approaches have problems.

1) The `is_early` feature was until very recently undocumented. Nevertheless it still lacks testing and is a imprecise way of optimizing test starting times.
2) The `early_tests` feature requires manual updates and doesn't scale.
3) `--incremental` is undocumented, untested, and it requires modifying the *source* file system by "touching" the file. This "touch" based approach is arguably a hack because it confuses editors (because it looks like the test was modified behind the back of the editor) and "touching" the test source file doesn't work if the test suite is read only from the perspective of `lit` (via advanced filesystem/build tricks).

This patch attempts to simplify and address all of the above problems.

This patch formalizes, documents, tests, and defaults lit to recording the execution time of tests and then reordering all tests during the next execution. By reordering the tests, high core count machines run faster, sometimes significantly so.

This patch also always runs failing tests first, which is a positive user experience win for those that didn't know about the hidden `--incremental` flag.

Finally, if users want, they can _optionally_ commit the test timing data (or a subset thereof) back to the repository to accelerate bots and first-time runs of the test suite.

Reviewed By: jhenderson, yln

Differential Revision: https://reviews.llvm.org/D98179

show more ...

[lit] Prevent hang when lit sees non-ASCII characters

As per discussion in D69207, have lit ignore UnicodeDecodeErrors
when running with python 2 in an ASCII shell.

Differential Revision: https://r

[lit] Prevent hang when lit sees non-ASCII characters

As per discussion in D69207, have lit ignore UnicodeDecodeErrors
when running with python 2 in an ASCII shell.

Differential Revision: https://reviews.llvm.org/D82754

show more ...

[lit] Improve naming of test result categories

Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing te

[lit] Improve naming of test result categories

Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines. For
example, "Failing Tests"/"Unexpected Failures". This commit changes lit
to label things consistently.

Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous. Some labels contain the word "Tests" and some don't.
Let's simplify the names.

Before:
```
Failing Tests (1):
...

Expected Passes : 3
Unexpected Failures: 1
```

After:
```
Failed Tests (1):
...

Passed: 3
Failed: 1
```

Reviewed By: ldionne

Differential Revision: https://reviews.llvm.org/D77708

show more ...

[lit] Fix internal diff newlines for -w/-b

For example, without this patch:

```
$ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt
*** /tmp/foo.txt
--- /tmp/bar.txt
***************
*** 1,2 ****
1!

[lit] Fix internal diff newlines for -w/-b

For example, without this patch:

```
$ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt
*** /tmp/foo.txt
--- /tmp/bar.txt
***************
*** 1,2 ****
1! 2--- 1,2 ----
1! 20
```

With this patch:

```
$ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt
*** /tmp/foo.txt
--- /tmp/bar.txt
***************
*** 1,2 ****
1
! 2
--- 1,2 ----
1
! 20
```

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D71577

show more ...

[lit] Extend internal diff to support `-` argument

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN

[lit] Extend internal diff to support `-` argument

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN: program | diff file -
```

Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-` as a command-line option. This patch adds support for
`-` to mean stdin.

Reviewed By: probinson, rnk

Differential Revision: https://reviews.llvm.org/D67643

show more ...

[lit] Make internal diff work in pipelines

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN: progra

[lit] Make internal diff work in pipelines

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```

Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.

To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.

Also, when lit's `diff` prints differences to stdout in Windows, this
patch ensures it always terminate lines with `\n` not `\r\n`. That
way, strict FileCheck directives checking the `diff` output succeed in
both Linux and Windows. This wasn't an issue when `diff` was internal
to lit because `diff` didn't then write to the true stdout, which is
where the `\n` -> `\r\n` conversion happened in Python.

Reviewed By: probinson, stella.stamenova

Differential Revision: https://reviews.llvm.org/D66574

show more ...

[lit] Don't fail when printing test output with special chars

This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows
10.

Reviewed By: rnk

Differential Revision: https://reviews.llv

[lit] Don't fail when printing test output with special chars

This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows
10.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D69207

show more ...

Revert r375114: "[lit] Make internal diff work in pipelines"

This series of patches still breaks a Windows bot.

llvm-svn: 375121

Revert r375116: "[lit] Extend internal diff to support `-` argument"

This series of patches still breaks a Windows bot.

llvm-svn: 375120

[lit] Extend internal diff to support `-` argument

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN

[lit] Extend internal diff to support `-` argument

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN: program | diff file -
```

Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-` as a command-line option. This patch adds support for
`-` to mean stdin.

Reviewed By: probinson, rnk

Differential Revision: https://reviews.llvm.org/D67643

llvm-svn: 375116

show more ...

[lit] Make internal diff work in pipelines

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN: progra

[lit] Make internal diff work in pipelines

When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:

```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```

Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.

To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.

Reviewed By: probinson, stella.stamenova

Differential Revision: https://reviews.llvm.org/D66574

llvm-svn: 375114

show more ...