Full path names are used in several unittests instead of the binary name. Fix up the testcase failures (#107974)

Encountered several testcase failures when running `ninja check-all`. It
was due to

Full path names are used in several unittests instead of the binary name. Fix up the testcase failures (#107974)

Encountered several testcase failures when running `ninja check-all`. It
was due to the full path name were shown in the error message instead of
the binary name, and therefore causing the check string mismatch.

The machine was running CentOS 9 with binfmt_misc setup that uses
qemu-aarch64 (8.1.2). Built and ran the unittest as aarch64 host
(through qemu user).

Co-authored-by: h2h <h2h@meta.com>

show more ...

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 rem

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 removes the "Script:" section that lit prints along with a
test's output, and it makes -v and -a imply -vv. For example, after
D154984, the "Script:" section below is never shown, but -v is enough
to produce the execution trace following it:

```
Script:
--
: 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
--
Exit Code: 2

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello"
# command output:
hello

$ "FileCheck" "bogus.txt"
# command stderr:
Could not open check file 'bogus.txt': No such file or directory

error: command failed with exit status: 2

--
```

In the D154984 review, some reviewers point out that they have been
using the "Script:" section for copying and pasting a test's shell
commands to a terminal window. The shell commands as printed in the
execution trace can be harder to copy and paste for the following
reasons:

- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to
copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and
are not clearly delineated.
- They don't always use proper shell quoting. Instead, they blindly
enclose all command-line arguments in double quotes.

Changes
-------

D154984 plus this patch converts the above example into:

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
echo hello | FileCheck bogus-file.txt && echo success
# executed command: echo hello
# .---command stdout------------
# | hello
# `-----------------------------
# executed command: FileCheck bogus-file.txt
# .---command stderr------------
# | Could not open check file 'bogus-file.txt': No such file or directory
# `-----------------------------
# error: command failed with exit status: 2

--
```

Thus, this patch addresses the above issues as follows:

- The entire execution trace can be copied and pasted in bulk to a
terminal for correct execution of the RUN lines, which are printed
intact as they appeared in the original RUN lines except lit
substitutions are expanded. Everything else in the execution trace
appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as
it executes to show (1) that the command actually executed (e.g.,
`echo success` above didn't) and (2) what stdout, stderr, non-zero
exit status, and output files are associated with the command, if
any. Shell quoting in the command is now correct and minimal but is
not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output
file is now delineated clearly in the trace.

To help produce some of the above output, this patch extends lit's
internal shell with a built-in `@echo` command. It's like `echo`
except lit suppresses the normal execution trace for `@echo` and just
prints its stdout directly. For now, `@echo` isn't documented for use
in lit tests.

Without this patch, libcxx's custom lit test format tries to parse the
stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's
internal shell) to extract the stdout and stderr produced by shell
commands, and that parse no longer works after the above changes.
This patch makes a small adjustment to
`lit.TestRunner.executeScriptInternal` so libcxx can just request
stdout and stderr without an execution trace.

(As a minor drive-by fix that came up in testing: lit's internal `not`
command now always produces a numeric exit status and never `True`.)

Caveat
------

This patch only makes the above changes for lit's internal shell. In
most cases, we do not know how to force external shells (e.g., bash,
sh, window's `cmd`) to produce execution traces in the manner we want.

To configure a test suite to use lit's internal shell (which is
usually better for test portability than external shells anyway), add
this to the test suite's `lit.cfg` or other configuration file:

```
config.test_format = lit.formats.ShTest(execute_external=False)
```

Reviewed By: MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D156954

show more ...

[lit] Drop "Script:", make -v and -a imply -vv

This patch and D156954 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

**Motivation**: -a shows output from al

[lit] Drop "Script:", make -v and -a imply -vv

This patch and D156954 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

**Motivation**: -a shows output from all tests, and -v shows output
from just failed tests. Without this patch, that output from each
test includes a section called "Script:", which includes all shell
commands that lit has computed from RUN directives and will attempt to
run for that test. The effect of -vv (which also implies -v if
neither -a or -v is specified) is to extend that output with shell
commands as they are executing so you can easily see which one failed.

For example, when using lit's internal shell and -vv:

```
Script:
--
: 'RUN: at line 1'; echo hello world
: 'RUN: at line 2'; 3c40 hello world
: 'RUN: at line 3'; echo hello world
--
Exit Code: 127

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello" "world"
hello world

$ ":" "RUN: at line 2"
$ "3c40" "hello" "world"
'3c40': command not found
error: command failed with exit status: 127

--
```

Notice that all shell commands that actually execute appear in the
output twice, once for "Script:" and once for -vv. Especially for
tests with many RUN directives, the result is noisy. When searching
through the output for a particular shell command, it is easy to get
lost and mistake shell commands under "Script:" for shell commands
that actually executed.

**Change**: With this patch, a test's output changes in two ways.
First, the "Script:" section is never shown. Second, omitting -vv no
longer disables printing of shell commands as they execute. That is,
-a and -v imply -vv, and so -vv is deprecated as it is just an alias
for -v.

**Secondary motivation**: We are also working to introduce a PYTHON
directive, which can appear between RUN directives. How should PYTHON
directives be represented in the "Script:" section, which has
previously been just a shell script? We could probably think of
something, but adding info about PYTHON directive execution in the -vv
trace seems more straight-forward and more useful.

(This patch also removes a confusing point in the -vv documentation:
at least when using bash as an external shell, -vv echoes commands to
the shell's stderr not stdout.)

Reviewed By: awarzynski, Endill, ldionne, MaskRay

Differential Revision: https://reviews.llvm.org/D154984

show more ...

Revert "[lit] Drop "Script:", make -v and -a imply -vv"

This reverts commit 09b6e457d91ce84088e6e21783913e5f1e5bd227.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-imp

Revert "[lit] Drop "Script:", make -v and -a imply -vv"

This reverts commit 09b6e457d91ce84088e6e21783913e5f1e5bd227.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Improve test output from lit's internal shell"

This reverts commit c981c533055e14302e7bff5d6898c9308065f665.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rf

Revert "[lit] Improve test output from lit's internal shell"

This reverts commit c981c533055e14302e7bff5d6898c9308065f665.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Try to fix c981c533055e test fails under windows"

This reverts commit f254bbf23374190c88a2b1a5f163622fbec9a936.

The reason for the revert is discussed at:
https://discourse.llvm.org/t

Revert "[lit] Try to fix c981c533055e test fails under windows"

This reverts commit f254bbf23374190c88a2b1a5f163622fbec9a936.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Fix c981c533055e's remaining test fails under windows"

This reverts commit 012d844fb856a89368aca95ca994726554b90f22.

The reason for the revert is discussed at:
https://discourse.llvm.

Revert "[lit] Fix c981c533055e's remaining test fails under windows"

This reverts commit 012d844fb856a89368aca95ca994726554b90f22.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

Revert "[lit] Fix yet another test fail under windows"

This reverts commit b6bd9d275f783f8150c8a04145ef2a31edb4fddf.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-impr

Revert "[lit] Fix yet another test fail under windows"

This reverts commit b6bd9d275f783f8150c8a04145ef2a31edb4fddf.

The reason for the revert is discussed at:
https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52

show more ...

[lit] Fix yet another test fail under windows

Another shell-quoting issue.

Seen in <https://lab.llvm.org/buildbot/#/builders/216/builds/26442>.

[lit] Fix c981c533055e's remaining test fails under windows

For `llvm/utils/lit/tests/shtest-output-printing.py`, the executable
in `%{python}` wasn't properly shell-quoted for windows. This caused

[lit] Fix c981c533055e's remaining test fails under windows

For `llvm/utils/lit/tests/shtest-output-printing.py`, the executable
in `%{python}` wasn't properly shell-quoted for windows. This caused
the 127 exit code mentioned in f254bbf23374. Fix quoting and expect
exit code 1 again.

Fix shell-quoting issue in a few more file names in
`llvm/utils/lit/tests/shtest-shell.py`, missed in f254bbf23374.

Test failures seen in
<https://lab.llvm.org/buildbot/#/builders/216/builds/26436>.

show more ...

[lit] Try to fix c981c533055e test fails under windows

Failures were seen at
<https://lab.llvm.org/buildbot/#/builders/216/builds/26431>.

All but one failure is due to different shell-quoting of fi

[lit] Try to fix c981c533055e test fails under windows

Failures were seen at
<https://lab.llvm.org/buildbot/#/builders/216/builds/26431>.

All but one failure is due to different shell-quoting of file names
because they contain special characters under windows. Generalize
associated FileCheck patterns.

`llvm/utils/lit/tests/shtest-output-printing.py` fails because the
exit code was 127 instead of the expected 1. Unfortunately, this CI
config doesn't pass `-dump-input-filter=all` to FileCheck, so we
cannot see the rest of the lit execution trace. For now, generalize
the FileCheck pattern to accept any non-zero exit code to get past
this error.

show more ...

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 rem

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 removes the "Script:" section that lit prints along with a
test's output, and it makes -v and -a imply -vv. For example, after
D154984, the "Script:" section below is never shown, but -v is enough
to produce the execution trace following it:

```
Script:
--
: 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
--
Exit Code: 2

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello"
# command output:
hello

$ "FileCheck" "bogus.txt"
# command stderr:
Could not open check file 'bogus.txt': No such file or directory

error: command failed with exit status: 2

--
```

In the D154984 review, some reviewers point out that they have been
using the "Script:" section for copying and pasting a test's shell
commands to a terminal window. The shell commands as printed in the
execution trace can be harder to copy and paste for the following
reasons:

- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to
copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and
are not clearly delineated.
- They don't always use proper shell quoting. Instead, they blindly
enclose all command-line arguments in double quotes.

Changes
-------

D154984 plus this patch converts the above example into:

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
echo hello | FileCheck bogus-file.txt && echo success
# executed command: echo hello
# .---command stdout------------
# | hello
# `-----------------------------
# executed command: FileCheck bogus-file.txt
# .---command stderr------------
# | Could not open check file 'bogus-file.txt': No such file or directory
# `-----------------------------
# error: command failed with exit status: 2

--
```

Thus, this patch addresses the above issues as follows:

- The entire execution trace can be copied and pasted in bulk to a
terminal for correct execution of the RUN lines, which are printed
intact as they appeared in the original RUN lines except lit
substitutions are expanded. Everything else in the execution trace
appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as
it executes to show (1) that the command actually executed (e.g.,
`echo success` above didn't) and (2) what stdout, stderr, non-zero
exit status, and output files are associated with the command, if
any. Shell quoting in the command is now correct and minimal but is
not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output
file is now delineated clearly in the trace.

To help produce some of the above output, this patch extends lit's
internal shell with a built-in `@echo` command. It's like `echo`
except lit suppresses the normal execution trace for `@echo` and just
prints its stdout directly. For now, `@echo` isn't documented for use
in lit tests.

Without this patch, libcxx's custom lit test format tries to parse the
stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's
internal shell) to extract the stdout and stderr produced by shell
commands, and that parse no longer works after the above changes.
This patch makes a small adjustment to
`lit.TestRunner.executeScriptInternal` so libcxx can just request
stdout and stderr without an execution trace.

(As a minor drive-by fix that came up in testing: lit's internal `not`
command now always produces a numeric exit status and never `True`.)

Caveat
------

This patch only makes the above changes for lit's internal shell. In
most cases, we do not know how to force external shells (e.g., bash,
sh, window's `cmd`) to produce execution traces in the manner we want.

To configure a test suite to use lit's internal shell (which is
usually better for test portability than external shells anyway), add
this to the test suite's `lit.cfg` or other configuration file:

```
config.test_format = lit.formats.ShTest(execute_external=False)
```

Reviewed By: MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D156954

show more ...

[lit] Drop "Script:", make -v and -a imply -vv

This patch and D156954 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

**Motivation**: -a shows output from al

[lit] Drop "Script:", make -v and -a imply -vv

This patch and D156954 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

**Motivation**: -a shows output from all tests, and -v shows output
from just failed tests. Without this patch, that output from each
test includes a section called "Script:", which includes all shell
commands that lit has computed from RUN directives and will attempt to
run for that test. The effect of -vv (which also implies -v if
neither -a or -v is specified) is to extend that output with shell
commands as they are executing so you can easily see which one failed.

For example, when using lit's internal shell and -vv:

```
Script:
--
: 'RUN: at line 1'; echo hello world
: 'RUN: at line 2'; 3c40 hello world
: 'RUN: at line 3'; echo hello world
--
Exit Code: 127

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello" "world"
hello world

$ ":" "RUN: at line 2"
$ "3c40" "hello" "world"
'3c40': command not found
error: command failed with exit status: 127

--
```

Notice that all shell commands that actually execute appear in the
output twice, once for "Script:" and once for -vv. Especially for
tests with many RUN directives, the result is noisy. When searching
through the output for a particular shell command, it is easy to get
lost and mistake shell commands under "Script:" for shell commands
that actually executed.

**Change**: With this patch, a test's output changes in two ways.
First, the "Script:" section is never shown. Second, omitting -vv no
longer disables printing of shell commands as they execute. That is,
-a and -v imply -vv, and so -vv is deprecated as it is just an alias
for -v.

**Secondary motivation**: We are also working to introduce a PYTHON
directive, which can appear between RUN directives. How should PYTHON
directives be represented in the "Script:" section, which has
previously been just a shell script? We could probably think of
something, but adding info about PYTHON directive execution in the -vv
trace seems more straight-forward and more useful.

(This patch also removes a confusing point in the -vv documentation:
at least when using bash as an external shell, -vv echoes commands to
the shell's stderr not stdout.)

Reviewed By: awarzynski, Endill, ldionne, MaskRay

Differential Revision: https://reviews.llvm.org/D154984

show more ...

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default t

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default to using lexical order and one job in the lit test
suite.

Differential Revision: https://reviews.llvm.org/D107695

show more ...

[lit][tests][AIX] Update expected form of diagnostic messages; use `not` to normalize non-zero exit values

Summary:
Various tests in the `lit` testing suite expect specific return codes
and forms of

[lit][tests][AIX] Update expected form of diagnostic messages; use `not` to normalize non-zero exit values

Summary:
Various tests in the `lit` testing suite expect specific return codes
and forms of diagnostic message from utility programs. As per
POSIX.1-2017 XCU Section 1.4, Utility Description Defaults, "[the]
format of diagnostic messages for most utilities is unspecified".
The STDERR subsections of the `cat` and `wc` utilities merely indicate
that "[the] standard error shall be used only for diagnostic messages".
The corresponding EXIT STATUS subsections merely indicate, with regard
to errors, an exit value of >0.

The affected tests are updated to accept the applicable diagnostic
message as produced by the utilities on AIX. The exit value is
normalized using `not` as necessary.

Reviewers: xingxue, sfertile, jasonliu

Reviewed By: xingxue

Subscribers: delcypher, jsji, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D60553

llvm-svn: 359690

show more ...

[lit] Report line number for failed RUN command

(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locat

[lit] Report line number for failed RUN command

(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333614

show more ...

Revert r333584: [lit] Report line number for failed RUN command

It breaks test-suite.

llvm-svn: 333592

[lit] Report line number for failed RUN command

(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this m

[lit] Report line number for failed RUN command

(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333584

show more ...

Revert r330755 "[lit] Report line number for failed RUN command"

It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211

llvm-svn:

Revert r330755 "[lit] Report line number for failed RUN command"

It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211

llvm-svn: 330848

show more ...

[lit] Report line number for failed RUN command

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example

[lit] Report line number for failed RUN command

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

Reviewed By: asmith, delcypher

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 330755

show more ...

[lit] Fix shtest-output-printing.py on Windows by matching either / or \\

Fixes PR33938

llvm-svn: 309198

[lit] Mark several of lit's tests XFAIL on Windows

Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Window

[lit] Mark several of lit's tests XFAIL on Windows

Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.

In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.

Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:

```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```

Verify that the test suite is run and a successful exit code is
returned.

Reviewers: mgorny, rnk, delcypher, beanz

Reviewed By: rnk

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35879

llvm-svn: 309123

show more ...

[lit] Improve logging with file redirection.

- This will cause lit to automatically include the first 1K of data in
redirected output files when a command fails (previously if the command
fai

[lit] Improve logging with file redirection.

- This will cause lit to automatically include the first 1K of data in
redirected output files when a command fails (previously if the command
failed, but the main point of the test was, say, a `FileCheck` later on, then
the log wasn't helpful in showing why the command failed).

llvm-svn: 272021

show more ...

[lit] Improve readability of failing scripts.

- This only applies to scripts executed by the _internal_ shell script
interpreter.

- This patch reworks the log to look more like a shell transcr

[lit] Improve readability of failing scripts.

- This only applies to scripts executed by the _internal_ shell script
interpreter.

- This patch reworks the log to look more like a shell transcript, and be less
verbose (but in the interest of calling attention to the important parts).

Here is an example of the new format, for commands with/without failures and
with/without output:
```
$ true
$ echo hi
hi

$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1

```

llvm-svn: 271610

show more ...