This commit is intended to be what was intended to happen in the
commit of Sun Nov 10 01:22:24 UTC 2024, see:

http://mail-index.netbsd.org/source-changes/2024/11/10/msg154310.html

The commit me

This commit is intended to be what was intended to happen in the
commit of Sun Nov 10 01:22:24 UTC 2024, see:

http://mail-index.netbsd.org/source-changes/2024/11/10/msg154310.html

The commit message for that applies to this one (wholly). I believe that
the problem with that version which caused it to be reverted has been found
and fixed in this version (a necessary change was made as part of one of
the fixes, but the side-effect implications of that were missed -- bad bad me.)

In addition, I found some more issues with setting close-on-exec on other
command lines

With:
func 3>whatever

fd 3 (anything > 2) got close on exec set. That makes no difference
to the function itself (nothing gets exec'd therefore nothing gets closed)
but does to any exec that might happen running a command within the function.

I believe that if this is done (just as if "func" was a regular command,
and not a function) such open fds should be passed through to anything
they exec - unless the function (or other command) takes care to close the
fd passed to it, or explicitly turn on close-on exec.

I expect this usage to be quite rare, and not make much practical difference.

The same applies do builtin commands, but is even less relevant there, eg:

printf 3>whatever

would have set close-on-exec on fd 3 for printf. This is generally
completely immaterial, as printf - and most other built-in commands -
neither uses any fd other than (some of) 0 1 & 2, nor do they exec anything.

That is, except for the "exec" built-in which was the focus of the original
fix (mentioned above) and which should remain fixed here, and for the "."
command.

Because of that last one (".") close-on-exec should not be set on built-in
commands (any of them) for redirections on the command line. This will
almost never make a difference - any such redirections last only as long
as the built-in command lasts (same with functions) and so will generally
never care about the state of close-on-exec, and I have never seen a use
of the "." command with any redirections other than stderr (which is unaffected
here, fd's <= 2 never get close-on-exec set). That's probably why no-one
ever noticed.

There are still "fd issues" when running a (non #!) shell script, that
are hard to fix, which we should probably handle the way most other shells
have, by simply abandoning the optimisation of not exec'ing a whole new
shell (#! scripts do that exec) and just doing it that way. Issues solved!
One day.

show more ...

Revert the recent change until I can work out how things are broken.

exec builtin command redirection fixes

Several changes, all related to the exec special built in command,
or to close on exec, one way or another. (Except a few white space
and comment additions,

exec builtin command redirection fixes

Several changes, all related to the exec special built in command,
or to close on exec, one way or another. (Except a few white space
and comment additions, KNF, etc)

1. The bug found by Edgar Fuß reported in:
http://mail-index.netbsd.org/tech-userlevel/2024/11/05/msg014588.html
has been fixed, now "exec N>whatever" will set close-on-exec for fd N
(as do ksh versions, and allowed by POSIX, though other shells do not)
which has happened now for many years. But "exec cmd N>whatever"
(which looks like the same command when redirections are processed)
which was setting close-on-exec on N, now no longer does, so fd N
can be passed to cmd as an open fd.

For anyone who cares, the big block of change just after "case CMDBUILTIN:"
in evalcommand() in eval.c is the fix for this (one line replaced by
about 90 ... though most of that is comments or #if 0'd example code
for later). It is a bit ugly, and will get worse if our exec command
ever gets any options, as others have, but it does work.

2. when the exec builtin utility is used to alter the shell's redirections
it is now "all or nothing". Previously the redirections were executed
left to right. If one failed, no more were attempted, but the earlier
ones remained. This makes no practical difference to a non-interactive
shell, as a redirection error causes that shell to exit, but it makes
a difference to interactive shells. Now if a redirection fails, any
earlier ones which had been performed are undone. Note however that
side-effects of redirections (like creating, or truncating, files in
the filesystem, cannot be reversed - just the shell's file descriptors
returned to how they were before the error).

Similarly usage errors on exec now exist .. our exec takes no options
(but does handle "--" as POSIX says it must - has done for ages).
Until now, that was the only magic piece of exec, running
exec -a name somecommand
(which several other shells support) would attempt to exec the "-a"
command, and most likely fail, causing immediate exit from the shell.
Now that is a usage error - a non-interactive shell still exits, as
exec is a special builtin, and any error from a special builtin causes
a non-interactive shell to exit. But now, an interactive shell will
no longer exit (and any redirections that were on the command will be
undone, the same as for a redirection error).

3. When a "close on exec" file descriptor is temporarily saved, so the
same fd can be redirected for another command (only built-in commands
and functions matter, redirects for file system commands happen after
a fork() and at that stage if anything goes wrong, the child simply
exits - but for non-forking commands, doing something like printf >file
required the previous stdout to be saved elsewhere, "file" opened to
be the new stdout, then when printf is finished, the old stdout moved
back. Anyway, if the fd being moved had close on exec set, then
when it was moved back, the close on exec was lost. That is now fixed.

4. The fdflags command no longer allows setting close on exec on stdin,
stdout, or stderr - POSIX requires that those 3 fd's always be open
(to something) when any normal command is invoked. With close-on-exec
set on one of these, that is impossible, so simply refuse it (when
"exec N>file" sets close on exec, it only does it for N>2).

Minor changes (should be invisible)

a. The shell now keeps track of the highest fd number it sees doing
normal operations (there are a few internal pipe() calls that aren't
monitored and a couple of others, but in general the shell will now
know the highest fd it ever saw allocated to it). This is mostly
for debugging.

b. calls to fcntl() passing an int as the "arg" are now all properly
cast to the void * that the fcntl kernel is expecting to receive.
I suspect that makes no actual difference to anything, but ...

show more ...

Add function names to relevant error messages.

When a shell detected error occurs while executing a function,
include the name of the function currently being executed in
the error message.

Change the "string" argument to evalstring() and setinputstring()
from being "char *" to being "const char *".

This is needed for a forthcoming change which needs to pass a const char *
to evalstrin

Change the "string" argument to evalstring() and setinputstring()
from being "char *" to being "const char *".

This is needed for a forthcoming change which needs to pass a const char *
to evalstring (and through it to setinputstring) and be assured that
nothing will alter the characters in the string supplied.

This is (aside from the additional compile time protection provided)
a no-op change, all evalstring() does with its string is pass it to
setinputstring() and all that does with it is determine its length
(strlen() which expects a const char *) and assign the string pointer
to parsenextc which is already a const char * - there never has been
any reason for these two functions to not include the "const" in
the arg declaration -- except that when originally written (early
1990's) I suspect "const" either didn't exist at all, or wasn't
supported by relevant compilers.

NFCI. Most probably (though I didn't check) no binary change at all.

show more ...

POSIX.1-2024 requires that when an async (background) job is started
at the top level (ie: not in any kind of subshell environment) of an
interactive shell, that the shell print the job number assign

POSIX.1-2024 requires that when an async (background) job is started
at the top level (ie: not in any kind of subshell environment) of an
interactive shell, that the shell print the job number assigned, and
the process id of the lead (or only) process in the job, in the form:

[JN] pid

Make that happen. (Other shells have been doing this for ages).

show more ...

PR bin/57773

Fix a bug reported by Jarle Fredrik Greipsland in PR bin/57773,
where a substring expansion where the substring to be removed from
a variable expansion is itself a var expansion where t

PR bin/57773

Fix a bug reported by Jarle Fredrik Greipsland in PR bin/57773,
where a substring expansion where the substring to be removed from
a variable expansion is itself a var expansion where the value
contains one (or more) of sh's CTLxxx chars - the pattern had
CTLESC inserted, the string to be matched against did not. Fail.
We fix that by always inserting CTLESC in var assign expansions.
See the PR for all the gory details.

Thanks for the PR.

XXX pullup to everything.

show more ...

Fix typo in a debug message.

The great shell trailing whitespace cleanup of 2023...
Inspired by private e-mail comments from mouse@

NFCI.

Use a volative local shadow of a field in an (on-stack) non-volatile struct
that is to be referenced after a return from setjmp() via longjmp().

This doesn't ever seem to have caused a problem, but

Use a volative local shadow of a field in an (on-stack) non-volatile struct
that is to be referenced after a return from setjmp() via longjmp().

This doesn't ever seem to have caused a problem, but I think using
volative vars is required here.

For reasons I never bothered to discover, even though this change
certainly requires a store into stack memory which wasn't required
before, earlier measurements showed the shell getting (slightly) smaller
with this change in place.

NFCI

show more ...

s/commmand/command/ in comment.

PR bin/53550

Here we go again... One more time to redo how here docs are
processed (it has been a few years since the last time!)

This is actually a relatively minor change, mostly to timimg
(to

PR bin/53550

Here we go again... One more time to redo how here docs are
processed (it has been a few years since the last time!)

This is actually a relatively minor change, mostly to timimg
(to just when things happen). Now here docs are expanded at the
same time the "filename" word in a redirect is expanded, rather than
later when the heredoc was being sent to its process. This actually
makes things more consistent - but does break one of the ATF tests
which was testing that we were (effectively) internally inconsistent
in this area.

Not all shells agree on the context in which redirection expansions
should happen, some make any side effects visible to the parent shell
(the majority do) others do the redirection expansions in a subshell
so any side effcts are lost. We used to have a foot in each camp,
with the majority for everything but here docs, and the minority for
here docs. Now we're all the way with LBJ ... (or something like that).

show more ...

Detect write errors to stdout, and exit(1) from some built-in
commands which (primarily) are used just to generate output
(or with a particular option combination do so).

Fix value of ${LINENO} in "for" commands.

This affects (as best I can tell) only uses of ${LINENO} in PS4
when -x is enabled (and perhaps only when the list contains no
expansions). "for" like "ca

Fix value of ${LINENO} in "for" commands.

This affects (as best I can tell) only uses of ${LINENO} in PS4
when -x is enabled (and perhaps only when the list contains no
expansions). "for" like "case" (which was already handled) is
special in that it generates trace output before actually executing
any kind of simple command.

show more ...

DEBUG mode changes only. NFC (NC) for any normally compiled shell.

Mostly adding DEBUG mode tracing (when appropriate verbose tracing
is enabled generally) whenever a shell (including sushell) pro

DEBUG mode changes only. NFC (NC) for any normally compiled shell.

Mostly adding DEBUG mode tracing (when appropriate verbose tracing
is enabled generally) whenever a shell (including sushell) process
exits, so shells that the tracing should indicate why ehslls that
vanish did that.

Note for future investigators: if the relevant tracing is enabled,
and a (sub-)shell still simply seems to have vanished without trace,
the likely cause is that it was killed by a signal - and of those,
the most common that occurs is SIGPIPE.

show more ...

Related to PR bin/48875

Correct an issue found by Oguz <oguzismailuysal@gmail.com> and reported
in e-mail (on the bug-bash list initially!) with the code changed to deal
with PR bin/48875

With:

Related to PR bin/48875

Correct an issue found by Oguz <oguzismailuysal@gmail.com> and reported
in e-mail (on the bug-bash list initially!) with the code changed to deal
with PR bin/48875

With:

sh -c 'echo start at $SECONDS;
(sleep 3 & (sleep 1& wait) );
echo end at $SECONDS'

The shell should say "start at 0\nend at 1\n", but instead (before
this fix, in -9 and HEAD, but not -8) does "start at 0\nend at 3\n"
(Not in -8 as the 48875 changes were never pulled up)>

There was an old problem, fixed years ago, which cause the same symptom,
related to the way the jobs table was cleared (or not) in subshells, and
it seemed like that might have resurfaced.

But not so, the issue here is the sub-shell elimination, which was part
of the 48875 "fix" (not really, it wasn't really a bug, just sub-optimal
and unexpected behaviour).

What the shell actually has been running in this case is:

sh -c 'echo start at $SECONDS;
(sleep 3 & sleep 1& wait );
echo end at $SECONDS'

as the inner subshell was deemed unnecessary - all its parent would
do is wait for its exit status, and then exit with that status - we
may as well simply replace the current sub-shell with the new one,
let it do its thing, and we're done...

But not here, the running "sleep 3" will remain a child of that merged
sub-shell, and the "wait" will thus wait for it, along with the sleep 1
which is all it should be seeing.

For now, fix this by not eliminating a sub-shell if there are existing
unwaited upon children in the current one. It might be possible to
simply disregard the old child for the purposes of wait (and "jobs", etc,
all cmds which look at the jobs table) but the bookkeeping required to
make that work reliably is likely to take some time to get correct...

Along with this fix comes a fix to DEBUG mode shells, which, in situations
like this, could dump core in the debug code if the relevant tracing was
enabled, and add a new trace for when the jobs table is cleared (which was
added predating the discovery of the actual cause of this issue, but seems
worth keeping.) Neither of these changes have any effect on shells
compiled normally.

XXX pullup -9

show more ...

Be less conservative about when we do clear_traps() when we have
traps_invalid (that is, when we actually nuke the parent shell's
caught traps in a subshell). This allows more reasonable use of
"tra

Be less conservative about when we do clear_traps() when we have
traps_invalid (that is, when we actually nuke the parent shell's
caught traps in a subshell). This allows more reasonable use of
"trap -p" (and similar) in subshells than existed before (and in
particular, that command can be in a function now - there can also
be several related commands like
traps=$(trap -p INT; trap -p QUIT; trap -p HUP)
A side effect of all of this is that
(eval "$(trap -p)"; ...)
now allows copying caught traps into a subshell environment, if desired.

Also att the ksh93 variant (the one not picked by POSIX as it isn't
generally as useful) of "trap -p" (but call it "trap -P" which extracts
just the trap action for named signals (giving more than one is usually
undesirable). This allows
eval "$(trap -P INT)"
to run the action for SIGINT traps, without needing to attempt to parse
the "trap -p" output.

show more ...

Remove extra semicolon.

Stop forcing the -e option off in the subshell createds for a command
substitution. This was inherited in the big "-e" fixup patch set (rev 1.50)
of Jan 2000, which came from dash. dash no longer

Stop forcing the -e option off in the subshell createds for a command
substitution. This was inherited in the big "-e" fixup patch set (rev 1.50)
of Jan 2000, which came from dash. dash no longer acts this way.

show more ...

After bug report 262 (from 2010)
https://austingroupbugs.net/view.php?id=252
the Austin Group decided to require processing of "--" by the "."
and "exec" commands to solve a problem where some shell

After bug report 262 (from 2010)
https://austingroupbugs.net/view.php?id=252
the Austin Group decided to require processing of "--" by the "."
and "exec" commands to solve a problem where some shells did
option processing for those commands (permitted) and others did
not (also permitted) which left no safe way to process a file
with a name beginning with "-".

This has finally made its way into what will be the next version of
the POSIX standard.

Since this shell did no option processing at all for those commands,
we need to update. This is that update.

The sole effect is that a "--" 'option' (to "." or "exec") is ignored.
This means that if you want to use "--" as the arg to one of those
commands, it needs to be given twice ". -- --". Apart from that there
should be no difference at all (though the "--" can now be used in other
situations, where we did not require it before, and still do not).

show more ...

Use fork() rather than vfork() when forking to run a background
process with redirects. If we use vfork() and a redirect hangs
(eg: opening a fifo) which the parent was intended to unhang,
then the

Use fork() rather than vfork() when forking to run a background
process with redirects. If we use vfork() and a redirect hangs
(eg: opening a fifo) which the parent was intended to unhang,
then the parent never gets to continue to unhang the child.

eg: mkfifo f; cat <f &; echo foo>f

The parent should not be waiting for a background process, even
just for its exec() to complete. if there are no redirects there
is (should be) nothing left that might be done that will cause any
noticeable delay, so vfork() should be safe in all other cases.

show more ...

PR bin/54743

Having traps set should not enforce a fork for the next command,
whatever that command happens to be, only for commands which would
normally fork if they weren't the last command expect

PR bin/54743

Having traps set should not enforce a fork for the next command,
whatever that command happens to be, only for commands which would
normally fork if they weren't the last command expected to be
executed (ie: builtins and functions shouldn't be exexuted in a
sub-shell merely because a trap is set).

As it was (for example)
trap 'whatever' SIGANY; wait $anypid
was guaranteed to fail the wait, as the subshell it was executed
in could not have any children.

XXX pullup -9

show more ...

When a return occurs in the test part of a loop statement (while/until)
(inside a function or dot script) the exit status of that return
statement should become the exit status of the function (or do

When a return occurs in the test part of a loop statement (while/until)
(inside a function or dot script) the exit status of that return
statement should become the exit status of the function (or dot
script) - we were ignoring it,

That is
fn() { while return 7; do return 9; done; return 11; }
should exit with status 7. It was exiting 0.

This is apparently another old ash bug that has been fixed
everywhere else in the past.

Issue pointed out by Martijn Dekker, (fairly obvious) fix borrowed
from FreeBSD, due for return sometime next century.

show more ...

DEBUG mode build changes - add extra trace output.

NFC for any normal shell build.

Delete extern decl for trap[] - hasn't been needed for a while now.