This commit is intended to be what was intended to happen in the
commit of Sun Nov 10 01:22:24 UTC 2024, see:

http://mail-index.netbsd.org/source-changes/2024/11/10/msg154310.html

The commit me

This commit is intended to be what was intended to happen in the
commit of Sun Nov 10 01:22:24 UTC 2024, see:

http://mail-index.netbsd.org/source-changes/2024/11/10/msg154310.html

The commit message for that applies to this one (wholly). I believe that
the problem with that version which caused it to be reverted has been found
and fixed in this version (a necessary change was made as part of one of
the fixes, but the side-effect implications of that were missed -- bad bad me.)

In addition, I found some more issues with setting close-on-exec on other
command lines

With:
func 3>whatever

fd 3 (anything > 2) got close on exec set. That makes no difference
to the function itself (nothing gets exec'd therefore nothing gets closed)
but does to any exec that might happen running a command within the function.

I believe that if this is done (just as if "func" was a regular command,
and not a function) such open fds should be passed through to anything
they exec - unless the function (or other command) takes care to close the
fd passed to it, or explicitly turn on close-on exec.

I expect this usage to be quite rare, and not make much practical difference.

The same applies do builtin commands, but is even less relevant there, eg:

printf 3>whatever

would have set close-on-exec on fd 3 for printf. This is generally
completely immaterial, as printf - and most other built-in commands -
neither uses any fd other than (some of) 0 1 & 2, nor do they exec anything.

That is, except for the "exec" built-in which was the focus of the original
fix (mentioned above) and which should remain fixed here, and for the "."
command.

Because of that last one (".") close-on-exec should not be set on built-in
commands (any of them) for redirections on the command line. This will
almost never make a difference - any such redirections last only as long
as the built-in command lasts (same with functions) and so will generally
never care about the state of close-on-exec, and I have never seen a use
of the "." command with any redirections other than stderr (which is unaffected
here, fd's <= 2 never get close-on-exec set). That's probably why no-one
ever noticed.

There are still "fd issues" when running a (non #!) shell script, that
are hard to fix, which we should probably handle the way most other shells
have, by simply abandoning the optimisation of not exec'ing a whole new
shell (#! scripts do that exec) and just doing it that way. Issues solved!
One day.

show more ...

Revert the recent change until I can work out how things are broken.

exec builtin command redirection fixes

Several changes, all related to the exec special built in command,
or to close on exec, one way or another. (Except a few white space
and comment additions,

exec builtin command redirection fixes

Several changes, all related to the exec special built in command,
or to close on exec, one way or another. (Except a few white space
and comment additions, KNF, etc)

1. The bug found by Edgar Fuß reported in:
http://mail-index.netbsd.org/tech-userlevel/2024/11/05/msg014588.html
has been fixed, now "exec N>whatever" will set close-on-exec for fd N
(as do ksh versions, and allowed by POSIX, though other shells do not)
which has happened now for many years. But "exec cmd N>whatever"
(which looks like the same command when redirections are processed)
which was setting close-on-exec on N, now no longer does, so fd N
can be passed to cmd as an open fd.

For anyone who cares, the big block of change just after "case CMDBUILTIN:"
in evalcommand() in eval.c is the fix for this (one line replaced by
about 90 ... though most of that is comments or #if 0'd example code
for later). It is a bit ugly, and will get worse if our exec command
ever gets any options, as others have, but it does work.

2. when the exec builtin utility is used to alter the shell's redirections
it is now "all or nothing". Previously the redirections were executed
left to right. If one failed, no more were attempted, but the earlier
ones remained. This makes no practical difference to a non-interactive
shell, as a redirection error causes that shell to exit, but it makes
a difference to interactive shells. Now if a redirection fails, any
earlier ones which had been performed are undone. Note however that
side-effects of redirections (like creating, or truncating, files in
the filesystem, cannot be reversed - just the shell's file descriptors
returned to how they were before the error).

Similarly usage errors on exec now exist .. our exec takes no options
(but does handle "--" as POSIX says it must - has done for ages).
Until now, that was the only magic piece of exec, running
exec -a name somecommand
(which several other shells support) would attempt to exec the "-a"
command, and most likely fail, causing immediate exit from the shell.
Now that is a usage error - a non-interactive shell still exits, as
exec is a special builtin, and any error from a special builtin causes
a non-interactive shell to exit. But now, an interactive shell will
no longer exit (and any redirections that were on the command will be
undone, the same as for a redirection error).

3. When a "close on exec" file descriptor is temporarily saved, so the
same fd can be redirected for another command (only built-in commands
and functions matter, redirects for file system commands happen after
a fork() and at that stage if anything goes wrong, the child simply
exits - but for non-forking commands, doing something like printf >file
required the previous stdout to be saved elsewhere, "file" opened to
be the new stdout, then when printf is finished, the old stdout moved
back. Anyway, if the fd being moved had close on exec set, then
when it was moved back, the close on exec was lost. That is now fixed.

4. The fdflags command no longer allows setting close on exec on stdin,
stdout, or stderr - POSIX requires that those 3 fd's always be open
(to something) when any normal command is invoked. With close-on-exec
set on one of these, that is impossible, so simply refuse it (when
"exec N>file" sets close on exec, it only does it for N>2).

Minor changes (should be invisible)

a. The shell now keeps track of the highest fd number it sees doing
normal operations (there are a few internal pipe() calls that aren't
monitored and a couple of others, but in general the shell will now
know the highest fd it ever saw allocated to it). This is mostly
for debugging.

b. calls to fcntl() passing an int as the "arg" are now all properly
cast to the void * that the fcntl kernel is expecting to receive.
I suspect that makes no actual difference to anything, but ...

show more ...

Meaningless gcc inspired change.

This is in code only compiled in DEBUG builds (so not part of any
normal NetBSD build).

NFC

The great shell trailing whitespace cleanup of 2023...
Inspired by private e-mail comments from mouse@

NFCI.

bin: remove unnecessary lint comment CONSTCOND

Since 2021-01-31, lint no longer warns about 'do ... while (0)'.

No functional change.

DEBUG mode only change. When pretty-printing a word from a parse
tree, don't display a CTLESC which is there only to protect a CTL*
char (a data char that happens to have the same value). No actua

DEBUG mode only change. When pretty-printing a word from a parse
tree, don't display a CTLESC which is there only to protect a CTL*
char (a data char that happens to have the same value). No actual
CTL* chars are printed as data, so no escaping is needed to protect
data which just happens to look the same. Dropping this avoids the
possibility of confusion/ambiguity in what the word actually contains.

NFC for any normal shell build (very little of this file gets compiled there)

show more ...

NFCI - DEBUG mode only change.

Add tracing of lexical analyser operations. This is deliberately
kept out of the normal "all on" set as it makes a *lot* of noise
when enabled (especially in verbose

NFCI - DEBUG mode only change.

Add tracing of lexical analyser operations. This is deliberately
kept out of the normal "all on" set as it makes a *lot* of noise
when enabled (especially in verbose mode) - but when needed, it
helps (evidence for which is coming soon).

As usual, no doc, you need the sources (and of course, a specially
built sh to even be able to enable it.)

show more ...

DEBUG mode shell cleanups (NFC for any normal shell).

Add an error DEBUG trace in exraise() (when the shell has detected
some error or signal, and is aborting what it is doing)

Fix an arith error i

DEBUG mode shell cleanups (NFC for any normal shell).

Add an error DEBUG trace in exraise() (when the shell has detected
some error or signal, and is aborting what it is doing)

Fix an arith error in DEBUG bit assignments (harmless as we haven't
reached the limit of flags yet), and add some missing (recently added)
debug flags so they are turned on when the user (ie: me) asks for
"everything".

show more ...

Remove a DEBUG mode transition mechanism (for the transition from
the ancient DEBUG TRACE() method, to the newer CTRACE() et. al.)
that turns out never really needed committing - the mechanism, and
t

Remove a DEBUG mode transition mechanism (for the transition from
the ancient DEBUG TRACE() method, to the newer CTRACE() et. al.)
that turns out never really needed committing - the mechanism, and
the code that obsoleted it, were committed together (May 2017).
[It was useful to me while getting to that state...]

NFC. Not even with DEBUG shells.

show more ...

NFC: DEBUG mode only change. When tracing, show >&- as ">&-"
rather than ">&-1" (the same op as >&n where internally n < 0
means "close")

DEBUG mode only change (ie: no effect to any normal shell).

Add tracing of pattern matching (aid in debugging various issues.)

NFC: DEBUG only change - provide an externally visible (to the DEBUG sh
internals) interface to one of the internal (private to trace code) functions

NFC - DEBUG mode only change - complete a change made earlier (marking
the line number when included in the trace line tag to show whether it
comes from the parser, or the elsewhere as they tend to b

NFC - DEBUG mode only change - complete a change made earlier (marking
the line number when included in the trace line tag to show whether it
comes from the parser, or the elsewhere as they tend to be quite different).
Initially only one case was changed, while I pondered whether I liked it
or not. Now it is all done... Also when there is a line tag at all,
always include the root/sub-shell indicator character, not only when the
pid is included.

show more ...

NFC (normal use) - DEBUG only change, when showing empty arg list don't
omit terminating \n.

Correct spelling in comments of DEBUG only code...

A better LINENO implementation. This version deletes (well, #if 0's out)
the LINENO hack, and uses the LINENO var for both ${LINENO} and $((LINENO)).
(Code to invert the LINENO hack when required,

A better LINENO implementation. This version deletes (well, #if 0's out)
the LINENO hack, and uses the LINENO var for both ${LINENO} and $((LINENO)).
(Code to invert the LINENO hack when required, like when de-compiling the
execution tree to provide the "jobs" command strings, is still included,
that can be deleted when the LINENO hack is completely removed - look for
refs to VSLINENO throughout the code. The var funclinno in parser.c can
also be removed, it is used only for the LINENO hack.)

This version produces accurate results: $((LINENO)) was made as accurate
as the LINENO hack made ${LINENO} which is very good. That's why the
LINENO hack is not yet completely removed, so it can be easily re-enabled.
If you can tell the difference when it is in use, or not in use, then
something has broken (or I managed to miss a case somewhere.)

The way that LINENO works is documented in its own (new) section in the
man page, so nothing more about that, or the new options, etc, here.

This version introduces the possibility of having a "reference" function
associated with a variable, which gets called whenever the value of the
variable is required (that's what implements LINENO). There is just
one function pointer however, so any particular variable gets at most
one of the set function (as used for PATH, etc) or the reference function.
The VFUNCREF bit in the var flags indicates which func the variable in
question uses (if any - the func ptr, as before, can be NULL).

I would not call the results of this perfect yet, but it is close.

show more ...

More DEBUG mode changes. As usual, read the source if you care.

DEBUG mode only change - mostly to output when option to show shell
internal sub-process nesting is enabled, and very deep nesting levels exist.

NFC for anyone else.

DEBUG mode shell update (changes nothing for shells which are not
compiled for DEBUG.)

Add debug builtin command, and corresponding -D command line option.
As usual, for DEBUG related stuff, read th

DEBUG mode shell update (changes nothing for shells which are not
compiled for DEBUG.)

Add debug builtin command, and corresponding -D command line option.
As usual, for DEBUG related stuff, read the source for info, that's
all there is about this.

This completes the infrastructure changes for the updated DEBUG TRACE
mechanism, so now converting the rest of the shell's internal tracing
can happen as desired - piecemeal.

show more ...

The beginnings of the great shell DEBUG (tracing) upgrade of 2017...

First, be aware that the DEBUG spoken of here has nothing whatever to
do with MKDEBUG=true type builds of NetBSD. The only way

The beginnings of the great shell DEBUG (tracing) upgrade of 2017...

First, be aware that the DEBUG spoken of here has nothing whatever to
do with MKDEBUG=true type builds of NetBSD. The only way to get a
DEBUG shell is to build it yourself manually.

That said, for non-DEBUG shells, this change makes only one slight
(trivial really) difference, which should affect nothing.

Previously some code was defined like ...

function(args)
{
#ifdef DEBUG
/* function code goes here */
#endif
}

and called like ...

#ifdef DEBUG
function(params);
#endif

resulting in several empty functions that are never called being
defined in non-DEBUG shells. Those are now gone. If you can detect
the difference any way other than using "nm" or similar, I'd be very
surprised...

For DEBUG shells, this introduces a whole new TRACE() setup to use
to assist in debugging the shell.

I have had this locally (uncommitted) for over a year... it helps.

By itself this change is almost useless, nothing really changes, but
it provides the framework to allow other TRACE() calls to be updated
over time. This is why I had not committed this earlier, my previous
version required a flag day, with all the shell's internal tracing
being updated a once - which I had done, but that shell version has
bit-rotted so badly now it is almost useless...

Future updates will add the mechanism to allow the new stuff to actually
be used in a productive way, and following that, over time, gradual
conversion of all the shell tracing to the updated form (as required,
or when I am bored...)

The one useful change that we do get now is that the fd that the shell
uses for tracing (which was usually 3, but not any more) is now protected
from user/script interference, like all the other shell inernal fds.

There is no doc (nor will there be) on any of this, if you are not reading
the source code it is useless to you, if you are, you know how it works.

show more ...

If we are going to permit
! ! pipeline
(And for now the other places where ! is permitted)
we should at least generate the logically correct exit
status:
! ! (exit 5); echo $?
should print 1, not 5

If we are going to permit
! ! pipeline
(And for now the other places where ! is permitted)
we should at least generate the logically correct exit
status:
! ! (exit 5); echo $?
should print 1, not 5. ksh and bosh do it this way - and it makes sense.
bash and the FreeBSD sh echo "5" (as did we until now.)
dash, zsh, yash all enforce the standard syntax, and prohibit this.

show more ...

This change affects shells compiled in DEBUG mode only, for normal
shells (anything made by build.sh) there is no change at all.

In DEBUG shells, when tree dumping, remember to include NNOT (same
om

This change affects shells compiled in DEBUG mode only, for normal
shells (anything made by build.sh) there is no change at all.

In DEBUG shells, when tree dumping, remember to include NNOT (same
omission as was just corrected in jobs.c :1.81) - of course, here there
are lots of other node types not handled as well.

ALso, avoid a core dump bug when doing a tree dump of a pieline
where the commands are not all simple commands (which can only
happen with a command string like " cmd | ! cmd | ... ". The "!"
in the middle is utter nonsense, and should be forbidden, but
for now, at least avoid a core dump.

show more ...

Undo local changes not intended to be committed (and certainly not
with that commit message) in the previous update. This stuff works,
and will probably appear sometime, but not right now.

Have "make clean" remove sh.html1 and adapt it to clean trace files
the way they have been generated the past 20 years or so...