This commit is intended to be what was intended to happen in the
commit of Sun Nov 10 01:22:24 UTC 2024, see:

http://mail-index.netbsd.org/source-changes/2024/11/10/msg154310.html

The commit me

This commit is intended to be what was intended to happen in the
commit of Sun Nov 10 01:22:24 UTC 2024, see:

http://mail-index.netbsd.org/source-changes/2024/11/10/msg154310.html

The commit message for that applies to this one (wholly). I believe that
the problem with that version which caused it to be reverted has been found
and fixed in this version (a necessary change was made as part of one of
the fixes, but the side-effect implications of that were missed -- bad bad me.)

In addition, I found some more issues with setting close-on-exec on other
command lines

With:
func 3>whatever

fd 3 (anything > 2) got close on exec set. That makes no difference
to the function itself (nothing gets exec'd therefore nothing gets closed)
but does to any exec that might happen running a command within the function.

I believe that if this is done (just as if "func" was a regular command,
and not a function) such open fds should be passed through to anything
they exec - unless the function (or other command) takes care to close the
fd passed to it, or explicitly turn on close-on exec.

I expect this usage to be quite rare, and not make much practical difference.

The same applies do builtin commands, but is even less relevant there, eg:

printf 3>whatever

would have set close-on-exec on fd 3 for printf. This is generally
completely immaterial, as printf - and most other built-in commands -
neither uses any fd other than (some of) 0 1 & 2, nor do they exec anything.

That is, except for the "exec" built-in which was the focus of the original
fix (mentioned above) and which should remain fixed here, and for the "."
command.

Because of that last one (".") close-on-exec should not be set on built-in
commands (any of them) for redirections on the command line. This will
almost never make a difference - any such redirections last only as long
as the built-in command lasts (same with functions) and so will generally
never care about the state of close-on-exec, and I have never seen a use
of the "." command with any redirections other than stderr (which is unaffected
here, fd's <= 2 never get close-on-exec set). That's probably why no-one
ever noticed.

There are still "fd issues" when running a (non #!) shell script, that
are hard to fix, which we should probably handle the way most other shells
have, by simply abandoning the optimisation of not exec'ing a whole new
shell (#! scripts do that exec) and just doing it that way. Issues solved!
One day.

show more ...

Revert the recent change until I can work out how things are broken.

Back out the fcntl() 3rd arg -> (void *) change, leave it as an
untouched int. The other way seems to break things.

exec builtin command redirection fixes

Several changes, all related to the exec special built in command,
or to close on exec, one way or another. (Except a few white space
and comment additions,

exec builtin command redirection fixes

Several changes, all related to the exec special built in command,
or to close on exec, one way or another. (Except a few white space
and comment additions, KNF, etc)

1. The bug found by Edgar Fuß reported in:
http://mail-index.netbsd.org/tech-userlevel/2024/11/05/msg014588.html
has been fixed, now "exec N>whatever" will set close-on-exec for fd N
(as do ksh versions, and allowed by POSIX, though other shells do not)
which has happened now for many years. But "exec cmd N>whatever"
(which looks like the same command when redirections are processed)
which was setting close-on-exec on N, now no longer does, so fd N
can be passed to cmd as an open fd.

For anyone who cares, the big block of change just after "case CMDBUILTIN:"
in evalcommand() in eval.c is the fix for this (one line replaced by
about 90 ... though most of that is comments or #if 0'd example code
for later). It is a bit ugly, and will get worse if our exec command
ever gets any options, as others have, but it does work.

2. when the exec builtin utility is used to alter the shell's redirections
it is now "all or nothing". Previously the redirections were executed
left to right. If one failed, no more were attempted, but the earlier
ones remained. This makes no practical difference to a non-interactive
shell, as a redirection error causes that shell to exit, but it makes
a difference to interactive shells. Now if a redirection fails, any
earlier ones which had been performed are undone. Note however that
side-effects of redirections (like creating, or truncating, files in
the filesystem, cannot be reversed - just the shell's file descriptors
returned to how they were before the error).

Similarly usage errors on exec now exist .. our exec takes no options
(but does handle "--" as POSIX says it must - has done for ages).
Until now, that was the only magic piece of exec, running
exec -a name somecommand
(which several other shells support) would attempt to exec the "-a"
command, and most likely fail, causing immediate exit from the shell.
Now that is a usage error - a non-interactive shell still exits, as
exec is a special builtin, and any error from a special builtin causes
a non-interactive shell to exit. But now, an interactive shell will
no longer exit (and any redirections that were on the command will be
undone, the same as for a redirection error).

3. When a "close on exec" file descriptor is temporarily saved, so the
same fd can be redirected for another command (only built-in commands
and functions matter, redirects for file system commands happen after
a fork() and at that stage if anything goes wrong, the child simply
exits - but for non-forking commands, doing something like printf >file
required the previous stdout to be saved elsewhere, "file" opened to
be the new stdout, then when printf is finished, the old stdout moved
back. Anyway, if the fd being moved had close on exec set, then
when it was moved back, the close on exec was lost. That is now fixed.

4. The fdflags command no longer allows setting close on exec on stdin,
stdout, or stderr - POSIX requires that those 3 fd's always be open
(to something) when any normal command is invoked. With close-on-exec
set on one of these, that is impossible, so simply refuse it (when
"exec N>file" sets close on exec, it only does it for N>2).

Minor changes (should be invisible)

a. The shell now keeps track of the highest fd number it sees doing
normal operations (there are a few internal pipe() calls that aren't
monitored and a couple of others, but in general the shell will now
know the highest fd it ever saw allocated to it). This is mostly
for debugging.

b. calls to fcntl() passing an int as the "arg" are now all properly
cast to the void * that the fcntl kernel is expecting to receive.
I suspect that makes no actual difference to anything, but ...

show more ...

PR bin/53550

Here we go again... One more time to redo how here docs are
processed (it has been a few years since the last time!)

This is actually a relatively minor change, mostly to timimg
(to

PR bin/53550

Here we go again... One more time to redo how here docs are
processed (it has been a few years since the last time!)

This is actually a relatively minor change, mostly to timimg
(to just when things happen). Now here docs are expanded at the
same time the "filename" word in a redirect is expanded, rather than
later when the heredoc was being sent to its process. This actually
makes things more consistent - but does break one of the ATF tests
which was testing that we were (effectively) internally inconsistent
in this area.

Not all shells agree on the context in which redirection expansions
should happen, some make any side effects visible to the parent shell
(the majority do) others do the redirection expansions in a subshell
so any side effcts are lost. We used to have a foot in each camp,
with the majority for everything but here docs, and the minority for
here docs. Now we're all the way with LBJ ... (or something like that).

show more ...

Detect write errors to stdout, and exit(1) from some built-in
commands which (primarily) are used just to generate output
(or with a particular option combination do so).

DEBUG mode changes only. NFC (NC) for any normally compiled shell.

Mostly adding DEBUG mode tracing (when appropriate verbose tracing
is enabled generally) whenever a shell (including sushell) pro

DEBUG mode changes only. NFC (NC) for any normally compiled shell.

Mostly adding DEBUG mode tracing (when appropriate verbose tracing
is enabled generally) whenever a shell (including sushell) process
exits, so shells that the tracing should indicate why ehslls that
vanish did that.

Note for future investigators: if the relevant tracing is enabled,
and a (sub-)shell still simply seems to have vanished without trace,
the likely cause is that it was killed by a signal - and of those,
the most common that occurs is SIGPIPE.

show more ...

Fix an ordering error in the previous (and even earlier, going back
a way, but made more serious with the recent changes).

The n>&n operation (more or less a no-op, except it clears CLOEXEC)
should

Fix an ordering error in the previous (and even earlier, going back
a way, but made more serious with the recent changes).

The n>&n operation (more or less a no-op, except it clears CLOEXEC)
should precede almost everything else - and simply be made to fail if
an attempt is made to apply it to a sh internal fd.

We were renumbering the internal fd (the n> part considered first)
which was dumb, but OK, before, but now rejecting the operation
(the >&n) part when n should not be visible to the script. That
made something of a mess (and could lead to the shell believing its
job control tty was at a fd it never got moved to).

Do things in the correct order, and simply fail that case for internal
fds (for every other n>xxx for any xxx sh simply renumbers its internal fd
n to some other fd before attempting the operation, even n>&- ... those are
all fine).

[In all the above the '>' is used in place of any redirect operator].

show more ...

Improve the solution for the 2nd access to a fd which shouldn't
be available ("13") issue reported by Jan Schaumann on netbsd-users.

This fixes a bug in the earlier fix (a day or so ago) which could

Improve the solution for the 2nd access to a fd which shouldn't
be available ("13") issue reported by Jan Schaumann on netbsd-users.

This fixes a bug in the earlier fix (a day or so ago) which could allow the
shell's idea of which fd range was in use by the script to get wildly
incorrect, but now also actually looks to see which fd's are in use as
renamed other user fd's during the lifetime of a redirection which needs
to be able to be undone (most redirections occur after a fork and are
permanent in the child process). Attempting to access such a fd (as with
attempts to access a fd in use by the shell for its own purposes) is treated
as an attempt to access a closed fd (EBADF). Attempting to reuse the fd
for some other purpose (which should be rare, even for scripts attempting
to cause problems, since the shell generally knows which fds the script
wants to use, and avoids them) will cause the renamed (renumbered) fd
to be renamed again (moved aside to some other available fd), just as
happens with the shell's private fds.

Also, when a generic fd is required, don't give up because of EMFILE
or similar unless there are no available fds at all (we might prefer >10
or bigger, but if there are none there, use anything). This avoids
redirection errors when ulimit -n has been set small, and all the fds >10
that are available have been used, but we need somewhere to park the old
user of a fd while we reuse that fd for the redirection.

show more ...

Deal with some issues where fds intended only for internal use
by the shell were available for manipulation by scripts (or the user).
These issues were reported by Jan Schaumann on netbsd-users.

The

Deal with some issues where fds intended only for internal use
by the shell were available for manipulation by scripts (or the user).
These issues were reported by Jan Schaumann on netbsd-users.

The first allows the user to reference sh internal fds, and is
a simple fix - any sh internal fd is simply treated as if it were closed
when referenced by the script. These fds can be discovered by
examining /proc/N/fd so it is not difficult for a script to discover
which fd it should attempt to access.

The second allows the user to reference a user level fd which is
one that is normally available to it, but at a point where it should
no longer be visible (when that fd has been redirected, for a built
in command, so the original fd needs to be saved so it can be restored,
the saving fd should not be accessible). It is not as easy for the
script to determine which fd to attempt here, as the relevant one
exists only during the lifetime of a built-in command (and similar),
but there are ways in some cases (aside from looking at /proc from
another process).

Fix this one by watching which fds the user script is attempting
to use, and avoid using those as temporary fds. This is possible in
this case as we know what command is being run, before we need to
save the fds it uses. That's different from the earlier case where
when the shell allocates its fds we have no idea what it might
reference later.

Also clean up a couple of other minor code issues (NFC intended) that
I noticed while here...

show more ...

The previous commit was obviously made by a broken mindless automoton
with an IQ that underflows when one attempts to enter it as an
unnormalised 160 bit long long double...

Whoever would believe th

The previous commit was obviously made by a broken mindless automoton
with an IQ that underflows when one attempts to enter it as an
unnormalised 160 bit long long double...

Whoever would believe that (~0 & anything) was a meaningful thing
to write? And three times in one #define. That could not possibly
have been me, could it?

Simplify, simplify, simplify. NFC.

show more ...

Inspired by (really the need for) Maya's patch to pkgsrc/shells/bash
to allow bash to build fdflags on Solaris 10, here are some mods that
fix that, and some other similar issues in the NetBSD versio

Inspired by (really the need for) Maya's patch to pkgsrc/shells/bash
to allow bash to build fdflags on Solaris 10, here are some mods that
fix that, and some other similar issues in the NetBSD version of fdflags.

The bash implementation of fdflags is based upon the one Christos did for
the NetBSD sh, so the issues are similar ... the NetBSD sh cannot yet
(easily anyway) build on anything except NetBSD, so this change makes
no current difference at all (just adds some compile time tests (#ifdef)
which always work out the way things did before, when built on NetBSD).

However, there is no system on which any modern shell can hope to work
which does not support close on exec, or fcntl(F_SETFD,...) to set it.
The O_CLOEXEC and FD_CLOEXEC definitions might not exist, but close on
exec can still be manipulated. Since the primary rationale for
the fdflags builtin was to be able to manipulate that state bit from
scripts, it would be annoying to lose that one, and keep all the (less
important) others, just because O_CLOEXEC is not defined, so do the
fix (workaround) a different way than was done in the bash patch.

Further, more than fdflags() will fail if O_CLOEXEC is not defined,
so handle that as well.

Also fix another oddity ... (noticed by reading the code) - if
fcntl(F_GETFL,...) returned any bits set that we don't understand,
the code was supposed to simply print their values as a hex constant,
when fdflags is run with -v. However, the getflags() function was
clearing all bits that the code did not know about ... so there is
no way any unknown bit could ever make it out to be printed. Handle
that a different way - instead of clearing unknown bits, clear any
bits that get returned which we understand, but do not want to deal
with (stuff like O_WRONLY, which should not be returned from the
fcntl(), but who knows...) Leave any unknown bits that happen to be
set, set, so that printone() can display them if appropriate.
(This is most likely to happen when running an older shell on a new
kernel where the kernel supports some new flag that the shell has
not been taught to understand).

NFCI that anyone should notice anytime soon.

show more ...

DEBUG mode change only. Add one extra trace point. NFC for normal builds.

INTON / INTOFF audit and cleanup.

No visible differences expected - there is a remote chance that
some internal lossage may no longer occur in interactive shells
that receive SIGINT (untrapped) at i

INTON / INTOFF audit and cleanup.

No visible differences expected - there is a remote chance that
some internal lossage may no longer occur in interactive shells
that receive SIGINT (untrapped) at inopportune times, but you would
have had to have been very unlucky to have ever suffered from that.

show more ...

Fix typo: O_ALTIO -> O_ALT_IO

Noted by @jbeich via GitHub.

Fix the <> redirection operator, which has been broken since it was
first implemented in response to PR bin/4966 (PR Feb 1998, fix Feb 1999).

The file named should not be truncated.

No other shell

Fix the <> redirection operator, which has been broken since it was
first implemented in response to PR bin/4966 (PR Feb 1998, fix Feb 1999).

The file named should not be truncated.

No other shell truncates the file (<> was added to FreeBSD sh in Oct 2000,
and did not include O_TRUNC) and POSIX certainly does not suggest that
should happen (just that the file is to be created if it does not exist.)

Bug pointed out in off-list e-mail by Martijn Dekker

show more ...

NFC: DEBUG (compile time) mode only change:
Add some extra redirection (fd manipulation) tracing.
While here, some white space fixes, and very minor KNF.

DEBUG mode only change. Add some tracing. NFC (without DEBUG).

Include redirections in trace output from "set -x"

Now that the shell is protecting its internal fds properly, moving
them whenever the user tries to step on one, we can change our behaviour
back to what the kernel considers to be that of a well beha

Now that the shell is protecting its internal fds properly, moving
them whenever the user tries to step on one, we can change our behaviour
back to what the kernel considers to be that of a well behaved shell
(wrt file descriptor usage). If our user causes problems, we will soon
move into recalcitrant process territory, but that should normally be
rare. This should reduce kernel overheads a little.

show more ...

Allow abbreviations of option names for the "fdflags -s" command.
While documenting that, cleanup markup of the fdflags section of the
man page.

When opening a file descritor with "exec n>/file" (and similar) only
set the close-on-exec bit when not in posix mode (to comply with posix...)
OK: christos@

Keep track of which file descriptors the shell is using for its
own purposes, and move them elsewhere whenever a user redirection
happens to pick the same number. With this we can move the shell
fi

Keep track of which file descriptors the shell is using for its
own purposes, and move them elsewhere whenever a user redirection
happens to pick the same number. With this we can move the shell
file descriptors back to lower values (be slightly kinder to the kernel)
since we can no longer clash. (Also get rid of a little old unneeded code.)

This also completes the fdflags command, which no longer permits access
to (by way or either obtaining, or changing) the shell's internal fds.

show more ...

Keep track of the biggest fd used by, or available to, the user/script
and use that to control which fd's are examined by a (bare) fdflags
(with no fd args).

Usually this will mean that fdflags will

Keep track of the biggest fd used by, or available to, the user/script
and use that to control which fd's are examined by a (bare) fdflags
(with no fd args).

Usually this will mean that fdflags will no longer show the shell's
internal use fds, only user fds.

This is only a partial fix however, a user can easily discover the
shell's fd usage (eg: using fstat) and can then still use fdflags to
manipulate those fds (or even send output to them).

The shell needs to monitor its own fd usage better, and keep out of
the way of user fds - coming sometime later...

show more ...

When verifying the size of the fd arg for fdflags skip leading 0's
(fdflags 0000000001 should work, fdflags 10000000 should not)