Fix processing of unknown variable expansion types.

Our shell is (was) one of the last not to do this correctly.

Expansions are supposed to happen only when the command in which
they occur is being

Fix processing of unknown variable expansion types.

Our shell is (was) one of the last not to do this correctly.

Expansions are supposed to happen only when the command in which
they occur is being executed, not while it is being parsed.
If the expansion only happens them, errors should only be
detected then.

Make it work like that (I saw after I fixed this that FreeBSD
had done it, long ago, almost the same way - it is kind of an
obvious thing to do).

This will allow code like

if test it is shell X
then
commands using shell X specific expansion ops
else if it is shell Y
then
commands using shell Y specific expansion ops
else ...
fi

Previously expansion errors were detected while parsing, so
if we're not shell X, and don't implement something that it
does (some extension to the standard) that would have generated
a parser syntax error, and the script could not be executed
(despite the line with the error never being executed).

Note that this change does not handle all such possible
extensions, just this one. Others are much harder.

One side effect of this change is that sh will now continue
reading a variable expansion until it locates the terminating
'}' (in ${var} forms) regardless of how broken it obviously
is (to our shell) whereas previously it would have bailed out
as soon as an oddity was spotted.

show more ...

bin: fix lint warning "effectively discards 'const'"

For example: src/bin/ed/io.c(339): warning: call to 'strchr' effectively
discards 'const' from argument [346]

No binary change.

Implement expandvar() : runs var/arith/cmdsub expansions on var value

expandvar() is like expandenv() and expandstr() - each expands
variable values in different contexts, expandenv() when processin

Implement expandvar() : runs var/arith/cmdsub expansions on var value

expandvar() is like expandenv() and expandstr() - each expands
variable values in different contexts, expandenv() when processing
environment vars at startup (and there is no existing state to lose),
but is always using unsafe data, so never runs command substitutions,
expandstr() while in the middle of processing commands (mostly to
expand the prompt variables, so there is probably half a command
tree in the process of being built) - it uses the promptcmds option
to decide whether to run command substitutions. expandvar() is for
intermediate situations, where a variable is to be used during
processing. It will run command substitutions if the variable
value concerned was not imported from the environment.

There are currently no uses of expandvar(), so this change alters
nothing in the shell - but there will be in the near future.

show more ...

Work around a probably gcc12 bug in detecting "potentially clobbered"
variables after longjmp() for some architectures (sh3 at least).

This should allow the workaround to disable those warnings for

Work around a probably gcc12 bug in detecting "potentially clobbered"
variables after longjmp() for some architectures (sh3 at least).

This should allow the workaround to disable those warnings for this
file to be removed.

In the affected function the extra var & assignment added should simply
be deleted by any good optimiser, but if not, it doesn't matter, as
performance of this function (expandonstack()) is almost irrelevant.

show more ...

The great shell trailing whitespace cleanup of 2023...
Inspired by private e-mail comments from mouse@

NFCI.

fix various typos in comments.

Avoid generating error messages implying that user errors are illegal.

s/desireable/desirable/ in comments.

s/existance/existence/ in comment.

Fix value of ${LINENO} in "for" commands.

This affects (as best I can tell) only uses of ${LINENO} in PS4
when -x is enabled (and perhaps only when the list contains no
expansions). "for" like "ca

Fix value of ${LINENO} in "for" commands.

This affects (as best I can tell) only uses of ${LINENO} in PS4
when -x is enabled (and perhaps only when the list contains no
expansions). "for" like "case" (which was already handled) is
special in that it generates trace output before actually executing
any kind of simple command.

show more ...

Improve the solution for the 2nd access to a fd which shouldn't
be available ("13") issue reported by Jan Schaumann on netbsd-users.

This fixes a bug in the earlier fix (a day or so ago) which could

Improve the solution for the 2nd access to a fd which shouldn't
be available ("13") issue reported by Jan Schaumann on netbsd-users.

This fixes a bug in the earlier fix (a day or so ago) which could allow the
shell's idea of which fd range was in use by the script to get wildly
incorrect, but now also actually looks to see which fd's are in use as
renamed other user fd's during the lifetime of a redirection which needs
to be able to be undone (most redirections occur after a fork and are
permanent in the child process). Attempting to access such a fd (as with
attempts to access a fd in use by the shell for its own purposes) is treated
as an attempt to access a closed fd (EBADF). Attempting to reuse the fd
for some other purpose (which should be rare, even for scripts attempting
to cause problems, since the shell generally knows which fds the script
wants to use, and avoids them) will cause the renamed (renumbered) fd
to be renamed again (moved aside to some other available fd), just as
happens with the shell's private fds.

Also, when a generic fd is required, don't give up because of EMFILE
or similar unless there are no available fds at all (we might prefer >10
or bigger, but if there are none there, use anything). This avoids
redirection errors when ulimit -n has been set small, and all the fds >10
that are available have been used, but we need somewhere to park the old
user of a fd while we reuse that fd for the redirection.

show more ...

Deal with some issues where fds intended only for internal use
by the shell were available for manipulation by scripts (or the user).
These issues were reported by Jan Schaumann on netbsd-users.

The

Deal with some issues where fds intended only for internal use
by the shell were available for manipulation by scripts (or the user).
These issues were reported by Jan Schaumann on netbsd-users.

The first allows the user to reference sh internal fds, and is
a simple fix - any sh internal fd is simply treated as if it were closed
when referenced by the script. These fds can be discovered by
examining /proc/N/fd so it is not difficult for a script to discover
which fd it should attempt to access.

The second allows the user to reference a user level fd which is
one that is normally available to it, but at a point where it should
no longer be visible (when that fd has been redirected, for a built
in command, so the original fd needs to be saved so it can be restored,
the saving fd should not be accessible). It is not as easy for the
script to determine which fd to attempt here, as the relevant one
exists only during the lifetime of a built-in command (and similar),
but there are ways in some cases (aside from looking at /proc from
another process).

Fix this one by watching which fds the user script is attempting
to use, and avoid using those as temporary fds. This is possible in
this case as we know what command is being run, before we need to
save the fds it uses. That's different from the earlier case where
when the shell allocates its fds we have no idea what it might
reference later.

Also clean up a couple of other minor code issues (NFC intended) that
I noticed while here...

show more ...

Fix a bug with here document processing reported on the austin group list
by oguzismailuysal@gmail.com (2021-09-08) (applies to all ash descendant
shells).

There were places in the parser where newl

Fix a bug with here document processing reported on the austin group list
by oguzismailuysal@gmail.com (2021-09-08) (applies to all ash descendant
shells).

There were places in the parser where newline tokens were handled
without reading any pending here documents (leaving those until a
later newline token).

Eg: in
: << do | for x in xxx
do
do echo $x
done

The here doc text (<<do) should start immediately after the next newline
(which is after xxx). But it wasn't happening until after the following
newline (after the line containing only "do").

Bizarrely, this
: << do | for x in xxx
do echo $x
do
done
"worked" because of that.

For other cases that also failed, see the hard_cases test case in
src/tests/bin/sh/t_here.sh Note that there's nothing magic about
the particular end delimiter word chosen here, any will do, using
the ones selected for the tests (and shown here) simply makes it
all more mysterious! The one here is the exact case reported.

Fix this by reading the here docs when processing newline tokens
when they are encountered in other than the "normal" way. Whether
this catches every possibility is unknown currently - may never be
known for certain, but there are no more I can currently think of.

No pullups needed, this isn't a significant enough bug (ie: no one
actually writes code like this) to warrant that.

show more ...

For now, probably forever, prohibit unquoted $ and ` in the names of
functions being defined (they can still be included if quoted).

If we parsed the way POSIX specifies (leaving the exact input tex

For now, probably forever, prohibit unquoted $ and ` in the names of
functions being defined (they can still be included if quoted).

If we parsed the way POSIX specifies (leaving the exact input text of
$ and ` expansions unaltered, until required to be expanded) this would
not be needed, as the name of a function being defined does not underbo
parameter, command, or arith expansions, so xxx$3() { : ; } would just
work. But for many reasons we don't do that (and are unlikely to ever,
though maintaing both forms might be an option someday) - which led to
very obscure behaviour (if sh were compiled in DEBUG mode, even an abort())
and certainly nothing useful. So just prohibit these uses for now.
(A portable function name must be a "name" so this makes no difference
at all to posix compat applications/scripts).

A doc update is pending (the updated sh.1 also contains updates in other
areas not yet appropriate to commit).

show more ...

Remove extra semicolon.

Correct a typo in a comment, 08x0 was meant to be 0x80 (duh!). NFC.

Fix an (apparent) ancient ash bug, that was apparently fixed sometime
in the past, but managed to re-surface...

The expression "${0+\}}" should expand to "}" not "\}"
Almost all other shells handle

Fix an (apparent) ancient ash bug, that was apparently fixed sometime
in the past, but managed to re-surface...

The expression "${0+\}}" should expand to "}" not "\}"
Almost all other shells handle it that way (incl FreeBSD & dash).

Issue pointed out by Martijn Dekker.

Add ATF sub-tests for the 4 old var expand operators (${var+word}
${var-word} ${var-word} and ${var?word} - including the forms
with the ':' included) and amongst those tests include test cases
for this issue, so if the bug tries to appear again, we can squash
it quicker. (The newer pattern matching operators are already
well tested as part of testing patterns.)

show more ...

Finish the fixes from Feb 4 for handling of random data that
matches the internal CTL* chars.

The earlier fixes handled CTL* char values in var expansions,
but not in various other places they can o

Finish the fixes from Feb 4 for handling of random data that
matches the internal CTL* chars.

The earlier fixes handled CTL* char values in var expansions,
but not in various other places they can occur (positional
parameters, $@ $* -- even potentially $0 and ~ expansions,
as well as byte strings generated from a \u in a $'' string).

These should all be correctly handled now. There is a new
ISCTL() macro to make the test, rather than using the old
BASESYNTAX[c]==CCTL form (which us still a viable alternative)
as the new way allows compiler optimisations, and less mem
references, so it should be smaller and faster.

Also, be sure in all cases to remove any CTLESC (or other)
CTL* chars from all strings before they are made available
for any external use (there was one case missed - which didn't
matter when we weren't bothering to escape the CTL* chars at
all.)

XXX pullup-8 (will need to be via a patch) along with the Feb 4 fixes.

show more ...

Add a check that the file descriptor mentioned in a N> or N< type
redirect operator is within range of what the code tree node can
hold. Currently this is a no-op change (the new error can never
oc

Add a check that the file descriptor mentioned in a N> or N< type
redirect operator is within range of what the code tree node can
hold. Currently this is a no-op change (the new error can never
occur) as the code already checks that N is in range for an int
(and errors if not) and the field in the node in which we store N
is also an int, so we cannot overflow - but fd's do not really need
to be that big (the max a typical kernel supports is < 10000) so
this just adds validation in case it ever happens that we decide we
can save some node size (ie: sh memory) by making that field smaller.

Note this is parse time error detection, and has no bearing upon
the execution time error that will occur if a script attempts to use
an fd that exceeds the process's max fd limit.

NFCI (for now anyway.)

show more ...

PR bin/53919

Suppress shell error messages while expanding $ENV (which also causes
errors while expanding $PS1 $PS2 and $PS4 to be suppressed as well).

This allows any random garbage that happens t

PR bin/53919

Suppress shell error messages while expanding $ENV (which also causes
errors while expanding $PS1 $PS2 and $PS4 to be suppressed as well).

This allows any random garbage that happens to be in ENV to not
cause noise when the shell starts (which is effectively all it did).

On a parse error (for any of those vars) we also use "" as the result,
which will be a null prompt, and avoid attempting to open any file for ENV.

This does not in any way change what happens for a correctly parsed command
substitution (either when it is executed when permitted for one of the
prompts, or when it is not (which is always for ENV)) and commands run
from those can still produce error output (but shell errors remain suppressed).

show more ...

lexical analysis fixes. This fixes the tests just committed in
src/tests/bin/sh/t_here.sh

The "magicq" magic was all wrong - it cannot be simply a parameter
to readtoken1() as its value needs to a

lexical analysis fixes. This fixes the tests just committed in
src/tests/bin/sh/t_here.sh

The "magicq" magic was all wrong - it cannot be simply a parameter
to readtoken1() as its value needs to alter during that routine
(eg: when magicq is set - processing here doc text, or whatever)
and we encountered ${var%pattern} "magicq" needs to be off for
"pattern" - and it wasn't.

To handle this magicq needs to be included in the token stack struct,
and simply init'd from the arg to readtoken1 (which we rename).
Then it can be manipulated as required.

Once we no longer have that problem, some other issues can be cleaned
up as well (some of this unbelievably fragile code was attempting to
cope with this in various ad-hoc - and mostly broken - ways).

Also, remove the magicq parameter from parsebackq() - it was not
used (at all) and should never be, a command substitution, wherever
it appears, always starts a new parsing context. How that applies
to old style command substitutions is less clear, but until we see
some real examples where we're not doing the right thing (slightly
less likely now than before ... nothing has changed here in the
way command substitutions are parsed, but quoting in general is
slightly better) I don't plan on worrying about it.

There are a couple of other minor cleanups, which make no actual
difference (like adding () around the use of the parameter in the
RETURN macro ... which is generally better, but makes no difference
here as the param is always a simple constant.

All the current ATF tests pass.

show more ...

NFCI - DEBUG mode only change.

Add tracing of lexical analyser operations. This is deliberately
kept out of the normal "all on" set as it makes a *lot* of noise
when enabled (especially in verbose

NFCI - DEBUG mode only change.

Add tracing of lexical analyser operations. This is deliberately
kept out of the normal "all on" set as it makes a *lot* of noise
when enabled (especially in verbose mode) - but when needed, it
helps (evidence for which is coming soon).

As usual, no doc, you need the sources (and of course, a specially
built sh to even be able to enable it.)

show more ...

Fix an amazing crazy botch (of mine) when expanding prompt strings
(PS1 etc) which, if the shell were already exiting, and a prompt
were to be expanded (which only really happens if -x is enabled,
an

Fix an amazing crazy botch (of mine) when expanding prompt strings
(PS1 etc) which, if the shell were already exiting, and a prompt
were to be expanded (which only really happens if -x is enabled,
and an exit trap is set, so the commands in the trap need PS4
expanded and written, last thing, before the shell exits) the shell
would instead simply exit when it finished expanding PS4 (before
even writing it, or the xtrace output).

There were more conditions required to set up the environment for
this to actually occur (it seems to only happen when the exit trap
is set in a function, called in a command substitution) but that's
unimportant, the code was nonsense.

Problem noticed by Martijn Dekker.

XXX pullup -8

show more ...

pgetc_linecont() needs to use pgetc() rather than pgetc_macro()
so the fake char returned by the latter when an alias ends (which
is there so we can correctly avoid alias recursion) is correctly
igno

pgetc_linecont() needs to use pgetc() rather than pgetc_macro()
so the fake char returned by the latter when an alias ends (which
is there so we can correctly avoid alias recursion) is correctly
ignored where it is not wanted.

show more ...

A similar fix to that added in 1.169 of eval.c, but here for when
processing command substitutions. If there is an error while processing,
the any pending queued input should be discarded. From F

A similar fix to that added in 1.169 of eval.c, but here for when
processing command substitutions. If there is an error while processing,
the any pending queued input should be discarded. From FreeBSD.

show more ...