Re: any plans for command substitution that preserves trailing newlines?

[Greg Wooledge]

On Tue, Jun 01, 2021 at 02:03:10PM +0200, Alex fxmbsw7 Ratchev wrote:
> when will u beginningly stop disrespecting the bash and co and stop giving
> misadvises like 'not possible in scripts' - 'dont use shell scripting' and
> 'bash is no codung language'

I'm not disrespecting bash.  But it has limitations, and is not suitable
for all tasks.

> sorry to say so but my full disrespect to not true anti shell \ bash claims

What have I said that isn't true?

Bash cannot store NUL bytes in shell variables.  This is a simple fact.
You can work around it, but then you're piling hacks upon hacks, and
eventually you should step back and realize that you're using the wrong
tool for the job.

> - the same as overinsulting ppl like me

I'm not insulting you.

> oh and yea not helping instead of helping

If I manage to convince someone that they're wasting their time doing
things in an inappropriate way, then I'm helping.

=====

Beyond that, I am forced to guess what *actual* problem the original 
poster is trying to solve, because (as usual) they aren't telling us.

In order to do that, I'm trying to walk backwards from the original
poster's question, and deduce what kind of problem could have led them
to this point.

Step 1: OP is concerned that var=$(prog; printf x) var=${var%x} may
fail with malformed input in some exotic character encoding.  Of course
we aren't told what "prog" is, or what "var" will be used for, or what
locale the OP was using when they encountered the problem, or what the
symptoms of the problem were....

Step 2: OP is trying to read the full output of "prog" including all
trailing newlines.  Is "prog" generating text?  Is it generating binary
data?  Is it generating a stream of filenames?

Step 3: OP is storing the full output of a program in a variable.  One
scalar variable.  So it can't be a delimited list, can it?  If it were
a delimited list, they'd be doing something different, like mapfile.  It
must be a scalar value.  Or maybe the OP just doesn't know how to write
shell scripts?  We can't tell.

Step 4: OP didn't simply use LC_ALL=C or some equivalent.  Why not?  Is
it not actually binary data?  Do they simply not know about LC_ALL=C?  Are
they moving data between different character encodings?  Do they even
know what character encodings *are*?  Or are they simply parroting some
crap they read on Stack Exchange that said "don't do this because it can
fail when 7 different stars align"?

And so on, and so on.

The more I ponder this, the more I think that there is *no* actual problem,
that this is all theoretical, that the OP simply stumbled across this
page on whatever Stack Overflow Exchange Forum, and decided that it was
a *crisis* and they should demand that an entirely new feature be added
to every shell to avoid it!  Starting with bash!

People lose sight of the fact that shells are not programming languages.
The command substitution syntax works the way it does because it's
designed to read *words* from *text files* and use them as values.  And
text files are defined to end with newline characters, which are not
part of the words that are being read.  So, the shell removes those
newlines for you.  Because that's what you want 99% of the time.

What about the other 1% of the time?  Well, that's a special case, and
you need to *describe* what you're doing if you expect anyone else to be
able to help you with it.

Remember, the shell "language" doesn't support lists or arrays, and uses
word splitting by default.  Why?  So that you can store a space-delimited
list of words in a scalar variable.  Why?  Because most of the time, the
words that you're storing in a shell script are filenames.  And filenames
in 1978 did not contain whitespace characters.  Why?  Because all of the
users of Unix at that time were programmers or scientists, not regular
people.