Re: -s flag (soelim) not working?

[Bjarni Ingi Gislason]

On Mon, Sep 19, 2022 at 01:38:35PM -0400, Peter Schaffter wrote:
> I recently discovered that documents with .so'd files containing
> non-ASCII input require preprocessing with soelim(1).  If we have
> 
> file: source
> .pl 1v
> .ds FOO àéîöù
> 
> file: main
> .so source
> .nop \*[FOO]
> 
> running
>   groff -Tutf8 -k main
> spits out garbage, whereas
>   soelim main | groff -Tutf8 -k
> correctly renders string FOO.
> 
> soelim(1) doesn't mention utf8 input, merely
>   "It is useful if files included with .so need to be preprocessed."
> 
> Since it is non-evident that utf8 input (as opposed to inputting
> named glyphs) in sourced files needs to be preprocessed (i.e.
> with soelim), saying something about it in the manpage would be
> helpful.  I'm fairly certain most users, on reading "...need to be
> preprocessed" think tbl, pic, eqn, grap, chem..., not utf8 input.
> 
> Now, the '-s' flag issue...  Unless I am misunderstanding or
> misreading something,
>   soelim main | groff -Tutf8 -k
> and
>   groff -Tutf8 -ks main
> should produce identical output, however the -s flag is ignored and
> groff spits out garbage.  What's up?
> 
> -- 
> Peter Schaffter
> https://www.schaffter.ca
> 

  This is bug #59442 (groff.cpp: move soelim before preconv in
constructed pipeline), which Ingo Swartze and Brandon have kept
unsolved, changed the "Category" (see History at the bottom of the bug
web pages), and thus forced others to rediscover the bug (not a feature)
and thus waste peoples time.

  And postpone to make current (at each time) state public.

  This is almost a two years old bug and to inhibit bug fixing,
publication and thus testing time is a purpose.

  What is the meaning of a "Wish" Category?

  Every bug report is a wish! (for attention?).

  People can add this simple patch and start using it (they do not need
a permission, neither from Ingo Swartze nor from G. Branden Robinson).

  How do you test something that is not there?

  Why is there no test to check the correct working of the current state
of pipelines with a ".so <file with utf-8 characters>" line in an input
file?

  What has the behaviour of Ingo Swartze cost you (single and plural)?
and what saved you (also single and plural)?

 In the attachments are files "latin1.utf8" and "latin1" which can
serve as test files. 

-----

commit 32d30bef7168bbc4f080cb24e0dfb8e78308884c
Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
Date:   Tue Nov 10 14:12:05 2020 +0000

    Subject: groff.cpp: correct the order of preprocessors in the pipeline
    
    src/roff/groff/groff.cpp: correct the order of preprocessors in the
    pipeline.
    
      The "soelim" command must be the first in the pipeline to incorporate
    all sourced files into the input file,
    before it is sent to the next command.
    
      Change the order from "preconv, soelim, refer" to "soelim, preconv, 
refer".
    
    Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>

diff --git a/src/roff/groff/groff.cpp b/src/roff/groff/groff.cpp
index f568ac029..cd61dbd07 100644
--- a/src/roff/groff/groff.cpp
+++ b/src/roff/groff/groff.cpp
@@ -53,9 +53,11 @@ extern "C" {
 
 // grap, chem, and ideal must come before pic;
 // tbl must come before eqn
-const int PRECONV_INDEX = 0;
-const int SOELIM_INDEX = PRECONV_INDEX + 1;
-const int REFER_INDEX = SOELIM_INDEX + 1;
+// soelim must come first to complete the input file for the later
+// commands in the pipeline
+const int SOELIM_INDEX = 0;
+const int PRECONV_INDEX = SOELIM_INDEX +1;
+const int REFER_INDEX = PRECONV_INDEX + 1;
 const int GRAP_INDEX = REFER_INDEX + 1;
 const int CHEM_INDEX = GRAP_INDEX + 1;
 const int IDEAL_INDEX = CHEM_INDEX + 1;

----


"This second radical novelty shares the usual fate of all
radical novelties: it is denied, because its truth would be
too discomforting.
I have no idea what this specific denial and disbelief costs
the United States, but a million dollars a day seems a modest
guess."

Page xxix in:

On the Cruelty of Really Teaching Computing Science

Edsger W. Dykstra (Dijkstra)

SIGCSE Bulletin 1989, 21(1), Pages xxv-xxxix.
Also "www.cs.utexas.edu/users/EWD/"
#
  "Since breaking out of bad habits, rather than acquiring new
ones, is the toughest part of learning we must expect from that
system permanent mental damage for most students exposed to
it."

Page xxxvii in:

On the Cruelty of Really Teaching Computing Science

Edsger W. Dykstra (Dijkstra)

SIGCSE Bulletin 1989, 21(1), Pages xxv-xxxix.
Also "www.cs.utexas.edu/users/EWD/"
#
  "The problems of the real world are primarily those you are
left with when you refuse to apply their effective solutions."

Page xxxviii í:

On the Cruelty of Really Teaching Computing Science

Edsger W. Dykstra (Dijkstra)

SIGCSE Bulletin 1989, 21(1), Pages xxv-xxxix.
Also "www.cs.utexas.edu/users/EWD/"
#

latin1
Description: Text document

latin1.utf8
Description: Text document