Re: Eshell's external pipe module interferes with other argument parsing hooks

[Jim Porter]

On 3/31/2022 2:56 PM, Sean Whitton wrote:
> On Thu 31 Mar 2022 at 01:58PM -07, Jim Porter wrote:
>
> > Another possibility would be to keep the current behavior (or close to
> > it), but to reconstruct the command to pass to `sh' during Eshell's
> > rewrite phase. I'm not quite sure if that would actually work, but if it
> > did, it would allow other argument parsers to run normally without
> > extpipe needing to know what parsers to try. Perhaps if we kept around
> > the substring that each argument parser consumed, it would be possible
> > to reconstruct the relevant bits for extpipe's purposes?
>
> Well, in your case (2), you don't want the other parsers to get a chance
> to run -- that's the whole point.

In practice, yes. However, the implementation could allow the other 
parsers to run, but then discard their parsed results during the rewrite 
phase if applicable. This is probably subject to a different set of 
unusual corner cases, but would allow other parsers to consume the parts 
of the command that they care about so that extpipe can just look for 
`*|' and friends. (A `*|' sequence inside quotes or something similar 
would already have been consumed by the time extpipe sees it.)

> > More generally though, maybe there are really two different use cases?
> >
> > 1) Eshell's built-in pipelines are slow because they go through Emacs
> > buffers.
> >
> > 2) It would be convenient to invoke a whole command (or some large part
> > of a command) using `sh' syntax.
>
> These are both things that extpipe is meant to make easy, though I'm not
> sure how separate they are -- often I want both.
>
> > For (1), Eshell could opportunistically use external pipelines without
> > any special syntax.
[snip]
> This could just be added to Eshell right now, right?  Definitely useful.

Unless there's a reason for Eshell's current behavior that I'm not aware 
of, I can't think of any problems with doing this, so long as everything 
is escaped properly.

> > For (2), we'd need a convenient syntax for forwarding some command
> > string to `sh'. Something like your proposed !! or || syntax, or maybe
> > something to wrap around part of a command?
>
> Yeah, extpipe's syntax covers most such cases but not quite all of them.

For the purposes of parsing, having the token that activates the extpipe 
module be at the beginning of the relevant portion would make things a 
lot easier. Then `eshell-parse-external-pipeline' can just check if 
that's the next token and if so, read until the end of the extpipe 
portion. That would eliminate all the complexity of trying to identify 
unquoted/literal `*|' operators.

In practice though, I'm happy with any syntax so long as the 
implementation is robust. If the current implementation using `*|' 
operators is significantly nicer to use (I don't have an opinion either 
way since I haven't used it enough), then we should stick with it, even 
if it makes it harder to implement.

> No problem, but could I request that you spend a little more time
> editing your messages for length?  And perhaps consider separating out
> discussion of significant future possible enhancements from fixing bugs
> with the existing code into separate bugs or ML threads, as I've done
> with this message.  Thanks in advance :)

I'll try. :) I usually prefer shorter messages myself, but I've had a 
hard time finding the right balance on the Emacs lists. Sometimes I 
think I'm keeping things focused, only to find that I haven't relayed 
enough information, and instead have just confused matters. Splitting 
these threads off is probably the right call though. Thanks.