Re: dired-do-find-regexp failure with latin-1 encoding

[Dmitry Gutov]

On 29.11.2020 17:06, Eli Zaretskii wrote:

> > Do we want to search the "binary" files at all?
>
> We don't.  I still hope to understand why -a was needed in this case.
> Stephen?

Looks like it actually depends on the encoding of the _output_. So if it 
can print some lines well but not others it can even print a line from a 
file and then later say it's a binary:

$ grep "prem" latin1.txt
premie?re is slightly different
Binary file latin1.txt matches

Adding -a or prepending 'LC_ALL=C' changes that:
$ LC_ALL=C grep "prem" latin1.txt
premi�re is first
premie?re is slightly different

So... looks like Grep searches through all files anyway. Just modifies 
its output in cases where it looks iffy.

> > > We should support Grep regardless, since not everyone will have
> > > ripgrep.  And in any case, "C-x RET c" will be needed with it as well,
> > > no?
> >
> > I'd have to test it explicitly to say for sure, but:
> >
> >     ripgrep supports searching files in text encodings other than UTF-8,
> >     such as UTF-16, latin-1, GBK, EUC-JP, Shift_JIS and more. (Some
> >     support for automatically detecting UTF-16 is provided. Other text
> >     encodings must be specifically specified with the -E/--encoding flag.)
> >
> > https://blog.burntsushi.net/ripgrep/#pitch
>
> What is not clear to me is whether the _output_ is always in some
> fixed encoding, like UTF-8.  That doesn't seem to be stated in the
> docs there.

Judging by a small experiment, rg's output is in the same encoding as 
input, for each file. Which can be a nuisance when looking at the search 
results, but that's probably all.

In any case, if one takes the pre-processing route, the end encoding 
will be UTF-8.