[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Ispell and unibyte characters
Re: Ispell and unibyte characters
Wed, 28 Mar 2012 21:18:21 +0200
On Mon, Mar 26, 2012 at 04:08:06PM -0400, Eli Zaretskii wrote:
> > Date: Mon, 26 Mar 2012 19:39:12 +0200
> > From: Agustin Martin <address@hidden>
> > Hi Eli,
> Thanks for responding, I was beginning to think that no one is
> interested. In general, I find that ispell.el is in sore need of
> modernization; at least that's my conclusion so far from playing with
> hunspell (with which I want to replace my aging collection of Ispell
> and its dictionaries that I use for many years).
> > At least for aspell ispell.el already uses utf8 as default communication
> > encoding and [:alpha:] as CASECHARS (and ^[:alpha:] as NOT-CASECHARS).
> > OTHERCHARS is guessed from aspell .dat file for given dictionary.
> The question is, why isn't this done for any modern speller. The only
> one I know of that cannot handle UTF-8 is Ispell.
I think the only real remaining reason is for XEmacs compatibility. AFAIK
XEmacs does not support [:alpha:].
I thought about filtering ispell-dictionary-base-alist when used from FSF
Emacs, so it uses [:alpha:] and still keeps compatibility. I am currently a
bit busy, but at some time I may try this for Debian and see what happens.
For XEMACS in Debian GNU/* even changing to [:alpha:] should have a reduced
impact, strings provided by dictionary maintainers take precedence, but
better if I can easily do the above anyway, so [:alpha:] is used if
Once release happens, I'd like to commit some other changes to decrease
XEmacs incompatibilities in ispell.el and flyspell.el, so my changes for
Debian GNU/* become smaller.
> OTHERCHARS are not very important anyway, at least for languages I'm
> interested in.
> > Since currently it is not possible to ask hunspell for installed
> > dictionaries (hunspell -D does not return control to the console)
> > no one tried something similar for hunspell.
> In what version do you have problems with -D?
Hunspell 1.3.2. Does not return control until I press ^C. This may be useful
if someone wants to know about installed hunspell dictionaries and prepare
something to play with that info, in a way similar to what is currently done
for aspell in ispell.el.
> In any case, hunspell supports multiple dictionaries in the same
> session. One can invoke it with, e.g., "-d en_US,de_DE,ru_RU,he_IL"
> and have it spell-check mixed text that uses all these languages in
> the same buffer (at least in theory; I didn't yet try that in my
> experiments). Clearly, this can only be done with UTF-8 or some such
> as the encoding.
> So I think we should deprecate usage of the unibyte characters in the
> ispell.el defaults, and simply use [:alpha:] for all languages. As a
> bonus, we can then get rid of the ridiculously long and hard to
> maintain customization of each new dictionary you add to your
> repertory. Just one entry will serve almost any language, or at least
> supply an excellent default.
> > > The only reason for this limitation I could find is in
> > > ispell-process-line, which assumes that the byte offsets returned by
> > > the speller can be used to compute character position of the
> > > misspelled word in the buffer. Are there any other places in
> > > ispell.el that assume unibyte characters?
> > Not sure if using utf8 and [:alpha:] has caused some problem for aspell,
> > I do not remember reports about this.
> Since I wrote that, I found that the problem was due to a bug in
> hunspell (which I fixed in my copy): it reported byte offsets of the
> misspelled words, rather than character offsets. After fixing that
> bug, there's no issue here anymore and nothing to fix in ispell.el.
> There's a bug report with a patch about that in the hunspell bug
> tracker, so there's reason to believe this bug will be fixed in a
> future release.
I filed that bug one year ago and received no reply from hunspell
maintainers. This year I received a followup with a proposed change, but
there is still no reply to it.
There is other problem that mostly hits re-using ispell default entries
[~ prefixed strings are treated as words in pipe mode]
that now stands for three years. I have waited in the hope this is fixed,
but I think I will soon commit to Emacs the same change I use for Debian,
making sure extended-character-mode is nil for hunspell. I do not think
extended-character-mode pseudo-charsets will ever be implemented in