[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Emacs completion matches selection UI

From: Dmitry Gutov
Subject: Re: Emacs completion matches selection UI
Date: Wed, 08 Jan 2014 03:23:21 +0400
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0

On 07.01.2014 07:32, Toby Cubitt wrote:
1. `prefix-replaced' and `common-substring' both seem to be tailored to
the dynamic interface.

Not at all. `prefix-replaced' tells you that the buffer substring that is
being completed has already been replaced with new text in the
buffer. This is important for non-prefix completion (e.g. pattern

I'm not sure if it's generally applicable. AFAIK, neither Company nor CAPF keep this kind of history: if the buffer was modified, what happens after is a new completion, or no completion at all (if circumstances told us we should abort).

Any UI that modifies the buffer text will almost certainly need to set

I don't really see why a non-prefix completion UI (or any other one) has to modify the buffer text. Company has an inline "preview" frontend, but it uses an overlay, and so the buffer text remains unmodified.

And the core Completion-UI code that ties all the UIs
together makes heavy use of the `prefix-replaced' property. So even in
Completion-UI it's not only used by the dynamic interface.

Could you explain how?

Similarly, `common-substring' demarcates the longest common substring if
completion text has been inserted in the buffer, which is information a
third-party UIs might very well want to make use of.

Why? How?

The properties documented in the `completion-ui-register-interface'
docstring are those that have a standard meaning in Completion-UI. Note
that nothing prevents a particular UI widget from storing its own data in
its own widget-specific overlay properties if so desired.


So how would one use an UI defined as such, in a different package?
Would Company define itself as a new source? Or add a source per each

Depends on what the generic Emacs API for specifying completion sources
ends up looking like.

Hmm, I was expecting something easier, since IIRC you said Company should have been able to use it without major changes.

If we end up using c-a-p-f, `completion-source'
will need to store something that identifies the c-a-p-f entry that
returned the completion candidates for this completion process.

Probably just the value returned by the successful completion function: start, end, collection (aka completion table, which is often a function).

The best comparison one can make is that
company-backends loosely corresponds to a mixture of
completion-ui-source-definitions and auto-completion-source-functions.

The latter: yes, the former: no. It's never used the same way as you use

I said "loosely corresponds", and I stand by that. One of the things a
`company-backends' entry does is to define when a given completion source
should be used. In Completion-UI, that part of company-backends' role is
performed by auto-completion-source-functions.

Yes, that's why I agreed with this comparison (and also made it earlier, IIRC). I disagree with the other comparison.

Could well be. But you seem to be arguing both ways. You also argue that
the c-a-p-f API is bad because it's opaque and hard for users to
configure. If backend definitions and selection logic are always supplied
by package authors, then the fact that c-a-p-f is opaque isn't so

The opaqueness in c-a-p-f is bad because the exact values of `company-backends' and `completion-at-point-functions' are significant. A third-party package author can only push a function in either of these lists, but they can be responsible for the eventual order of the elements.

And the order matters, because it influences which completion backend will get picked when several are suitable. When we get to the grouped backends, which I've mentioned several times, which Company supports, and CAPF will hopefully support in the future, being able to understand and maybe change the values of either list becomes even more important.

Please take a look at `company-etags' and `company-dabbrev' and see if
you can point out the situations when the user might find the `prefix'
code of either too limiting.

dabbrev and etags are both sources that, if they're useful in a given
buffer, they're useful in the entire buffer.

Consider the LaTeX math mode source, and the LaTeX environment name

My question was, if you could point out problems with any of our bundled backends (or, failing that, third-party ones). If they look okay, maybe

source, and the LaTeX preamble source, etc. Trying to code the selection
logic for all of these using the Company API looks very awkward to me,
compared to the simplicity of setting a few buffer-local variables in

If performance is the problem, we could solve that by either:

1) Adding some pre-completion hook which would allow you to run some code once, set a buffer-local variable, which all backend functions for LaTeX can refer to later in `prefix' action.

2) Create a special "merged" backend that would collect the results from all LaTeX backend in a carefully defined fashion.

3) Indeed add some hook analogous to `auto-completion-source-functions'. But yeah, CAPF already does that.

Sure, I could move that logic into Predictive itself, and have a single
Predictive LaTeX backend. But that serves to demonstrate that the API
isn't flexible enough to let me do what I want easily. Other markup
languages and programming languages make similar demands on the API.

We bundle several backends, some of them for programming modes, and so far you haven't pointed out specific problems with any of them.

Or you can define new backends that would do some common checks in
`prefix' (maybe calling an extracted function with common code) and
simply delegate all other actions to the respective base backend.
Implementing this is trivial.

And once you've finished doing this, and factored out common selection
mechanisms like regexps, faces and text properties into utilities

Regexps we have already (company-grab-...), instead of faces one should be using `syntax-ppss', font-lock isn't always available, and text properties... hmm.

functions...you'll have reimplemented something closer the Completion-UI
API or c-a-p-f :)

Only if we add a similar hook, see 3) above.

I understand the principle, really. But the more one "cleanly separates"
code, the harder it can be sometimes to read it, to get the full picture.

Indeed, which is why I listed grouping the completion and selection logic
into one place as one of the things I liked about Company's API.

Perhaps the cleanest and most flexible solution would be to have a list
with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.

I believe this suffers from the LaTeX problem you've described above. If you have a dozen of completion functions for LaTeX, this scheme expects you to have a dozen of corresponding test functions, and when one fails, the next will be called, and it won't be able to use the results of the previous call (unless they set and use some common buffer-local variable, which Company backends could also do; but that's ugly).

> And then
supply a bunch of standard utility functions for use as TEST-FUNCTION's,
for testing regexp matches, faces, text-properties, etc.

Like I described previously, such stand-alone tests probably won't be very useful as values of this alist. One usually has to call several of them to see if a completion-function is suitable.

It makes certain amount of sense, although it looks like it could make
creating a "merged" completion function more difficult.

I doubt it'll be insurmountable. Also, merged completion functions are a
rather advanced feature that may not belong in core Emacs anyway (though
it would be good if the API supported them without ugly hacks).

It's a good feature enabling some kinds of backends that aren't usually useful on their own (like Yasnippet). In my book, that's a good argument to accept or reject an API.

We'll see.

Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
and opaque c-a-p-f API replaced with something cleaner and simpler, I
don't see us winning that argument.

Personally, I'd probably be fine with c-a-p-f as long as it's powerful enough. Moving to a less featureful API is likely out of the question, but if it's demonstrated that c-a-p-f is fairly unsuitable for implementing some features, I believe it would be a good reason to rule it out.

But the general approach, while flexible on its surface, complicates
things if I intend to use any existing sources, written by third
parties. Because their authors are unlikely to have anticipated the
logic I'll add in my custom predicate function and to have written any
code in their packages I might use. Or, at least, that's considerably
less likely.

I don't get your argument here. You have to wrap the third-party
completion function in Company in order to code the appropriate backend
selection logic. In Completion-UI, you put that code in an
`auto-completion-source-function' instead, and probably don't need to

Which code? There's likely to be none.

Hence, more effort required on my part.

Here I'm describing an organizational problem caused by an API. Not a technical one.

Like mentioned above, delegating the search for completions to an
existing backend is trivial. These are functions, and as such they are
stateless. Just call (other-backend 'candidates current-prefix).

But this results is convoluted and somewhat confusing code. If the
selection logic and completion function were separate, you wouldn't need
to use such hacks.

Looks straightforward to me. Function delegation is a rather simple concept.

We have examples of that in the Company frontends: take a look at `company-pseudo-tooltip-unless-just-one-frontend' and `company-preview-if-just-one-frontend'.

`company-dabbrev-code' also delegates to `company-dabbrev', but it just uses a public function from that backend's package, which is also a valid approach.

It understand why you're harping on about the limitations of the
default hook entries

I just can't see how you find them useful on their own. I'll shut up now. :)

Indeed, though this is starting to look a lot like defining some standard
hook functions for auto-completion-source-functions. As you make the
company-backends API more flexible and convenient, you'll increasingly
find yourself reimplementing equivalent functionality to that of c-a-p-f
and the Completion-UI API. If were to start simplifying the Completion-UI
API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
the sweet spot is in the middle?

Maybe. I can certainly see myself adding a auto-completion-source-functions analog in Company, as an advanced feature.

Actually, I was thinking about the former option. Let's define widgets
with an API in the usual sense, so it can be used by both
`completion-at-point' and external packages. We'll need this kind of API
either way, in order to be able to write new widgets.

I'm confused. `completion-at-point' will never use an API for *defining*
new completion UI widgets (`completion-ui-register-interface'). It will
need to be modified to *invoke* the new UI widgets.

Here I'm thinking in terms of Company approach, I guess. The API for defining a backend or frontend and the way it is used are the same: a protocol consisting of messages and proper reactions or responses to them.

Thus, "widget has to respond to..." would be in the API for defining a widget, and "widgets respond to..." would be in the usage API. Essentially the same.

If you like to add convenience macros, extra infrastructure, etc, they may diverge, but we'll need an API a completion package can use easily use anyway, not just a bunch of private functions that can change will every release.

Copying and adapting some of the code from `complete-in-buffer' into
`completion-in-region' would suffice for that. Then
`completion-in-region' would remain the generic Emacs API for displaying
and selecting completion candidates (only now it would display them in a
nicer interface).

Maybe. At a first glance, `completion-in-region' will need access to PROPS returned by completion functions, not just COLLECTIONS.

Yes, but see above. Using 1. from Company would be the current next step
toward integration, as I see it.

Really? Why should we use the Company UI widget API and interfaces in
Emacs, when the API is less flexible and less customizable than that the
Completion-UI widgetAPI, and it implements fewer widgets?

Sorry for being unclear. I meant the reverse: Company would use the new widget API defined here, while still retaining its backends, at least initially.

Although while you don't own the popup widget, we do have one. Maybe porting that code won't be too hard.

Did you really mean that you wanted to throw Completion-UI in the
garbage, rewrite Company yourself to be suitable as a generic Emacs
completion API, rewrite/port the missing UIs and features, and persuade
people to merge it into Emacs core? (If so, great! Let me know when
you're done, and then I can stop maintaining Completion-UI and switch
Predictive over to the new interface :-)

I'll think about it. :) Not sure about the "persuade people to merge in into Emacs core", though.

But you could help kick this process off my filing an issue describing Company's backend API shortcomings. Is it just the lack of `auto-completion-source-functions'? Non-prefix completion?

Probably I misunderstood what you wrote.

Yep. :)

I only really have one strong requirement: if some form of
auto-completion mode gets included in the Emacs completion UI, I will
argue hard for it to be as powerful as Completion-UI's
`auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
and `auto-completion-override-syntax-alist', and at the way all the UI
widgets are integrated.) Anything less powerful, and it will be useless
to me for Predictive.

I'll take a look, thanks.

Customizing hooks is a tricky business. I believe the opaqueness of
c-a-p-f to the user is the main problem with the current API.

But above you argued that backends and selection logic are supplied and
configured by package authors, not by users!? In which case package
authors can simply supply a c-a-p-f function, and users can add the
functions they want to c-a-p-f. (I believe Stefan made a similar point

Customizing a hook is tricky for user. Try `M-x customize-variable RET find-file-hook'. You'll only see the buffer-local value, and not the global one. But `run-hooks' uses both if the local value includes `t', which it usually does.

Users can modify the hooks programmatically, of course, but that's a step more difficult. And they'll also need to understand the values already there, to be able to remove or rearrange them.

We'll also should discourage lambdas there. At the moment, my find-file-hook contains this beauty, courtesy of autorevert.el:

#[nil "\302\301!\210\303\304!8\211\207"
[buffer-file-name auto-revert-tail-pos make-local-variable 7 file-attributes]

We should have less of that.

Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you seem
to be arguing both ways.

Of course I'm arguing toward a middle ground comfortable to me. So: package authors deciding when their backend is suitable to use -- good; users deciding the order of trying backends and their groupings -- also good.

Swings and roundabouts. I could always add a
`completion-ui-major-mode-source' function that checks an alist, and then
it would be:

Yes, well, here you are discarding the "standard, tried-and-tested Emacs mechanisms" of major mode hooks and buffer-local values. Which was exactly my point.

- foo completion function
- bar completion function
- foo predicate function
- bar predicate function
- add two entries to the alist

You forgot adding `completion-ui-major-mode-source' to the relevant list.

- code 1st bundled completion function and selection logic
- code 2nd bundled completion function and selection logic (which
   duplicates the logic in the 1st one, with different parameters)
- code 8th bundled completion function and selection logic (which
   duplicates the logic in the last 7, with different parameters)

Could that be a one-line function call, in each case with different arguments? Aside from potential performance problems, it sounds rather neat.

- add 8 entries to company-backends
- arrange for package-specific sources to be added to company-backends

Contrast with Completion-UI's, which involves a *lot* less coding:

- register 1st source (no need to code a new completion function, just
   set one parameter when calling `completion-ui-register-source')

You can likewise delegate to the same completion function in Company backends. It's the same amount of code (1 line).

It would be almost
trivial to switch Completion-UI over to c-a-p-f. All I'd need to do to
replicate the existing functionality is add a few additional properties
to the PROPS return argument.

Great. I'd like to see the patch. :)

Presumably I could always set a buffer-local
`company-backends' from predictive-mode that only contains relevant
backends if necessary.

Yes, of course.

c-a-p-f is trivially customizable by users, as long as packages / Company
supplies suitable c-a-p-f functions for the backends.

When you say "trivially customizable", do you mean via the Customize interface?

*Writing* c-a-p-f
is non-trivial, but as you said earlier those are/should be supplied by
package authors. Or have I misunderstood?

I don't think I said that about c-a-p-f, but yes, they should be.

3. The most popular and useful "list-the-available-completions" UI is
     popup.el. Is there any chance of getting copyright assignment for
     this? Or will we be forced to code something equivalent from scratch?


Fingers crossed...!

I wouldn't hold my breath: IME, getting a response from that crowd on non-trivial issues is hard. Collecting copyright assignments could be harder still.

P.S. My email client displays each new message from you as a separate thread, possibly because your email address is timestamped, or maybe because they don't contain a "References" header. Could you do something about that?

reply via email to

[Prev in Thread] Current Thread [Next in Thread]