Programmed completions are utterly confusing

[Ag]

Before someone tells me that asking for help should be posted in
help-gnu-emacs@gnu.org, I apologize for the noise, but I'm hoping maybe
this could open a discussion for documentation improvements for
completions. In my opinion, documentation around the subject is very
terse and is lacking examples.

I've been trying to understand how completions work, and I needed a
completion that searches not only within the displayed rows but also in
annotations. Someone suggested concatenating the annotation info into
the main string, but I don't like this for a few reasons. I've even been
told that the annotations are not designed for that, and if I want them
to be searched upon, they really can't be annotations.

- Use case I have right now: I want to search through URLs and their
  page titles. If I put each url into the main display string, and
  annotate them with titles, every row would be recognized as a "proper
  url", and packages like Embark could use that, e.g., I can dispatch
  embark-url-action and browse them. But if I concatenate the url and
  the page title, I would have to use tricks so Embark recognizes it as
  a url (and not a url with a string attached to it)

- Another use case comes to mind, let's say I want to sift through some
  kind of log entries. Imagine I would want to provide a feature, where
  if typed something like "last 5 mins", it would limit the rows to
  include only log events with timestamps no older than five minutes
  ago. Obviously, there's no practical reason or a sensible way to
  encode dynamic time value into each row,

So I dug a bit further and found:
https://www.gnu.org/software/emacs/manual/html_node/elisp/Programmed-Completion.html

I attempted to try it with something like this:

```lisp
(defun dogs-filter (seq)
  (lambda (str pred flag)
    (pcase flag
      ('metadata
       (list 'metadata
             (cons 'annotation-function
                   (lambda (c)
                     (format "\n\t%s" (alist-get c seq nil nil #'string=))))))
      ('t
       (if (string-blank-p str)
           (all-completions str seq)
         (all-completions
          str
          (lambda (s _ _)
            (seq-map
             #'car
             (seq-filter
              (lambda (x)
                (unless (string-blank-p str)
                  (or
                   (s-contains-p str (car x) :ignore-case)
                   (s-contains-p str (cdr x) :ignore-case))))
              seq)))))))))

(let* ((coll '(("Affenpinscher" .  "Loyal and amusing")
               ("Akita" . "Ancient Japanese")
               ("Bulldog" . "Kind but courageous")
               ("Caucasian Shepherd" . "Serious guarding breed")
               ("Miniature Schnauzer" . "Long-lived and low-shedding"))))

  (completing-read
   "Select a breed: "
   (dogs-filter coll)))
```

It looks like it works, but I don't think it really does. First of all,
I have to use that weird "if" check, otherwise it won't display things
when the completion string is empty. Also, it doesn't work like, for
example, Consult does, which allows you to type multiple words and
search for both occurrences. In the example above, `str` never gets set
to be multiple words, always uses only the first one that's typed and
ignores everything after the first whitespace.

Also, I figured out how to attach metadata, but couldn't wrap my head
around other flags, I don't know how `nil`, `lambda` and `boundaries`
work.

I understand some of the metadata properties: `category`,
`annotation-function`, `affixation-function`, `group-function`. I know
how `display-sort-function` works, but have no clue what
`cycle-sort-function` is for and how it can be used.

I don't understand all these `completion-table-dynamic`,
`...-table-with-cache`, `...-with-predicate`, `...-case-fold`, etc.

Can someone please help me learn all this? And can you please tell me
what's wrong with the snippet?