ido-switch-buffer is slow with many buffers; others are fast

[catern]

I recently became annoyed enough in my 1000-buffer Emacs instance to
start profiling things.

I was disappointed to find that the buffer-switching slowness I'd seen
is the fault of ido-switch-buffer.

ido-switch-buffer takes over a second before it starts accepting user
input, with this many buffers.

It appears that the slowness is completely due to ido building a list of
all buffers in ido-make-buffer-list-1. This function calls
ido-ignore-item-p on every buffer, which checks buffer names against a
regex to ignore ones with leading spaces, which seems to be fairly slow.

I tested a few other packages to see how they compared.

- switch-to-buffer, without modification: This is fast. Compared to ido,
  I believe this is because switch-to-buffer ultimately uses read-buffer
  and internal-complete-buffer to read the buffer, which are implemented
  in C and presumably have been well-optimized. ido doesn't use those.

- switch-to-buffer with icomplete-mode: This is also fast, I suppose
  because it also uses read-buffer and internal-complete-buffer, and
  just customizes completion, unlike ido-mode.

- switch-to-buffer with fido-mode: This is also fast, again for the same
  reasons as icomplete-mode.

- iswitchb-mode: This is fairly fast - faster than ido-mode. But, it
  still has a noticeable delay. Like ido, it doesn't use read-buffer,
  and it builds a list of all buffers. Relative to ido, it seems to have
  a more efficient (and less generic) implementation of ignoring
  irrelevant buffers, which explains its speed-up. This is a bit
  unfortunate since iswitchb-mode has been deprecated in favor of
  ido-mode.

I also tried some third-party packages.

- helm: Slow. Looks like it builds a list of all buffers. (Note that
  helm is not in GNU ELPA, but is quite popular.)

- ivy: Fast. It uses read-buffer and only customizes completion.
  Notably, ivy seems to have many, if not a superset, of the features of
  ido-mode. (ivy is in GNU ELPA)

What is the solution? It seems to me that there are two candidates:

1. Given that ivy has a similar feature set to ido, and uses
   read-buffer, and is fast, it seems that it should be possible to make
   ido use read-buffer without loss of functionality.

2. ido should be deprecated or discouraged in favor of ivy, possibly as
   part of including ivy into core Emacs.

Discovering this inefficiency of ido (which probably applies to other
ido functions, since they have similar implementations) is fairly
disappointing to me, because it's frequently recommended to new
users.

There have been many proposals to turn ido on by default in Emacs.
Alas, given the current state of ido's performance, it seems to me to be
clearly unacceptable to turn ido on by default.