bug#48734: 28.0.50; Performance regression in `string-width`?

[Imran Khan]

A package I use (deft-mode) has been hanging for minutes with high cpu
use recently. Profiler says most time is spent in `string-width`, and
upon looking it seems to happen in files that have multibyte characters
in them.

I reproduced the problem by creating a file that has both single and
multi byte characters:

with open("/tmp/test", "w") as f:
    for i in range(50_000):
        print("1", file=f, end="")
    print("α", file=f, end="")

And now:

(benchmark-run 1
  (let ((str))
    (with-temp-buffer
      (insert-file-contents-literally "/tmp/test")
      (setq str (buffer-string)))
    (string-width str)))

This takes 20 seconds in my machine (if string is exclusively full of
either single or multibyte characters, weirdly it seems to finish
instantly).

--------

In GNU Emacs 28.0.50 (build 1, x86_64-unknown-linux-gnu, X toolkit, cairo 
version 1.16.0)
 of 2021-05-29 built on 9d4a5f294007
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: void

Configured using:
 'configure --with-x-toolkit=athena --without-toolkit-scroll-bars
 --without-ns --with-xft --without-dbus --without-gconf
 --without-gsettings --with-zlib --prefix=/usr --sysconfdir=/etc
 --sbindir=/usr/bin --bindir=/usr/bin --mandir=/usr/share/man
 --infodir=/usr/share/info --localstatedir=/var
 --host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu
 '--libdir=${exec_prefix}/lib64' --with-file-notification=inotify
 --with-modules --with-native-compilation --with-jpeg --with-tiff
 --with-gif --with-png --with-xpm --with-rsvg --with-imagemagick
 --with-json --with-xml2 --with-gnutls --with-sound --with-m17n-flt
 --with-harfbuzz --with-cairo --with-lcms --without-gmp 'CFLAGS=-fno-PIE
 -mtune=generic -O2 -pipe ' 'CPPFLAGS= ' 'LDFLAGS=-no-pie
 -Wl,--as-needed ''

Configured features:
ACL CAIRO FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ IMAGEMAGICK JPEG JSON
LCMS2 LIBOTF LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PNG RSVG SECCOMP SOUND THREADS TIFF X11 XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $LC_ALL: en_GB.UTF-8
  value of $LC_COLLATE: en_GB.UTF-8
  value of $LC_CTYPE: en_GB.UTF-8
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Memory information:
((conses 16 677156 287115)
 (symbols 48 39801 7)
 (strings 32 181641 67878)
 (string-bytes 1 6097769)
 (vectors 16 66337)
 (vector-slots 8 1256518 570753)
 (floats 8 909 1141)
 (intervals 56 1260 531)
 (buffers 992 22))