Couple of days ago I submitted a question to Stackoverflow regarding an adequate way to handle many (thousands, that is) overlays. I wanted to know if there are any alternatives to using overlays (which is slow) for cases similar to mine. To make a long story short, I am building a generalized incremental tokenizer, details can be read on Stackoverflow (http://stackoverflow.com/questions/25915908/best-way-to-save-lots-of-position-pairs-in-emacs-lisp
It turns out that the only viable alternative to overlays is using text properties, and text properties were not meant for the purpose. All I need is an ability to save position pairs, the positions should survive text insertion/deletion and there should be a way to find those pairs given a buffer point. Overlay interface is perfect (consider overlays-at, overlays-in and the dynamic (re-)positioning system).
After answering my question Stefan (Monnier?) mentioned that overlay behavior can be fixed from N^2 to N*logN, which would make them asymptotically equivalent to text properties; adding that he has no time to implement the fix.
I believe that given some time, a few tips from experienced Emacs developers and enough coffee I can at least try to fulfil the task. I am not in a hurry, and wanted to get into Emacs internals anyway.
What do you think?
PS I already have my Cedet/Emacs FSF papers signed - sent a minor patch to Cedet some time ago.