Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 01.10.2021 19:41, João Távora wrote:
> João Távora <joaotavora@gmail.com> writes:
>
> > João Távora <joaotavora@gmail.com> writes:
> >
> > > > (benchmark 1 '(dolist (dir load-path)
> > > >                   (when (file-exists-p dir)
> > > >                     (let ((files (directory-files dir t "\\.el\\'")))
> > > >                       (dolist (file files)
> > > >                         (unless (file-directory-p file)
> > > >                           (with-temp-buffer
> > > >                             (insert-file file)
> > > >                             (read-from-string (buffer-string)))))))))
> > > >
> > > > Reports 2.5 seconds here.
> >
> > > Are these Emacs's files?
> >
> > Nevermind, just saw it's load-path.  Which makes sense.
> >
> > So I have to ask: is this an 'Emacs -Q' load-path?
>
> Anyway, the good news is that:
>
>   * avoiding read-from-string, and using read isntead
>   * using a single buffer
>   * using insert-file-contents (insert-file is for interactive use)
>   * using the Emacs -Q load-path
>
> speeds your snippet down from 2 to 0.2 seconds.
>
> The bad news is that your snippet was only reading the first form in a
> file, so when you read the whole file it takes 2.8 seconds on my corei7
> machine to read the whole Emacs sources.

Right.

Note that the above code uses very little Lisp and mostly calls into C. 
Any kind of deeper analysis based on reading the forms would need to 
have more processing implemented in Lisp around it. Which is often the 
slow part.

> So performance wise, it's not bad but can probably be improved.  I don't think
> it's gonna be stupidly hard because:
>
> * this is one of those embarassingly parallel problems.  We could just spawn
>    spawn `invocation-name' to use SMP

If analysis is implemented in Lisp, the concurrency will be limited by 
that part. Though I guess you could launch multiple Emacs instances and 
then combine their results somehow.

> * designing a caching strategy seems fairly easy here.

Probably doable, yes, though both harder than the approach I described, 
and almost certainly slower even with cache (which needs regular 
invalidation). And the first search in the session (without cache to 
fall back on) will likely hurt the most.

Further, the code above only looks inside load-path one level deep. But 
we'd usually need to scan each directory recursively (there are packages 
like cedet with nested file structure).

Anyway, good luck. I just hope you don't end up with etags-like approach 
where the user will need to built and rebuild the index manually every 
time they want it to be fresh.