Re: OMG - performance issue with Scheme engraver

[Urs Liska]

Am 10.07.2018 um 16:56 schrieb Urs Liska:

Am 10.07.2018 um 16:48 schrieb David Kastrup:

Urs Liska <address@hidden> writes:

Hi all,

after completing most of the work of reviewing the scholarly.annotate
module I realize that it (presumably one small change) is a total
performance killer, and I need some help tracking it down.

Well, one thing just jumping out at me is

      ((process-acknowledged translator)
       (for-each
        (lambda (grob)
[...]
                     ;; reset list to prevent multiple processing.
                     ;
                     ; TODO: I don't understand why I can kill *all* the list
                     ; after having processed *one* grob.
                     ; What happens to any other annotated grobs (at the same time)?
                     ; I know it is possible to annotate multiple post-events, for example.
                     (set! all-grobs '())))))))
        all-grobs))

That's just garbage.  If you want to stop processing the list in spite
of having started it with for-each, you need a non-local jump
(catch/throw).

The list processed by for-each has no connection at all with what you
store in all-grobs inside of the loop.  And it does not appear like you
even bother resetting all-grobs at all when you don't reach the
conditional passage inside, letting it grow at will.

... which is basically what I arrived at with more investigation in my other post.

I was aware that this "set!" doesn't break the for-each, but I thought I should do let *this* loop run through to the end because in the *current* list there might be other grobs I need to process.

What I missed was that the list wasn't emptied when *no* grob matched the condition. So ironically the list grew longer with less annotations present.

What seems to fix it (at least with the one test I could do before leaving) is setting the list to an empty list in start-translation-timestep.

Still I don't feel confident about what exactly happens in what step of the process ...
Let's see how that latest change plays out.

OK, now I repeated the tests with the counters after resetting the list in start-translation-timestep:

For a short 1-page score:

acknowledgers:            980
process-acknowledged:     711
lambda in process-ack: 
old:                   330421 ratio: 1 / 464 (ratio between calls to process-acknowledged to the inner for-each loop)
new:                     4404 ratio: 1 / 6,19
 

For the mentioned 3-page score:

acknowledgers:            5037
process-acknowledged:     2700
lambda in process-ack: 
old:                   6726982 ratio: 1 / 2.491
new:                     18678 ratio: 1 / 6,91

For the 21-page three-movement score

acknowledgers:             21518      30695     46492
process-acknowledged:      17656      25520     35183
lambda in process-ack:
old:                   187253429  363211651 779357617    ratios: 1 / 10.605, 1 / 14.232, 1 / 22.151
new:                      125061     172616    240311    ratios: 1 / 7,08    1 / 6,76    1 / 6,83

The 21-page document that took 14 seconds without the engraver and 12 minutes with this morning's engraver now takes 15 seconds (although I'd have to do a better sequence to average the results).

So it seems I have a working solution - but I would like to take the opportunity to learn a little more.

Obviously for each call to process-acknowledged there is an average of ~ 6-7 grobs in the all-grobs list which seems consistent throughout the three different scores. But what grobs *are* they? When I create a list of grobs in the acknowledgers clause, which grobs end up there? is this all grobs that happen to be created at this timestep in this context?

I have printed all-grobs for the first ten occurences and this very much looks like it:

2 grobs in list
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
2 grobs in list
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
9 grobs in list
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
10 grobs in list
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
10 grobs in list
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
11 grobs in list
#<Grob VerticalAxisGroup >
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
11 grobs in list
#<Grob VerticalAxisGroup >
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
17 grobs in list
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob VerticalAxisGroup >
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
18 grobs in list
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob VerticalAxisGroup >
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >
18 grobs in list
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob VerticalAxisGroup >
#<Grob NoteColumn >
#<Grob NoteSpacing >
#<Grob Flag >
#<Grob StemStub >
#<Grob Stem >
#<Grob Script >
#<Grob NoteHead >
#<Grob NoteHead >
#<Grob MultiMeasureRestNumber >
#<Grob MultiMeasureRest >

This seems to confirm that I should *not* break out of the for-each loop that iterates over all-grobs in process-acknowledged.

Does that mean that resetting the all-grobs list in start-translation-timestep is the right approach?

Thanks Urs

Thanks
Urs

_______________________________________________
lilypond-user mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/lilypond-user