[Monotone-devel] Getting all certificates efficiently

[Wim Oudshoorn]

For my emacs monotone mode I want to fetch all certificates
of all revisions in a somewhat efficient manner.  

With monotone version 0.23 I used

     monotone db execute "SELECT id,name,value FROM revision_certs"

and this returned all the certificates. the format 

    revision_id | name | base64-encoded-value

Now this is not perfect, but it is fast.  With a hot cache
the command line above (with output redirected to /dev/null) 
takes less than a second.

Unfortunately in monotone 0.26 the output format is changed to

    revision_id | name | value (not encoded at all)

This makes parsing really hard, if the value contains a newline,
which often happens for changelog certs, or a pipe symbol
you can't reliable tell where one certificate stops and the next
one starts.  

I expect people to immediately suggest to use 

  monotone automate stdio           

and feed it all the revisions one by one with lines like

l5:certs40:fff5dc58f79e696858a39004c8d125a35e182e6ae

and parse the output.  I have two problems with that,
one is laziness, the other performance.

* Laziness
The stanza output format is perfect,  no problem with parsing at all,
except for the fact that stdio adds text like:
       0:0:l:556:
which I need to filter out.  No big deal, I said I am lazy.

* Performance
this is a bigger issue, for the monotone db I do:

     time mtn automate stdio < /tmp/revisions  > /dev/null

     real    0m30.459s
     user    0m16.359s
     sys     0m2.566s

This is with a hot cache.  The file /tmp/revisions contains 
all revision of the monotone database in the form:

    l5:certs40:fff5dc58f79e696858a39004c8d125a35e182e6ae

Now of course, I do postprocessing and this is done in emacs
so this adds some time.  But do the same experiment with 0.23 
on my big database yields:

   time monotone db execute ....
   real 0.7sec

With postprocessing in emacs the time is about 2 a 3 seconds.

Now 
    time monotone automate stdio < /tmp/ap-rev > /dev/null

    real    0m14.581s
    user    0m6.931s
    sys     0m0.630s
   
So with automate stdio it will take at least 6 times as long :-(

So any ideas how to efficiently get all the certificates in monotone
post 0.26?

Wim Oudshoorn.