monotone-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Monotone-devel] Re: ideas to speed up monotone SQLite database access


From: Graydon Hoare
Subject: [Monotone-devel] Re: ideas to speed up monotone SQLite database access
Date: Mon, 06 Feb 2006 10:38:10 -0800
User-agent: Thunderbird 1.5 (X11/20051201)

Joe Wilson wrote:
I noticed a recent post on the SQLite mailing list
(http://www.mail-archive.com/sqlite-users%40sqlite.org/msg12839.html) asking 
how monotone might
speed up its database access. Two simple optimizations that come to mind are:

(1) Use a larger default SQLite page size and/or cache size:

 http://www.sqlite.org/pragma.html

SQLite's default page size is 1024 bytes, which may not be optimal for 
monotone's typical file
deltas and usage pattern. For instance, OpenEmbedded's monotone database is 
reduced from 80 megs
to around 62 megs just by using a larger page size. A smaller database 
footprint often translates
into better speed.

Huh. Weird. I did actually play with page-size adjustments quite a bit in older (sqlite 2) code. Then with 3, the sqlite documentation suggested that this twiddling was obsolete, and sqlite would automatically pick appropriate page sizes dynamically.

If that's not true, I guess it's worth revisiting. I had thought it a solved issue years ago.

(2) Use a ":memory:" database for the initial monotone pull and then dump the 
resultant database
to file. This would eliminate the disk latency entirely. Since most monotone 
databases I've seen
are generally under 100 megs, this is a viable option. Naturally, the 
file-based database must be
locked during the entire pull/dump operation. Unfortunately, this would prevent 
more than one user
from accessing the database at a time.

I'd prefer not to do this. We know there's a tradeoff between transaction granularity and performance, and we've set that granularity explicitly: if you look in database.cc, you'll see our transaction guards support explicit batching of subtransactions.

During netsync, we batch at a granularity of 100 packets. So for example, if there's a crash you lose everything since the last 100-packet boundary. But this can be adjusted; directing to a :memory: database would make netsync an all-or-nothing operation, and I'd prefer to keep it under our control.

-graydon





reply via email to

[Prev in Thread] Current Thread [Next in Thread]