|
From: | Graydon Hoare |
Subject: | [Monotone-devel] Re: ideas to speed up monotone SQLite database access |
Date: | Mon, 06 Feb 2006 10:38:10 -0800 |
User-agent: | Thunderbird 1.5 (X11/20051201) |
Joe Wilson wrote:
I noticed a recent post on the SQLite mailing list (http://www.mail-archive.com/sqlite-users%40sqlite.org/msg12839.html) asking how monotone might speed up its database access. Two simple optimizations that come to mind are: (1) Use a larger default SQLite page size and/or cache size: http://www.sqlite.org/pragma.html SQLite's default page size is 1024 bytes, which may not be optimal for monotone's typical file deltas and usage pattern. For instance, OpenEmbedded's monotone database is reduced from 80 megs to around 62 megs just by using a larger page size. A smaller database footprint often translates into better speed.
Huh. Weird. I did actually play with page-size adjustments quite a bit in older (sqlite 2) code. Then with 3, the sqlite documentation suggested that this twiddling was obsolete, and sqlite would automatically pick appropriate page sizes dynamically.
If that's not true, I guess it's worth revisiting. I had thought it a solved issue years ago.
(2) Use a ":memory:" database for the initial monotone pull and then dump the resultant database to file. This would eliminate the disk latency entirely. Since most monotone databases I've seen are generally under 100 megs, this is a viable option. Naturally, the file-based database must be locked during the entire pull/dump operation. Unfortunately, this would prevent more than one user from accessing the database at a time.
I'd prefer not to do this. We know there's a tradeoff between transaction granularity and performance, and we've set that granularity explicitly: if you look in database.cc, you'll see our transaction guards support explicit batching of subtransactions.
During netsync, we batch at a granularity of 100 packets. So for example, if there's a crash you lose everything since the last 100-packet boundary. But this can be adjusted; directing to a :memory: database would make netsync an all-or-nothing operation, and I'd prefer to keep it under our control.
-graydon
[Prev in Thread] | Current Thread | [Next in Thread] |