[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Monotone-devel] merged: sqlite 3
From: |
graydon hoare |
Subject: |
[Monotone-devel] merged: sqlite 3 |
Date: |
Mon, 10 Jan 2005 03:51:51 -0500 |
User-agent: |
Mozilla Thunderbird 1.0 (X11/20041206) |
hi,
I've merged in christof's sqlite 3 branch. thanks christof, this is very
helpful! (sorry I haven't merged the ssh one yet, but it seems to be
using stuff which I'm not sure will work on win32 just yet). anyways, I
made the following small changes to your work:
- I added a function which detects the sqlite3 file format and
complains if you're trying to use a non-sqlite3 file, so that it
is a little more obvious what to do to migrate.
- I adjusted the default and maximum page sizes. these are only
advisory in sqlite 3; the same binary can read and write databases
with any such settings, since the page size is variable at the
database level. I checked, it really does work.
this is really great to have working. in particular, it means we are in
a much better state as far as size limits. as a test, I loaded a 107mb
mp3 file into a database. it's slow and eats a lot of memory (> 500mb)
but it works. the resulting database isn't even that big (144mb). the
only remaining hard size limit is on the database file as a whole (2TB),
the rest is just us being sloppy with memory use.
I've synced this change up to the net.venge.monotone head, so 0.17 will
have it. you'll have to dump and reload if you update to it; the 2.x
file format is now officially deprecated for monotone :)
now there are a bunch of follow-on tasks someone can do if they want to
have a go at further optimization:
- make a cache of parsed queries in database.cc and have the query
execution reuse them, binding to parameters. this will mostly
eliminate the sql parsing cost (though not the logging cost).
- play with switching to blobs for files, or possibly for everything
blobby. we're under no *real* obligation to keep files in base64,
except that it makes them not screw up your xterm when you do
monotone db dump. maybe that's not worth the price (?)
- play with switching the file_data object to a lazy / streamy sort
of thing, so that it keeps most data out of memory and can handle
larger data better.
- as a less intrusive step, figure out where the worst number of
in-memory copies of a file are made, and see if we can get that
down a bit.
next up: the restrictions branch...
-graydon
- [Monotone-devel] merged: sqlite 3,
graydon hoare <=