Re: [Qexo-general] qexo speed

[Per Bothner]

Craig Munday wrote:

> I was just wondering whether you had tried files in the order of 10Mb or
> larger (say 100mb).  We are expecting our application will need to deal with
> this size of file and would like to know what is possible.

No, I haven't tried running very large data myself.
While I'm Qexo sure will need to tuning to work well for large
files, the basic design has some advantages:
* The implementation is compiler-based, so it can do various
compile-time optimizations, including those that make use of
static typing.  (The latter is very limited now, but the hooks
are there.)
* The default data structure uses for documents and nodes
(gnu.kawa.xml.NodeTree) is very space-efficient.  It will probably
need some tweaking as we gain experience with larger data-sets,
but the basic data structure works very well.
* Most of the code uses abstract classes, so you could use
alternative data structures - including potentially file-based
data bases.  (Again, this is not going to work out of the box, but
the framework is there.)
* The compiler knows about streaming (sax-like) apis, and tries
to use those when it can. Again, this is a place we can improve
the compiler's intelligence as we get more application experience.

Bottom line:  I don't know how well Qexo will handle large files
right now, but the basic design is very flexible and efficient.
Right now I'm concentrating on correctness and completeness - and
actually this weekend's check-ins will almost certainly be a
performance regression - but I know how to fix that.
--
        --Per Bothner
address@hidden   http://per.bothner.com/