Re: [Bug-apl] ISO Component File API preview

[Elias Mårtenson]

Would you be willing to move the CF stuff on top of GDBM? It's a much better fit for it, and will give you everything you do with SQLite today, without the extra overhead (SQLite is also not very efficient if you open multiple databases).

Regards,

Elias

On 11 July 2014 12:21, David B. Lamkins <address@hidden> wrote:

On Thu, 2014-07-10 at 18:52 -0500, Blake McBride wrote:

>
>
> The component file system is for APL-only applications.  The databse
> API is to communicate with the rest of the world.  It depends on what
> you need.
>

and...

>
> The problem is that you are creating a component file system that
> _depends_ on features of SQLite and is _needlessly_ dependent on those
> features.  I understand it works on other databases, but it is an
> unreasonable burden on those systems.  You could just as easily create
> the same system that doesn't depend on particulars of SQLite.  That
> would work everywhere in a way that is reasonable to any of the
> systems.
>

Ahhh... OK. Perhaps I've had an "aha!" moment. :)

If I understand you correctly, generalizing the current component file
implementation to work with databases other than SQLite is problematic
for all of the reasons you've discussed: poor generality, poor use of
resources, and - in general - a poor fit to SQL.

Therefore, here's my proposal:

I'll drop all further work on generalizing the current component file
API to work with SQL databases. IOW, the component file system will
remain - literally - a component *file* system. The quick patch that I
worked up last night will never become part of my code.

Programmers who want a way to persist data for an APL-only single-user
application and a shallow learning curve and no sysadmin or dbadmin
concerns will (I hope) appreciate and use the simple component-file API.
Programmers who need the functionality of a real database - and all that
it offers in terms of access control, concurrency and remote access -
can use lib_sql with a PostgreSQL backend.

The presence of SQLite will remain an incidental implementation detail
of the component file library. All further concerns specific to the use
of SQLite as an implementation vehicle will be treated as out-of-scope
for the project.

The current component file code can serve as a reference implementation
to validate the design choices not implicitly required by the ISO APL
Annex. There are a number of such choices that I was forced to make in
the absence of detailed guidance from the spec. For example: the choice
to ensure that component IDs assigned by CF_APPEND are always
nondecreasing for a given component file. There are many other detailed
choices that had to be made. If you have a concern with how I've
implemented the API to meet the spec, we can work out the details
together.

Meanwhile, if someone feels that it's important to layer a component
file API on top of lib_sql *and* allow a choice of database backends,
they may either start from scratch or fork the existing code.
Personally, I don't see the need for this particular layering of
abstractions; for anything more complex than an integer-indexed file of
APL objects, I'd prefer to work directly with lib_sql. But if someone
else sees value in the layered approach, by all means pursue it as a
separate project.

Meanwhile, I'll continue working on design sketches for a component file
system that doesn't build upon SQLite. The primary reason for doing so
will be to increase performance while not requiring use of explicit
transactions to reduce single-record insert times below the 100ms mark.
Of course the content of the component file will need to remain reliable
in the face of software and hardware failures; that's one of the
implementation attributes that I got "for free" by choosing SQLite as a
short-term implementation tactic. An alternative implementation will be
challenging, no doubt, but worth the effort in the long run. Whether
such an implementation should be done as a native library or as a hybrid
is not clear to me at this time.

The take-away from all of this should be the following: if you use the
existing component file library (and I certainly hope that you will do
so in cases where it best serves your needs), you can be assured that
any future work that I do with the component file library will preserve
your data. Any change in the file format will be accompanied by a
utility workspace to migrate your existing component files.

>
> This is the most I've been involved with a community project.  Fun!
>

Indeed. :)

Also, it's worth once again acknowledging Jüergen's work in making all
of this possible. All of these little libraries and utilities are
penny-ante stuff compared to the core interpreter. I just want to say,
again, that I am grateful for the ability to work with a full-featured
open-source APL interpreter for the first time ever. If you look at the
history of free APL systems, they've traditionally been thin on the
ground, undermaintained and not very interesting.

A lot of technologies have developed or matured in the past 30 years to
get to the point where we are today with free APL: Unicode, Linux,
Emacs, gcc, fast hires video and the x86 hardware platform being
foremost among a plethora of candidates. None of us would be here on
this mailing list today, however, had Jüergen not put in an immense
amount of design and implementation effort to create GNU APL. I
sincerely appreciate his continuing support and involvement.