Re: [Txr-users] TXR, similar tools

[Russell Adams]

On Mon, Apr 25, 2011 at 05:32:27PM -0700, Kaz Kylheku wrote:
>
> On Mon, 25 Apr 2011 08:27:50 -0500, Russell Adams
> <address@hidden> wrote:
> > Having reviewed TXR, I adore what I see!
> >
> > I must ask, are there similar tools (ie: language libraries for Perl
> > or Scheme) that inspired TXR?
>
> The inspiration is shell "here docs" and
> pattern matching / unification. Also, the patch utility
> (patch performs line-based pattern matching on context
> lines to find where a hunk applies). I don't use Scheme or Perl,
> but I have a lot of experience with Common Lisp. Internally in Txr,
> symbols and lists work more like Common Lisp than like Scheme.
> (Lists are terminated by the nil symbol, and can be improper).

I learned CL to basic proficiency, and then switched wholesale to
Racket Scheme. No more packaging or library issues, and most of the
power. Both have made me a better programmer. ;]


> The basic idea was that: whatever we can synthesize by substitution
> into templates, we ought to be able to analyze likewise,
> in any language.

Exactly! I've explored many toolkits that purport to do this and only
work on a small scale. Perl's regexp named capture comes close, but
TXR looks much better.

> For instance, in C, we have the crude printf to format text, and a
> roughly analogous scanf to pull apart formatted text. If you
> squint your eyes, they are similar. E.g. print two numbers
> separated by a space:  "%d %d", x, y .   Scan two numbers
> separated by whitespace "%d %d", &x, &y.
>
> In Lisp we can use unification libraries to match between
> nested list structures, and use backquote to generate new list
> structures.
>
> It's irksome how in shell programming easily we can generate
> entire text files with here docs, but we can't reverse the
> effect in the same way when we need to.
>
> I developed early versions of Txr while working on a project
> that involved applying patches to an entire installed system.
> However, the patches had to be rolled out to installations
> which might contain variations: for instance, differences
> in an /etc/fstab file, and the like.
>

I'm generally working with records from many different vendors where I
need to get the data back into tables from output formats I don't
control. Think the occasional application, UNIX shell, hex / debugger
dump. I've been really burned on using SQL, my new toy is regexp
capture to Mongo where I can store data as hashes where I don't have
to manually vet every potential column.

FYI, TXR + Mongo would be a killer combo!

> I needed to generate patch files from templates, substituting pieces
> of text into the actual diff context lines, and the - and + lines.
>
> I used here docs to generate the patches, but I found it irksome
> to extract the pieces of info to interpolate, even though it closely
> matched the shape of the patches being generated. I could imagine:
> what if we could pretend we are applying this patch, find where
> in the file it matches, and then actually pull out the nonmatching
> pieces of text to update the patch so that it does actually apply.
>
> I.e. I essentially wanted a robot that could fix up patches
> based on looking at a base file that is somewhat different
> from the original.
>

Puppet. ;]

> So I started working on Txr; and early versions were put to
> immediate use on that project.
>
> > Text extraction is a frequent chore of mine, and I'll have to see what
> > I can apply TXR to!
>
> If it gets mind-bending, post here.

I think my worst problem is changing formats. It always seems that
formats of the same records change slightly over time, incrementally
with few large breaks. Making parsers to compensate for every variant
is such a chore, while still obtaining the data I need.

I was asking about libraries, because I'm not typically working in the
shell to parse data. Perl's good, I wonder if FFI would help.

Thanks.


------------------------------------------------------------------
Russell Adams                            address@hidden

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3