[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Announcement stis-engine

From: Stefan Israelsson Tampe
Subject: Re: Announcement stis-engine
Date: Thu, 19 Aug 2021 01:37:02 +0200

HI all,

I added a new gem, it' a c based serializer and deserializer of guile data which is relative feature rich and will ping pong 
between scheme and c-land in order to be able to be fiber friendly. So currently the benchmarks are for the following copy

,time (load (dump (make-bytevector N 1))
1s for ~ 1GB (test of memcpy)

,time (load (dump (make-vector N 1))
1s for ~ 100MB (will be less pipelined)

,time (load (dump (iota N))
1s for ~ 50MB (will create in total N cons cells with 5 bytes)

The scheme based version of these serializers and deserializers was able to perform 5MB per second, but thats with much higher allocation overhead

This means that we match pythons dump and load functions. I will add those to my fibers piping tool

Currently supported data structures
all numbers yes even bignums
all immediates  ('(), booleans characters and whatnot)
strings (wide strings and narrow strings)

the location of the c-code is in 

(use-modules (persist persistance)) => atom-load-bv and atom-dump-bv is defined.

This library is used by python-on-guile and I will replace the current dump and load with this in the end. Also this tool
will be added to the fiber piping system I am working with in stis-engine.

On Sat, Aug 14, 2021 at 2:07 PM Stefan Israelsson Tampe <> wrote:
Next is to add support for
1. self referential data structures like #1 = '(1 2 3 #1), string compression
2. Structs
3 .Goops Classes and Objects

Then I will add
1. Lambdas
and then Continuations to be able to serialize (if you turn off unboxing optimizationsas in guile)

After that a toolbox to customize tserializing just as with python

Note all this is supported in python-on-guile, just that the new code is better suited for proper streaming 
and the code is much nicer in my personal view so you do have all these features already if you use pythons
dumps and loads commands together with string->fpipe and fpipe->string.  

The new thing is essentially stream compression and streams to zmq's. Stream compression wll come to 
python as well as the zlib python module is very rudementary

reply via email to

[Prev in Thread] Current Thread [Next in Thread]