Re: [Chicken-users] Enterprise and scalability

[Alaric Snell-Pym]

On 19 Jan 2009, at 9:59 pm, Jörg F. Wittenberger wrote:

> Roll out is currently stalled (hence we do not _yet_ _deploy_ in a
> enterprise environment, but hope to do so RSN) -- actually because  
> we're
> busy elsewhere here and I'm sort of waiting for Felix to include those
> scheduler modifications I came up with -- wich incidentally address  
> some
> of those "scalability" issues.  See
> http://lists.gnu.org/archive/html/chicken-users/2008-12/msg00122.html
> and related threads from the last 3 month.

Yes! I've been impressed with Jorg's work there, even though it won't  
directly affect me since I'm not doing anything highly multithreaded.

> There are a few more minor issues related to higher load, which I  
> shall
> post about (the one I currently have in my mind: beware of garbage
> collection between fork and exec if finalizers might come into play,
> since those might fail in the child [[for instance because they access
> file descriptors close just before the fork]] and hence the gc will  
> loop
> forever) as time permits.

That is an interesting one! I can see a few solutions.

1) Force a gc before the fork - but that blocks up the parent process  
unnecessarily, and is only a stochastic fix; there's no real guarantee  
you might not have a small nursery and end up doing another gc in the  
child anyway.

2) Do a combined fork-and-exec, which is fine if there's no pre-exec  
setup you want to do in the child (setuids, environment variables,  
that sort of thing). Although perhaps many cases can be covered by a  
suitable broad "spawn" function that has keyword arguments to do all  
of the common things, and is itself a native C function so immune to GCs

3) Have the finalizers written to deal with this situation, ranging  
from checking the pid before cleaning up resources known to be pid- 
local to some way of, as part of the fork, filter out finalisers that  
shouldn't happen in the child.

> > > You wanna utilize more cores and more cpus at your setup, how would
> > > you go about this?
> >
> > Well, just run more chicken processes. Design your app so that lots  
> > of
> > processes can cooperate.
>
> I can second that.  Really.

*high five*!

> As a rough estimation, based on the algorithm (byzantine agreement)
> implemented in askemos expect: 2*P+S where P is the "ping time at
> application level" - including overhead for SSL and friends or  
> whatever
> you settle on above the wire - and S is the time spent in local
> processing to commit the transaction to persistent storage.  The point
> is: P >> S (read: P much larger than S); let alone the variance of P.
> And *then* add clock deviation!  I fact we measure anything from
> responses at 10ms past the request - which is less then the ICMP ping
> time for the same pair of hosts - to 13 seconds with a "felt"  
> average of
> 250ms.  To sum it up: more cores will make your reads faster.  For  
> write
> performance tweak your whole setup instead, especially the network.

I need to read more about askemos - my day job is writing a replicated  
database (in C, alas). We have a local database on each node, which  
many processes can read but only one writes; the one that listens to  
multicasted updates and performs them locally. Client processes read  
directly from the on-disk database (via a shared-memory cache,  
although I'm not sure how much difference that makes compared to OS- 
level disk block buffering), and do writes by multicasting them to all  
nodes (including their own). A client process is welcome to be highly  
threaded within itself, if the application demands it, but it works  
just as well with lots of clients. And the facts that, for each node,  
writes all go through a single process means we can avoid most locking  
issues in one fell sweep - and there's no point in multithreading that  
process, since it's highly I/O bound, reading from a single multicast  
socket and writing to a single local database.

(We use spread to handle the reliable multicasting for us)

> /Jörg

ABS

--
Alaric Snell-Pym
Work: http://www.snell-systems.co.uk/
Play: http://www.snell-pym.org.uk/alaric/
Blog: http://www.snell-pym.org.uk/?author=4