Re: Dealing with thread storms

[olafBuddenhagen]

Hi,

On Wed, Apr 07, 2010 at 12:37:59PM +0200, Sergio Lopez wrote:

>   - Implement a queue in libdiskfs to deal with m_o_data_return
>   requests and a static amount of consumers which will do the real
>   work. Each time the kernel sends us a m_o_data_return, we just add
>   the request to the queue and return the thread to the pool.

I think this is a very good idea. This kind of piplined processing is
actually the best approach when optimizing for high throughput.

(Some time ago I stumbled over some research paper, where they
implemeted an experimental HTTP server with such a design, and claimed
considerably higher performance than the established servers -- despite
being implemented in Java...)

While for most Hurd servers such an implementation probably would be too
complex (and not really necessary anyways) -- a continuation-based
approach seems more realistic for the generic case (we talked about this
in the last discussion about thread storms, when the thread throtteling
patch was dropped) -- I think it makes perfect sense to use the
pipelined approach for more critical pieces like the pager.

-antrik-