libmicrohttpd
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[libmicrohttpd] Re: libmicrohttp upload issue


From: John Muth
Subject: [libmicrohttpd] Re: libmicrohttp upload issue
Date: Fri, 15 May 2009 12:09:34 -0700

Hi Christian,

Yes that seems to help. I'm still seeing the reads shrink from ~5700 B down to a little less than 2K and then they go back to ~5700 but it is a lot better than it was before. Thanks.

I'm seeing a 20-1 performance difference between PUT and GET when transferring 8MB as a series of 64K requests. I'm thinking of playing with MHD_BUF_INC_SIZE to see what happen (for example, set it to 64k). Are there any gotcha's I should be aware of before I try it?

-John

On 5/15/09, Christian Grothoff <address@hidden> wrote:
Ah, I think I see what's going on.  MHD initially grabs 2k (out of a 1 MB
overall buffer) for read-IO.  It would grab more if necessary, but as it
happens 2k is just barely enough -- you have close to 2000 bytes of header,
which needs to be allocated for good, and then it uses the remaining few bytes
to incrementally process the PUT.

The required change is to make it grow the 2k buffer not only when it is down
to exactly zero bytes, but also possibly a bit earlier (so that a reasonable
number of bytes are transferred per system call), assuming that overall buffer
space is available (which at the default limit of 1 MB there obviously is).

I've changed the code in SVN HEAD to grow the buffer to allow reads of (at
least) 2K (unless the overall per-connection memory limit is reached). This is
now in SVN HEAD, revision 8461.  Please confirm that this fixes your problem.

Best,


Christian


On Friday 15 May 2009 10:12:49 am John Muth wrote:
> Hi Christian,
>
> I'm pretty sure what's happening is the read buffer keeps getting shrunk as
> a result of reading headers for each request. It is indeed the server
> issuing small reads to the socket.
>
> I added some messages inside libmicrohttpd to try to figure out what's
> going on. I added messages to every place where
> connection->read_buffer_size is changed. What I found is the read buffer
> shrinks for every header that gets read in. For example, from a message at
> the end of
> connection.c:get_next_header_line():
>
> --- 111 set read_buffer_size=2048
> --- recv: req len: 2048 got:111 bufsz:2048 off:0
> --- 333 set read_buffer_size=1974 GET
> /stor/stat/vol=1/fsid=1/fnum=2/chunk=3/version=4/ticket=TBD HTTP/1.1
> --- 333 set read_buffer_size=1952 Host: localhost:9000
> --- 333 set read_buffer_size=1939 Accept: */*
> --- 333 set read_buffer_size=1937
> --- 555 pool_reset size=1937
> --- recv: req len: 1937 got:179 bufsz:1937 off:0
> --- 333 set read_buffer_size=1840 PUT
> /stor/write/offset=0/length=65536/vol=1/fsid=1/fnum=2/chunk=3/version=4/tic
>ket=TBD HTTP/1.1
> --- 333 set read_buffer_size=1818 Host: localhost:9000
> --- 333 set read_buffer_size=1805 Accept: */*
> --- 333 set read_buffer_size=1782 Content-Length: 65536
> --- 333 set read_buffer_size=1760 Expect: 100-continue
> --- 333 set read_buffer_size=1758
> --- recv: req len: 1758 got:1758 bufsz:1758 off:0
> --- recv: req len: 1758 got:1758 bufsz:1758 off:0
>
> Later I see:
> --- 555 pool_reset size=1758
> --- recv: req len: 1758 got:183 bufsz:1758 off:0
> --- 333 set read_buffer_size=1657 PUT
> /stor/write/offset=65536/length=65536/vol=1/fsid=1/fnum=2/chunk=3/version=4
>/ticket=TBD HTTP/1.1
> --- 333 set read_buffer_size=1635 Host: localhost:9000
> --- 333 set read_buffer_size=1622 Accept: */*
> --- 333 set read_buffer_size=1599 Content-Length: 65536
> --- 333 set read_buffer_size=1577 Expect: 100-continue
> --- 333 set read_buffer_size=1575
> --- recv: req len: 1575 got:1575 bufsz:1575 off:0
> --- recv: req len: 1575 got:1575 bufsz:1575 off:0
> --- recv: req len: 1575 got:1575 bufsz:1575 off:0
>
> After that I see:
>
> --- 555 pool_reset size=1575
> --- recv: req len: 1575 got:184 bufsz:1575 off:0
> --- 333 set read_buffer_size=1473 PUT
> /stor/write/offset=131072/length=65536/vol=1/fsid=1/fnum=2/chunk=3/version=
>4/ticket=TBD HTTP/1.1
> --- 333 set read_buffer_size=1451 Host: localhost:9000
> --- 333 set read_buffer_size=1438 Accept: */*
> --- 333 set read_buffer_size=1415 Content-Length: 65536
> --- 333 set read_buffer_size=1393 Expect: 100-continue
> --- 333 set read_buffer_size=1391
> --- recv: req len: 1391 got:1391 bufsz:1391 off:0
> --- recv: req len: 1391 got:1391 bufsz:1391 off:0
>
>
> And so on until libmicrohttpd is issuing very small reads to to socket.
>
> Is there anything I can do to help you with this? I'd really like to use
> libmicrohttpd for this project.
>
> -John
>
> On 5/15/09, Christian Grothoff <address@hidden> wrote:
> > Simple answer: no ;-).  I'd try strace first to see if either the
> > *client* is
> > sending in strange increments or if the server is reading in small
> > increments
> > (which it should not be).
> >
> >
> > Christian
> >
> > On Thursday 14 May 2009 11:54:10 pm John Muth wrote:
> > > Hi,
> > >
> > > I'm doing a project with libmicrohttp 0.4.1 and I'm seeing some weird
> > > behavior when I upload data (I'm using libcurl 7.19.4 to drive the
> > > client side to issue a PUT and push the data up). I'm doing 64K
> > > transfers.
> > >
> > > On the server side, I'm seeing my access handler called with upload
> > > data but in very small increments. For example, the first transfer will
> > > be a series of 1758 byte uploads. The next transfer will be a series of
> > > 1575 bytes transfers. The 3rd will be 1391 byte transfers. And so on
> > > until I
> >
> > see
> >
> > > a series of 2 byte transfers.
> > >
> > > Have you seen anything like this before?
> > >
> > > -John Muth
> > > Parascale




reply via email to

[Prev in Thread] Current Thread [Next in Thread]