Re: [lwip-users] losing input packets and repeated acks

[Carl D. Blake]

Thank you for your help.  I've listed the things I've tried below.

On Fri, 2007-01-12 at 19:53, Jonathan Larmour wrote:
> Carl D. Blake wrote:
> > 
> > I've attached my lwipopts.h file and a tcpdump trace.
> 
> I just noticed a few things in your config too...
> 
> > /* #define TCP_TMR_INTERVAL 250 */
> > #define TCP_TMR_INTERVAL    10
> 
> That is very very small. You will probably be too busy processing the TCP 
> timer to do any real work. The default should be adequate.
> 
I set it back to 250.  I noticed that I was getting some 250 ms delays
between packets, so I reduced this to try and improve throughput.

> > #define PBUF_POLL_SIZE              30
> 
> This should be PBUF_POOL_SIZE, not POLL. You will therefore have 16 
> buffers by default instead.

Oops.  I changed the name to PBUF_POOL_SIZE and set it to 90.
> 
> > #define PBUF_POOL_BUFSIZE   1536
> 
> Are you sure you mean 1536 and not 1500?
> 

Per your suggestion below.  I changed this to 500.  I was trying to set
it to the maximum size of my ethernet packet aligned to 8 bytes.

> > #define PBUF_LINK_HLEN              16
> 
> Are you sure you mean 16 and not 14?
> 

I definitely mean 16 here.  I tried 14, but some combination of lwip,
the GNU C compiler V3.4.4, MEM_ALIGNMENT set to 4, and my processor
(MPC8271) makes the whole thing not work if HLEN is 14.  16 seems to
work.

> > #define TCP_MSS                     1476
> 
> Again I would have expected this to be 1460 (just like your peer in the 
> trace below). Fortunately for you, your peer won't go above 1460, so you 
> won't see any problem, but this value still seems wrong, unless there's 
> something unconventional about your setup.
> 
I changed this to 1460.  I saw 1476 in some lwip example somewhere.

> > #define TCP_WND                     (16 * 1024)
> 
> 16K, but you only have 16 pbufs, so it will be easy to run out of pbufs. 
> You will use 1 pbuf even if you only receive a tiny packet, and that 
> includes acks. It does dependent somewhat on your data pattern, but it is 
> usually more efficient to have a larger number of smaller pbufs. It's 
> usually better to have 3 500-byte pbufs than 1 1500-byte pbuf. Remember 
> that pbufs can be chained.
> 
OK.  I've changed my setup to have 90 pbufs.  Each one is 500 bytes.

> > ------------------------------------------------------------------------
> > 
> > 13:53:58.646727 boromir.tucson.boeckeler.com.10001 > 10.0.1.222.4098: P 
> > 904882:904904(22) ack 1 win 5840 (DF)
> > 13:53:58.649988 boromir.tucson.boeckeler.com.10001 > 10.0.1.222.4098: . 
> > 904904:906364(1460) ack 1 win 5840 (DF)
> > 13:53:58.651482 boromir.tucson.boeckeler.com.10001 > 10.0.1.222.4098: . 
> > 906364:907824(1460) ack 1 win 5840 (DF)
> > 13:53:58.654904 10.0.1.222.4098 > boromir.tucson.boeckeler.com.10001: . ack 
> > 906364 win 11094 (DF)
> > 13:53:58.654912 boromir.tucson.boeckeler.com.10001 > 10.0.1.222.4098: P 
> > 907824:909284(1460) ack 1 win 5840 (DF)
> > 13:53:58.654915 boromir.tucson.boeckeler.com.10001 > 10.0.1.222.4098: P 
> > 909284:909926(642) ack 1 win 5840 (DF)
> > 13:53:58.655868 boromir.tucson.boeckeler.com.10001 > 10.0.1.222.4098: . 
> > 909926:911386(1460) ack 1 win 5840 (DF)
> > 13:53:58.661166 10.0.1.222.4098 > boromir.tucson.boeckeler.com.10001: . ack 
> > 906364 win 12576 (DF)
> 
> Nevermind the huge delay later, you are getting problems right here. You 
> will probably find you are out of pbufs. You are advertising a larger 
> window than you have memory to accept, so the remote size is still 
> flinging data at you. If you allow out of sequence packets, this may also 
> explain the large delays, as they will use up pbufs waiting for any 
> missing data, but there would be no pbufs left to allow the missing data 
> to come in!
> 
I made the changes you suggested and things are a little better.  I'm
still getting the huge delay periodically, but the multiple acks of the
same packet and the same window size are gone.  I still get multiple
acks of the same packet, but the system seems to be advertising a larger
window size for each ack.  I seem to be getting some corrupt data, but
it's not nearly as bad as it was before.

I enabled LWIP_STATS and printed out the stats before and after the
transfer.  TCP stats is reporting several checksum errors.  PBUF stats
is reporting several errors.  I enabled debug messages for pbufs and I'm
getting the message "pbuf_alloc: Out of pbufs in pool."  This happens
quite a bit during the transfer of 1 Mbyte.  I hadn't expected this to
be a problem since my pbuf memory is 45Kbytes and my TCP window is
16Kbytes.  Any other suggestions?

> Jifl