With some "segment loss emulation" code I was able to reproduce the deadlock frequently.
My summary first:
It is not a high ACK loss problem. The congestion window test in tcp_output() fails, cause the unacked queue gets misordered in the situation I will further describe. This is a severe bug in my opinion and I have no idea how to solve it (despite I wrote a workaround, but that's not a properly styled solution). I could imagine that some ghost like troubles of other lwip users are caused by this bug too.
The "segment loss emulation"
To gulp only one segment is not sufficient to reproduce the problem frequently. Therefore I decided to gulp the first retransmission of every 10th segment too. And here we are:
packets_sent++;
if ((packets_sent % 10) == 0)
{
//we enter here on every 10th segment and gulp it
//(we omit to call ip_output()
//we keep the sequence number of the gulped segment
kept_seqno = seg->tcphdr->seqno;
}
else if (kept_seqno == seg->tcphdr->seqno)
{
//if the gulped segment gets retransmitted the first
//time we gulp it once again.
kept_seqno = 0;
}
else {
//in every other case we do normal output
ip_output(seg->p, &(pcb->local_ip), &(pcb->remote_ip), pcb->ttl, pcb->tos,
IP_PROTO_TCP);
}
// ... Some statements ...
}
/* end of code snippet */
And this is whats happening on the "ether". In my queue representation I use the sequence number of the segments (not tcp_seg pointers). The sequence numbers are given from the last traces I made on our GPRS system.
1. Segment 8720:10085 was the last acknowledged segment from our gprs remote peer.
2. Segments 10085 to 14183 get enqueued by the local application the unsent queue is as follows:
unsent->10085->11453->12818->14183.
3. Segment 10085 should be the next "in-sequence" to be sent, hovever the gulp
mechanism of our local peer emulates a segment loss. The queues are as follows:
unsent->11453->12818->14183
unacked->10085.
4. Due to the available congestion window (cwnd) segment 11453 is sent (not gulped).
unsent->12818->14183
unacked->10085->11453
5. Due to the available congestion window (cwnd) segment 12818 is sent (not gulped).
Unsent->14183
unacked->10085->11453->12818
6. Due to the available congestion window (cwnd) segment 14183 is sent (not gulped).
Unsent->empty
unacked->10085->11453->12818->14183
7. Due to high round trip time in the gprs network we get the first dupack for
segment 10085 from our remote peer
8. We get the second dupack for 10085
9. We get the third dupack for 10085. According to RFC2581 we shall start a fast retransmission now
10. For fast retransmission tcp_process() calls tcp_receive() calls tcp_rexmit() calls tcp_output()
11. Cause tcp_output() was invoked by an initial tcp_input() it bailes out on
if (tcp_input_pcb == pcb)
==> !!! This violates RFC2581 IMHO !!!
12. But tcp_rexmit() already tinkered our queues by placing the first unacked segment to the
unsent queue.
unsent->10085
unacked->11453->12818->14183
13. The next few ouput attempts bail out in tcp_output() due to the nagle algorith
(tcp_do_output_nagle()). Thus nothing more hapens till a retransmission timeout occurs
14. tcp_slowtmr() requires a retransmission (pcb->rtime >= pcb->rto). This shrinks down the
congestion window to the maximum segment size (1390 in my case).
BTW: A retransmission is triggered by segment 14183 and not by 10085 in this case
which is an aftereffect of the underlying bug IMHO.
15. tcp_slowtmr() calls tcp_rexmit_rto(). The rto function moves all unacked segments to the head
of the unsent queue. This is final step causing the deadlock in tcp_output() cause the
smallest sequence number is now at the end of the queue.
Unsent->11453->12818->14183->10085.
16. tcp_ouput() is finally called. Instead of retransmitting 10085 it retransmits 11453 but fails
on (seg->tcphdr->seqno - pcb->lastack + seg->len > wnd) cause
seg->tcphdr->seqno = 11453
pcb->lastack = 10085
seg->len = 12818-11453 = 1365
wnd = 1390
11453-10085+1365=2733 which is greater than 1365 and therefor the test fails.
17. from now on we have a deadlock cause the queue stays misordered and tcp_output() always
fails on this test.
I needed a quick fix for our project and therefore I reordered the queue in tcp_output before the
While loop in tcp_output. However this is just a quick fix to fight the symptoms. Therefore I ask
for other suggestions or perhaps a patch.
Remark: Perhaps this situation is hard to reproduce. Without the "segment loss emulation" the
Deadlock only occured by using very paricular memory (pbuf etc.) configurations and by using the gprs
Network with its high round trip delays and relays.
/* begin of quick fix */
tcp_reorder_segments(&seg);
while ((seg != NULL) && (seg->tcphdr->seqno - pcb->lastack + seg->len > wnd))
{
//.... Some statements ...//
}