[lwip-users] lwIP Ethernet (TCP and UDP) communication slows down and acting weird

[Márk Bartha]

Hi all,

 

I'm developing a MicroBlaze application for Nexys™4 Artix-7 FPGA Board using lwIP 1.4.1 RAW API for Ethernet communication. This is about sending some data from PC to the FPGA that will do sg. and sends back the same amount of data to the PC using TCP communication. Everything is working fine, for example sending 16000 bytes from PC to FPGA and receiving it takes 450 ms, it's fine. And then I am doing a stress test, and repeating this TCP communication continuously will slow down but only after some time (ie. one communication time will increase to 4500 ms) and after some time it will suddenly slow down to 9000 ms and later to 35000 ms etc. I really took a lot of time finding out why this is happening but there are some weird stuff you need to know to answer my question:

 

Why is this happening and what suggestions do you have on fixing it?

 

- It's not deterministic when it will slow down. Sometimes it happens after the 6th communication but sometimes it won't happen either after 5 minutes of testing.

 

- Everything will slow down: accepting TCP connection, receiving packet, sending packet etc. I can see retransmissions and dup ACKs everywhere in the Wireshark.

 

- This only happens if I am using the FPGA connected to the router. But no, it has no connection with the DHCP, if I use it peer-to-peer connected to my PC, I am also using DHCP server, and it won't slow down.

 

- I don't think that is about memory leak (or not in the application layer or any leak in the TCP communication). I tried to change the lwIP memory options, everything, but if it happend to run out of memory the FPGA freezed, not slowed down. Increasing memory-related values also didn't fix it.

 

- It was tested on other FPGA device (same family). Same result.

 

- I didn't see any special in the lwIP STATS, no errors, no drops at all. Also DEBUG was enabled, nothing special, but if you need some text, I can show you. Actually when TCP debug was enabled nothing happened (the communication was slower of course).

 

- And here is my favourite one: After days of hard debugging I still thought it was about sg. in connection with the TCP. Meanwhile in the slowdown it had a stable ping response time, and really short, 50 ms. But guess what happened after I stopped the slow TCP communication: the UDP slowed down and became instable. And vice versa: If the TCP communication slows down (to 4500 ms) It became faster (3000 ms) while pinging it in every seconds and will be 700 ms (!) if I send UDP broadcast messages to the router in every 5 ms!

 

Really thanks for any suggestions. If you need any more information I can provide stat results, wireshark capture or anything about the communication algorithm.

 

Thanks,

 

Mark Bartha