[lwip-users] Netconn API, e.g. lwip_close, netconn_close, or netconn

Hello,

Using lwIP on top of FreeRTOS, I have been having problems where the Netconn API

does not return to my application.

This happens mostly on netconn_close and netconn_delete.

To cause the problem to happen more frequently, my network is setup like this:

internet <-> AppleAirPortExpress <-> EthernetSwitch <-> lwipUnitUnderTest

WebPowerSwitch

A script controls the WebPowerSwitch and power cycles the

AppleAirportExpress on for two minutes and then off for two minutes.

During these power cycles the Ethernet Switch and the

lwIP Unit Under Test remain powered on, and maintains an Ethernet Link.

My lwipopts.h is appended at the end of this email. At the top level I have TCP_MSL

set to 4000 msec because the embedded real time nature of my application is not tolerant

of blocking on hung TCP connections for long periods of time.

My application uses the sockets API on top of the netconn API. On lwip_send I

set a SO_SNDTIMEO of 500 msec.

uint32_t timeout = 500;

socklen_t sizeOfTimeOut = sizeof(timeout);

int result = setsockopt (sd, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeOfTimeOut);

The application thread detects that the internet is down (powered off Apple AirPort) by getting

too many send timeouts, it then tries to close the socket, where I get this calling tree which never

returns: lwip_close->netconn_delete-> (mailbox to TCP/IP task) lwip_netconn_do_delconn->

lwip_netconn_do_close_internal

lwip_netconn_do_close_internal calls this path: lwip_close->tcp_close_shutdown->tcp_send_fin->

tcp_enqueue_flags

Which returns an error, and lwip_netconn_do_close_internal decides to let tcp_slowtmr call

poll_tcp to finish the close, which it never does.

I think my problems relate to https://savannah.nongnu.org/bugs/?func=detailitem&item_id=19157

which does not seem to have been reproduced and fixed, and also to

https://lists.gnu.org/archive/html/lwip-users/2013-09/msg00052.html which also does not appear to

be solved.

The tip of the tree lwIP does better than lwIP 1.4.1 because it does not allow TCP_SND_QUEUELEN

to block adding a segment to set FIN in tcp_enqueue_flags:

err_t

tcp_enqueue_flags(struct tcp_pcb *pcb, u8_t flags)

{

/* ... snip ... */

/* check for configured max queuelen and possible overflow (FIN flag should always come through!) */

if (((pcb->snd_queuelen >= TCP_SND_QUEUELEN) || (pcb->snd_queuelen > TCP_SNDQUEUELEN_OVERFLOW)) &&

((flags & TCP_FIN) == 0)) {

I have suggested fixes to ensure that the netconn API and specifically netconn_delete and

netconn_close always return.

They are here https://github.com/MikeStitt/lwip/tree/closeAlwaysReturnsAndOthers

on this commit:

https://github.com/MikeStitt/lwip/commit/d2658f1f5e1f8e822ecf7e3abcb52938e3cd7c9c

Within the suggested fixes, there are 3 areas of improvement

1)#define TCP_CLOSE_ALWAYS_RETURNS 1

Targets the cases where tcp_enqueue_flags can not allocate buffer space for the segment needed

to add the FIN bit. With TCP_CLOSE_ALWAYS_RETURNS, we choose to return an error in this case rather

than letting tcp_slowtmr try to resolve the memory issue later.

2)#define TCP_DO_NOT_ADD_FIN_TO_RST 1

Targets a corner case that occurs in this scenario where lwIP has decided that the connection should

be RST, but has not been able to push out the segment with the RST, and now on the close, wants to

add a FIN. By setting TCP_DO_NOT_ADD_FIN_TO_RST we close the connection without trying to add the FIN

in this case.

3)#define LWIP_PCB_COMPLETED_BOOKKEEPING 1

Adds a more robust solution to the application to TCP/IP thread netconn interface. It adds a to_be_completed

byte to the netconn data structure. For the sockets and netconn API, it the keeps track of whether or not the

application thread is blocked on the TCP/IP task for PCBs that are purged by tcp_slowtmr. tcp_slowtmr calls

the pcb->errf which will sys_sem_signal the application thread with a ERR_ABRT when the pcb is purged.

This ensures that the application netconn_close, netconn_delete or any socket/netconn API return

when a pcb is purged.

This is an important fix in my application because we do not want to have to reset the system to unblock

the application when the Ethernet link stays up but the connection to the internet goes down.

Thanks for your consideration of these suggested fixes.

-best

-Mike

Mike Stitt | Engineering Lead
SpinDance, Inc. | Smart Choice. Smart Products.
SpinDance.com | #SMARTproduct

#--------------------

# my lwipopts.h:

#--------------------

#ifndef __LWIPOPTS_H__

#define __LWIPOPTS_H__

#include "autoconf.h"

#include "logger.h"

#include "quickStart.h"

#include "incidentLog/log.h"

#include "rnd/rnd.h"

#include <sys/time.h>

/**

* TCP_CLOSE_ALWAYS_RETURNS==1: To force lwip_close and lwip_shutdown

* to always return. Otherwise lwIP can be deadlocked when it can not allocate

* the buffer space for the FIN to end a message.

#define TCP_CLOSE_ALWAYS_RETURNS 1

/**

* TCP_DO_NOT_ADD_FIN_TO_RST==1: When trying to close a connection, if

* we have already set the RST flag don't expend the effort to try to allocate

* a segment to add a RST. Just hang up quietly.

#define TCP_DO_NOT_ADD_FIN_TO_RST 1

/**

* LWIP should keep track of calling threads that are waiting on

* a call back and return an error to threads that are waiting when the

* pcb is purged.

#define LWIP_PCB_COMPLETED_BOOKKEEPING 1

/**

* SYS_LIGHTWEIGHT_PROT==1: if you want inter-task protection for certain

* critical regions during buffer allocation, deallocation and memory

* allocation and deallocation.

#define SYS_LIGHTWEIGHT_PROT 1

#define ETHARP_TRUST_IP_MAC 0

#define ETHARP_SUPPORT_VLAN 0

#define IP_REASSEMBLY 0

#define IP_FRAG 0

#define ARP_QUEUEING 0

#define TCP_LISTEN_BACKLOG 1

/**

* LWIP_RAW==1: Enable application layer to hook into the IP layer itself.

#define LWIP_RAW 0

/**

* NO_SYS==1: Provides VERY minimal functionality. Otherwise,

* use lwIP facilities.

#define NO_SYS 0

/* ---------- Memory options ---------- */

/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which

lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2

byte alignment -> define MEM_ALIGNMENT to 2. */

#define MEM_ALIGNMENT 4

/* MEM_SIZE: the size of the heap memory. If the application will send

a lot of data that needs to be copied, this should be set high. */

#define MEM_SIZE (5*1024)

/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application

sends a lot of data out of ROM (or other static memory), this

should be set high. */

#define MEMP_NUM_PBUF 10

/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One

per active UDP "connection". */

#define MEMP_NUM_UDP_PCB 6

/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCP

connections. */

#define MEMP_NUM_TCP_PCB 10

/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP

connections. */

#define MEMP_NUM_TCP_PCB_LISTEN 2

/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP

segments. */

#define MEMP_NUM_TCP_SEG 40

/* MEMP_NUM_SYS_TIMEOUT: the number of simultaneously active

timeouts. */

#define MEMP_NUM_SYS_TIMEOUT 10

/* MEMP_NUM_NETCONN: the number of struct netconns.

* (only needed if you use the sequential API, like api_lib.c) */

#define MEMP_NUM_NETCONN 5

/* The number of struct tcpip_msg, which are used for ALL incoming packets.

* Despite the name, this is NOT for TCP only,

* but UDP and other inbound packet types as well.

#define MEMP_NUM_TCPIP_MSG_INPKT 10

/* ---------- Pbuf options ---------- */

/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */

#define PBUF_POOL_SIZE 20

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */

#define PBUF_POOL_BUFSIZE 500

/* ---------- TCP options ---------- */

#define LWIP_TCP 1

#define TCP_TTL 255

/* The maximum segment lifetime in milliseconds */

#define TCP_MSL 4000

/* Controls if TCP should queue segments that arrive out of

order. Define to 0 if your device is low on memory. */

#define TCP_QUEUE_OOSEQ 0

/* TCP Maximum segment size. */

#define TCP_MSS (1500 - 40) /* TCP_MSS = (Ethernet MTU - IP header size - TCP header size) */

/* TCP sender buffer space (bytes). */

#define TCP_SND_BUF (2 * TCP_MSS)

/* TCP_SND_QUEUELEN: TCP sender buffer space (pbufs). This must be at least

as much as (2 * TCP_SND_BUF/TCP_MSS) for things to work. */

#define TCP_SND_QUEUELEN (2 * TCP_SND_BUF/TCP_MSS)

/* TCP_SNDQUEUELOWAT: TCP writable bufs (pbuf count). This must be less

* than TCP_SND_QUEUELEN. If the number of pbufs queued on a pcb drops below

* this number, select returns writable (combined with TCP_SNDLOWAT).

#define TCP_SNDQUEUELOWAT 3

/* TCP receive window. */

#define TCP_WND (2*TCP_MSS)

/* Random local ports */

#define LWIP_RANDOMIZE_INITIAL_LOCAL_PORTS 1

#define LWIP_RAND() rndGetNextRndUInt()

/* Random ISS - requires LWIP_RAND() */

#define LWIP_TCP_RANDOMIZE_INITIAL_ISS 1

/* value to increment ISS every coarse timer tick */

#define LWIP_TCP_ISS_TICK_MULTIPLIER 64000

/* ---------- ICMP options ---------- */

#define LWIP_ICMP 1

/* ---------- DHCP options ---------- */

/* Define LWIP_DHCP to 1 if you want DHCP configuration of

interfaces. */

#define LWIP_DHCP CONFIG_LWIP_DHCP

/* Enable AUTOIP and DHCP/AUTOIP cooperation to enable fallback to AUTOIP when

DHCP is not available. */

#define LWIP_AUTOIP CONFIG_LWIP_AUTOIP

#define LWIP_DHCP_AUTOIP_COOP (LWIP_DHCP && LWIP_AUTOIP)

/* Bump this down from the default of 9 to decrease the time it takes for DCHP

to timeout and switch over to AUTOIP */

#define LWIP_DHCP_AUTOIP_COOP_TRIES 5

#define LWIP_AUTOIP_TRIES_UNTIL_TRY_STATIC 5

/* ---------- UDP options ---------- */

#define LWIP_UDP 1

#define UDP_TTL 255

/* ---------- Statistics options ---------- */

#define LWIP_STATS CONFIG_LWIP_STATS

#define LWIP_PROVIDE_ERRNO 1

#if LWIP_STATS

#define LWIP_STATS_LARGE CONFIG_LWIP_STATS_LARGE

#define LINK_STATS CONFIG_LINK_STATS

#define ETHARP_STATS CONFIG_ETHARP_STATS

#define IPFRAG_STATS CONFIG_IPFRAG_STATS

#define IP_STATS CONFIG_IP_STATS

#define ICMP_STATS CONFIG_ICMP_STATS

#define IGMP_STATS CONFIG_IGMP_STATS

#define UDP_STATS CONFIG_UDP_STATS

#define TCP_STATS CONFIG_TCP_STATS

#define MEM_STATS CONFIG_MEM_STATS

#define MEMP_STATS CONFIG_MEMP_STATS

#define SYS_STATS CONFIG_SYS_STATS

#define LWIP_STATS_DISPLAY CONFIG_LWIP_STATS_DISPLAY

#endif

/* ---------- link callback options ---------- */

/* LWIP_NETIF_LINK_CALLBACK==1: Support a callback function from an interface

* whenever the link changes (i.e., link down)

#define LWIP_NETIF_LINK_CALLBACK 1

--------------------------------------

---------- Checksum options ----------

--------------------------------------

The STM32F4x7 allows computing and verifying the IP, UDP, TCP and ICMP checksums by hardware:

- To use this feature let the following define uncommented.

- To disable it and process by CPU comment the the checksum.

#define CHECKSUM_BY_HARDWARE

#ifdef CHECKSUM_BY_HARDWARE

/* CHECKSUM_GEN_IP==0: Generate checksums by hardware for outgoing IP packets.*/

#define CHECKSUM_GEN_IP 0

/* CHECKSUM_GEN_UDP==0: Generate checksums by hardware for outgoing UDP packets.*/

#define CHECKSUM_GEN_UDP 0

/* CHECKSUM_GEN_TCP==0: Generate checksums by hardware for outgoing TCP packets.*/

#define CHECKSUM_GEN_TCP 0

/* CHECKSUM_CHECK_IP==0: Check checksums by hardware for incoming IP packets.*/

#define CHECKSUM_CHECK_IP 0

/* CHECKSUM_CHECK_UDP==0: Check checksums by hardware for incoming UDP packets.*/

#define CHECKSUM_CHECK_UDP 0

/* CHECKSUM_CHECK_TCP==0: Check checksums by hardware for incoming TCP packets.*/

#define CHECKSUM_CHECK_TCP 0

/* CHECKSUM_CHECK_ICMP==0: Check checksums by hardware for incoming ICMP packets.*/

#define CHECKSUM_GEN_ICMP 0

#else

/* CHECKSUM_GEN_IP==1: Generate checksums in software for outgoing IP packets.*/

#define CHECKSUM_GEN_IP 1

/* CHECKSUM_GEN_UDP==1: Generate checksums in software for outgoing UDP packets.*/

#define CHECKSUM_GEN_UDP 1

/* CHECKSUM_GEN_TCP==1: Generate checksums in software for outgoing TCP packets.*/

#define CHECKSUM_GEN_TCP 1

/* CHECKSUM_CHECK_IP==1: Check checksums in software for incoming IP packets.*/

#define CHECKSUM_CHECK_IP 1

/* CHECKSUM_CHECK_UDP==1: Check checksums in software for incoming UDP packets.*/

#define CHECKSUM_CHECK_UDP 1

/* CHECKSUM_CHECK_TCP==1: Check checksums in software for incoming TCP packets.*/

#define CHECKSUM_CHECK_TCP 1

/* CHECKSUM_CHECK_ICMP==1: Check checksums by hardware for incoming ICMP packets.*/

#define CHECKSUM_GEN_ICMP 1

#endif

------------------------------------

---------- LOOPIF options ----------

------------------------------------

/**

* LWIP_HAVE_LOOPIF==1: Support loop interface (127.0.0.1) and loopif.c

#define LWIP_HAVE_LOOPIF 1

----------------------------------------------

---------- Sequential layer options ----------

----------------------------------------------

/**

* LWIP_NETCONN==1: Enable Netconn API (require to use api_lib.c)

#define LWIP_NETCONN 1

#define LWIP_SO_RCVTIMEO 1

#define LWIP_SO_SNDTIMEO 1

------------------------------------

---------- Socket options ----------

------------------------------------

/**

* LWIP_SOCKET==1: Enable Socket API (require to use sockets.c)

#define LWIP_SOCKET 1

/**

* LWIP_TIMEVAL_PRIVATE==1: Use the struct timeval provided in sockets.h

* Otherwise include <sys/time.h> to use the one provided by C Library.

#define LWIP_TIMEVAL_PRIVATE 0

LWIP_DNS

Required for librabbitmq to build.

#define LWIP_DNS 1

-----------------------------------

---------- DEBUG options ----------

-----------------------------------

#if defined CONFIG_LWIP_DEBUG && CONFIG_LWIP_DEBUG && defined CONFIG_LWIP_DIAG_TO_SERIAL && CONFIG_LWIP_DIAG_TO_SERIAL

#define LWIP_PLATFORM_DIAG(x) {lTimePrintf x;}

#elif defined CONFIG_LWIP_DEBUG && CONFIG_LWIP_DEBUG && defined CONFIG_LWIP_DIAG_TO_LOG && CONFIG_LWIP_DIAG_TO_LOG

#define LWIP_PLATFORM_DIAG(x) { char * msgP = lwipPrintf x ; \

logCoreApi( \

__FUNCTION__, \

__LINE__, \

eLogLevelInterface, \

LOG_ALL_FLAGS_ON, \

msgP ); }

#else

#define LWIP_PLATFORM_DIAG(x) { }

#endif

/**

* Set LWIP_DEBUG to 1

* and uncomment *_DEBUG feature lines

* to enable debugging.

* TBDRMS debugging is also on with LWIP_DEBUG 0

* and the feature lines uncommented. Hmmm....

#if defined CONFIG_LWIP_DEBUG && CONFIG_LWIP_DEBUG

# define LWIP_DEBUG 1

#define LWIP_DBG_TYPES_ON (LWIP_DBG_ON|LWIP_DBG_TRACE|LWIP_DBG_STATE|LWIP_DBG_FRESH)

///#define LWIP_DBG_MIN_LEVEL LWIP_DBG_LEVEL_SEVERE

#define LWIP_DEBUG_TIMERNAMES 0

//#define ETHARP_DEBUG LWIP_DBG_ON

//#define NETIF_DEBUG LWIP_DBG_ON

//#define PBUF_DEBUG LWIP_DBG_ON

//#define API_LIB_DEBUG LWIP_DBG_ON

//#define API_MSG_DEBUG LWIP_DBG_ON

//#define SOCKETS_DEBUG LWIP_DBG_ON

//#define ICMP_DEBUG LWIP_DBG_ON

//#define IGMP_DEBUG LWIP_DBG_ON

//#define INET_DEBUG LWIP_DBG_ON

//#define IP_DEBUG LWIP_DBG_ON

//#define IP_REASS_DEBUG LWIP_DBG_ON

//#define RAW_DEBUG LWIP_DBG_ON

//#define MEM_DEBUG LWIP_DBG_ON

//#define MEMP_DEBUG LWIP_DBG_ON

//#define SYS_DEBUG LWIP_DBG_ON

//#define TIMERS_DEBUG LWIP_DBG_ON

#if 0

# if defined CONFIG_INCDNTLOG_LWIP_TCP_GENERAL_DEBUG && CONFIG_INCDNTLOG_LWIP_TCP_GENERAL_DEBUG

# define TCP_DEBUG LWIP_DBG_ON

# define TCP_INPUT_DEBUG LWIP_DBG_ON

# define TCP_FR_DEBUG LWIP_DBG_ON

# define TCP_RTO_DEBUG LWIP_DBG_ON

# define TCP_CWND_DEBUG LWIP_DBG_ON

# define TCP_WND_DEBUG LWIP_DBG_ON

# define TCP_OUTPUT_DEBUG LWIP_DBG_ON

# define TCP_RST_DEBUG LWIP_DBG_ON

# define TCP_QLEN_DEBUG LWIP_DBG_ON

# endif

#elif 1

# if defined CONFIG_INCDNTLOG_LWIP_TCP_GENERAL_DEBUG && CONFIG_INCDNTLOG_LWIP_TCP_GENERAL_DEBUG

# define TCP_CLOSE_DEBUG LWIP_DBG_ON

# endif

#else

# if defined CONFIG_INCDNTLOG_LWIP_TCP_GENERAL_DEBUG && CONFIG_INCDNTLOG_LWIP_TCP_GENERAL_DEBUG

# define IP_DEBUG LWIP_DBG_ON

# define TCP_DEBUG LWIP_DBG_ON

# define TCP_INPUT_DEBUG LWIP_DBG_ON

//# define TCP_FR_DEBUG LWIP_DBG_ON

//# define TCP_RTO_DEBUG LWIP_DBG_ON

//# define TCP_CWND_DEBUG LWIP_DBG_ON

//# define TCP_WND_DEBUG LWIP_DBG_ON

# define TCP_OUTPUT_DEBUG LWIP_DBG_ON

//# define TCP_RST_DEBUG LWIP_DBG_ON

//# define TCP_QLEN_DEBUG LWIP_DBG_ON

#define SOCKETS_DEBUG LWIP_DBG_ON

#define TCPIP_DEBUG LWIP_DBG_ON

#define MEM_DEBUG LWIP_DBG_ON

#define MEMP_DEBUG LWIP_DBG_ON

# endif

#endif

//#define UDP_DEBUG LWIP_DBG_ON

//#define TCPIP_DEBUG LWIP_DBG_ON

//#define PPP_DEBUG LWIP_DBG_ON

//#define SLIP_DEBUG LWIP_DBG_ON

//#define DHCP_DEBUG LWIP_DBG_ON

//#define AUTOIP_DEBUG LWIP_DBG_ON

//#define SNMP_MSG_DEBUG LWIP_DBG_ON

//#define SNMP_MIB_DEBUG LWIP_DBG_ON

//#define DNS_DEBUG LWIP_DBG_ON

#else

# define LWIP_DEBUG 0

#endif

---------------------------------

---------- OS options ----------

---------------------------------

#define TCPIP_THREAD_NAME "TCP/IP"

#define TCPIP_THREAD_STACKSIZE 800

#define TCPIP_MBOX_SIZE 40

#define DEFAULT_UDP_RECVMBOX_SIZE 6

#define DEFAULT_TCP_RECVMBOX_SIZE 6

#define DEFAULT_ACCEPTMBOX_SIZE 6

#define DEFAULT_THREAD_STACKSIZE 500

#ifndef TCPIP_THREAD_PRIO

#define TCPIP_THREAD_PRIO (configMAX_PRIORITIES - 2)

#endif

#define LWIP_COMPAT_MUTEX 1

---------------------------------

--------- HTTPD options ---------

---------------------------------

#define LWIP_HTTPD_SSI CONFIG_FACILITY_SSI

#define LWIP_HTTPD_SSI_MULTIPART CONFIG_FACILITY_SSI

#define HTTPD_USE_CUSTOM_FSDATA CONFIG_FACILITY_HTTPD

#define LWIP_HTTPD_CGI CONFIG_FACILITY_CGI

#define LWIP_HTTPD_CUSTOM_FILES CONFIG_FACILITY_CGI

#define LWIP_HTTPD_MAX_TAG_NAME_LEN 64

#define LWIP_HTTPD_DYNAMIC_FILE_READ 1

#define LWIP_HTTPD_MAX_TAG_INSERT_LEN 192

#define LWIP_HTTPD_DYNAMIC_HEADERS 0

#define LWIP_HTTPD_SUPPORT_POST 0

#endif /* __LWIPOPTS_H__ */

From:	Mike Stitt
Subject:	[lwip-users] Netconn API, e.g. lwip_close, netconn_close, or netconn_delete, sometimes does not return, with suggested fixes.
Date:	Sun, 30 Nov 2014 14:51:41 -0500

[lwip-users] Netconn API, e.g. lwip_close, netconn_close, or netconn_del