Re: [libmicrohttpd] Sockets remain in CLOSE_WAIT state

[Santos Das]

Hi Christian,

 

We see that at the end of the load run, the connections at the TCP server remain in CLOSE_WAIT state. We didn’t see this for single call. Also, I wanted to tell you that all our connection are initiated from a single IP as that is the load balancer IP and we are behind that.

We also observed that it  gets stuck in the CLOSE_WAIT state and remains there forever , i.e Sockets are seen in this state even after the default of TCP connection timeout.

From all my study, it points out the TCP stack  will keep the connection in CLOSE_WAIT state till the application call close on the socket.  

Do we know when MHD call close normally ? Is there any instrumentation that could tell us why /whether FIN is sent from the Server or NOT ? 

 

https://access.redhat.com/solutions/437133

 

CLOSE_WAIT - Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed. This means the socket is waiting for the application to execute close(). A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like a file descriptor leak: server not executing close() on sockets leading to pile up of CLOSE_WAIT sockets    

On Sat, Jan 5, 2019 at 10:04 AM Santos Das <address@hidden> wrote:

Hi Christian,

We see that it never times out, i.e remain in CLOSE_WAIT state forever.

Also, it is seen irrespective of whether we use loop back ip or a regular eth0 IP.

From my reading, I understand the following and we don't see that FIN is received from Client but , we don't send FIN.

CLOSE_WAIT means that the local end of the connection has received a FIN from the other end, but the OS is waiting for the program at the local end to actually close its connection.  

On Sat, Jan 5, 2019 at 4:36 AM Christian Grothoff <address@hidden> wrote:

Hi Santos,

CLOSE_WAIT is a TCP state in the kernel after a TCP connection is
finished. It typically lasts like 60--300s, depending on your kernel
configuration (you may want to read up on the TCP state machine).
You can change the time by changing
/proc/sys/net/ipv4/tcp_fin_timeout, but usually this is not an issue
(unless you have _way_ too many TCP connections from the same host, I've
only seen this matter on loopback, in which case usually enabling
keep-alive on the client-side helps).

Summary: this is not an issue in MHD, likely not one for anyone, just a
TCP artifact.

Happy hacking!

-Christian

On 1/4/19 6:29 PM, Santos Das wrote:
> Hi Christian,
>
> Please note I am using suspend-resume and not
> implementing MHD_OPTION_NOTIFY_CONNECTION to close . Not sure when the
> library closes the connection after it receives FIN and move to CLOSE_WAIT.
>
> Any pointers?
>
> thanks, santos
>
> On Fri, Jan 4, 2019 at 10:27 PM Santos Das <address@hidden
> <mailto:address@hidden>> wrote:
>
>     Hi Christian,
>
>     Wishing you a happy new year!
>
>     We have set the MHD_OPTION_CONNECTION_TIMEOUT to 0 and we see that
>     all the connections remain in CLOSE_WAIT state. 
>
>     We also have set the MHD_OPTION_CONNECTION_LIMIT to 1000 .
>
>     Any idea what could be wrong ?
>
>     Thanks, Santos
>