Re: [libmicrohttpd] Trouble getting a response sent from a separateworker thread (with external select)

[Marcos Pindado Sebastian]

Hi all, may I share some comments to this scenario?
we have just implemented the external select with asynchronous tasks performed 
in different threads in centos+redhat in 0.37-0.38, so it obviously works.

1. You have to supply your own file descriptor to the external select in order 
to notify the select thread the arrival of new messages. So the scenario should 
be:
 - In access_handler you create an async task (thread)
 - the task finishes and notifies the main thread (awaiting in select) writing 
in an FD
- The main thread calls MHD_run and the access_handler gets called again, this 
time queuing the response

So one thing to do to make this work is to call mhd functions only on the 
external select thread and to have a certain "state" in the access_handler, 
knowing that it can be called several times and only should queue the response 
when the worker thread has finished.

2. This supplied file descriptor can be anything readable+writeable and should 
be set in your fdset. We have used pipes (the old way) and eventFds (in recent 
linux kernels) which are optimized.
In each iteration, the external select thread calls mhd_getfdset and then the 
eventFd is added to the set. This fd is known by the worker threads:
      - The worker threads do the job and insert the response in a shared 
memory object (obviously synchronized with a mutex).
      - the worker threads writes to the fd => The select is notified and the 
main thread wakes up.

3. The shared memory obviously should be created using the void** con_cls 
facility in the access_handler, creating an structure or class. The transaction 
state should be stored there also (similarly as the post examples).

4. About suspend/resume. If using external SELECT and NO EPOLL (linux only), we 
found not necessary to use suspend+resume. In fact, suspending the connection 
has one disadvantage: you do not notice when the client disconnects, or better, 
you do not notice at the time disconnect happens.
BUT, if using external SELECT and EPOLL, (linux only), MHD do busy-waiting. And 
to avoid this you should call suspend+resume. With busy-waiting I mean that MHD 
begins to call the access_handler in a loop until a response is queued.
To handle disconnects you should pass a request_completed_callback to mhd and 
check the termination code.

EPOLL+external+busy is something I would like to check with sources when I have 
time, having that would permit mhd handle thousands of connections in this mode.

Best regards
Marcos

On 11/04/2014 08:11 PM, Tom Cornell wrote:
>> Date: Tue, 04 Nov 2014 18:44:53 +0100
>> From: Christian Grothoff <address@hidden>
>> To: address@hidden
>> Subject: Re: [libmicrohttpd] Trouble getting a response sent from a
>>      separate worker thread (with external select)
>> Message-ID: <address@hidden>
>> Content-Type: text/plain; charset="windows-1252"
>>
>> Dear Tom,
>>
>> Did you pass the "MHD_USE_SUSPEND_RESUME" flag? Also, you're not
>> supposed to call 'MHD_queue_response" from anywhere but the access
>> handler callback, so if you call it from another thread, the behavior is not
>> defined (or, more likely, you got an error value returned and thus your
>> response was NOT actually queued but refused --- did you check the return
>> value?).
>>
>> So please (1) check return values, (2) pass USE_SUSPEND_RESUME, and
>> (3) call 'resume' from the other thread but queue the reply in the access
>> handler.  After that, it should work. If it does not, make sure you run a 
>> recent
>> MHD release. If that still fails, a testcase to reproduce would likely be
>> helpful...
>>
>> Happy hacking!
>>
>> Christian
>
> First off, thanks for the quick reply!
>
> (1) Yes -- In particular, the call to MHD_queue_response returned MHD_YES.
>  (And the response did eventually get sent, only select() had to time out 
> first.
> Then control returned to the main loop and the next call to select()
> picked up the response and sent it.)

Ah, yes, I overlooked that detail in your message.

> (2) I did set the USE_SUSPEND_RESUME flag on starting the daemon.
> (Learned that lesson early on.)

:-).

> (3) I guess this must be the problem then.
> So I can create the MHD_Response and queue it in the access handler,
> but fill in the response data (status code, body) elsewhere, at a later time?
> Do I need to be using MHD_create_response_from_callback to create
> the MHD_Response? Or does that not matter?

You cannot set the status code after queuing.  But the *idea* is that
you call resume, and then the access handler *should* be called
"immediately" allowing you to queue the response with the status code.

> This is version 0.9.37, built from source on CentOS 6, just for completeness.

Ah, CentOS.  That platform is obviously less tested as I have no access.
So if, after doing (3), you still have the issue, producing a testcase
would be particularly helpful as I could check if the same issue is
present on other platforms or if it is CentOS-specific.

Happy hacking!

Christian

Antes de imprimir, piensa en el Medio Ambiente!

Confidencialidad Este correo electrónico y, en su caso, cualquier documento 
anexo al mismo, puede contener información confidencial o de datos de carácter 
personal, dirigida a su destinatario o destinatarios. Queda prohibida su 
divulgación, copia o distribución a terceros sin la previa autorización escrita 
del remitente. Si Ud. recibe este correo electrónico y cualquier documento 
anejo al mismo, por error, no debe divulgarlo, copiarlo, ni entregarlo a 
terceros. En el supuesto de que usted no fuera el destinatario, le rogamos que 
lo indique a la dirección electrónica del remitente y no comunique su contenido 
a terceros, procediendo a su destrucción.
Las opiniones, conclusiones y demás información incluidas en este correo y, en 
su caso, en cualquier documento anejo al mismo, que no estén relacionadas con 
asuntos estrictamente profesionales de AUBAY SPAIN S.A.U,  se entenderán como 
no respaldadas por la entidad, siendo de la exclusiva responsabilidad del 
remitente.
El remitente no garantiza la integridad, rapidez o seguridad del presente 
correo, ni se responsabiliza de posibles perjuicios derivados de su captura, 
incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por 
terceros.

____________
Verificada la ausencia de virus por G Data MailSecurity
 
Versión: AVA 24.4788 del 05.11.2014 
Noticias de virus: www.antiviruslab.com