Re: celery connection problems to RabbitMQ

[Ben Sturmfels]

Hi Fernando,

Please post a patch or a link to a remote branch to an issue on the
issue tracker - ideally a separate change for celery and spectrograms.

Regarding PySoundFile, the file you're after is setup.py.

For what it's worth, I see that the Python SoundFile library is
available in Debian, but PySoundFile doesn't appear to be. This isn't a
complete showstopper, but it would help us when tackling distro
packaging in the near future.

Regards,
Ben

On Mon, 11 May 2020, Fernando Gutierrez wrote:

> Hi Ben
>
> Sorry I think I didn't explain clearly. I only fixed the connection reset
> exceptions in celery but the bug with media changed to failed state is
> not fixed.
>
> I will continue debugging but it may take some time. I don't know
> why celery thinks a completed task needs to be run again.
>
> In the meantime I will submit a patch for the systemd file, the
> BROKER_HEARTBEAK issue and also a fix for the audio spectrogram
> code as I mentioned in the IRC channel.
>
> I have a couple of questions:
>
> 1) I'm not familiar with the development process, I already created an
> account in savannah.gnu.org but I don't see how to submit a patch
> for review.
> 2) For the spectrogram I used the PySoundFile package. What file do
> I need to modify so it gets pulled during setup?, in my setup I
> manually called ./bin/pip install PySoundFile
>
> Thanks
> Fernando
>
> On Sun, May 10, 2020 at 6:39 AM Ben Sturmfels
> <address@hidden> wrote:
>
>  Hi Fernando,
>
>  On Sun, 10 May 2020, Fernando Gutierrez wrote:
>
>  > I recently asked in the IRC channel about RabbitMQ connection
>  reset
>  > errors in celeryd logs.
>  >
>  > I think there are two issues:
>  >
>  > 1) The example systemd file (mediagoblin-celeryd.service) from
>  >
>  https://mediagoblin.readthedocs.io/en/stable/siteadmin/deploying.html
>  
>  > does not specify that celeryd must be started after RabbitMQ,
>  so it is
>  > sometimes started before and fails because RabbitMQ is not
>  running
>  > yet.
>  >
>  > 2) In mediagoblin/mediagoblin/init/celery/__init__.py, it sets
>  > celery_settings['BROKER_HEARTBEAT'] = 1. In slower systems
>  or
>  > under heavy load if the worker is too slow to respond in < 1
>  second it
>  > will miss the heartbeat and after a few missed heartbeats the
>  > connection is considered dead and reset.
>  > I'm not sure what is the purpose of changing
>  BROKER_HEARTBEAT to
>  > 1 but the celery docs recommend not using such a small
>  value. In my
>  > install I changed it to 20 and I no longer see any connection
>  > problems.
>  >
>  > Are you willing to accept a patch for
>  > mediagoblin/docs/source/siteadmin/deployment.rst and
>  > mediagoblin/mediagoblin/init/celery/__init__.py to fix those two
>  > problems?
>
>  Thank you very much for diving in and investigating the issue.
>  We'd be
>  happy to take a patch on this. If you can add a comment to
>  explain the
>  new BROKER_HEARTBEAT value in the code, that would be great.
>
>  I wonder if there there might still be a problem lurking here
>  though,
>  even if your system is now working properly Not being able to
>  connect to
>  RabbitMQ or an unresponsive celery worker probably shouldn't
>  change
>  existing processed media items to failed.
>
>  Thanks for your work on this!
>
>  Regards,
>  Ben