[Maposmatic-dev] MapOSMatic daemon rewrite

[Maxime Petazzoni]

Hi all,

As promised, I started rewriting the MapOSMatic rendering daemon. This patch
series brings the basis for a new rendering daemon, currently with the same
level of functionality as the existing daemon, but with I hope a much more
cleaner, Python-y and extensible code base.

My previous, unmerged patch "Improve the file cleanup mechanism" is included in
this patch series too, and further work is based on it.

Patch series, on latest master:

  [PATCH maposmatic 1/9] Improve the file cleanup mechanism
  [PATCH maposmatic 2/9] Cornerstones for a new MapOSMatic daemon
  [PATCH maposmatic 3/9] Update the .gitignore list
  [PATCH maposmatic 4/9] Add MAPOSMATIC_LOG_LEVEL to the environment wrapping
  [PATCH maposmatic 5/9] Merge the StandaloneMapOSMaticDaemon into the base
    MapOSMaticDaemon class
  [PATCH maposmatic 6/9] Revamp the job renderer
  [PATCH maposmatic 7/9] Rework the daemon to use the new JobRenderers
  [PATCH maposmatic 8/9] Frequency parameters passing improvement
  [PATCH maposmatic 9/9] Provide a map_areas prefix to the TimingOutJobRenderer

As of now, rendering of several jobs in parallel is still not implemented, but
the framework for doing so is, and I should get a working implementation very
soon using either the multiprocessing or the subprocess module, whichever is
easier and/or more efficient.

Except from the major refactoring/rewrite, there are two main differences with
the previous version of the daemon with regard to the way the daemon works:

  - the files cleanup is part of a standalone thread, so cleaning can occur
    while a rendering is processed by the daemon, thus making sure space will
    be available for the rendering as it goes.

    Seeing the cleanup thread kick in to make space for a rendering being
    processed is kinda nice :-)

  - the rendering is no longer done in a child process. I know we've discussed
    possible memory leaks in Mapnik/Psycopg2 clubbering the main daemon
    process. But the code presented here is, you can't deny it, much more
    simple, clean and efficient.

    I would like to see how it performs on dev.maposmatic.org before completely
    closing the "threading option" from us. And quite frankly, I think it
    should perform beautifully.

Before continuing working on the parallelization of the renderings, I would
like a first round of reviews on this. I've been using this new version of the
daemon locally without any problems and it seems to be working very nicely.

If you like it, we could deploy it on dev.maposmatic.org and let it run for a
while (so maybe we could first put the current dev.m.o to production?), and see
what's what.

Regards,
- Maxime