Build refactor and dockerisation of Mediagoblin

[Olivier Mehani]

Ahoy there,

I have been working on trying to refactor the build system and better 
dockerise the Mediagoblin source for a few months, and have something 
that is _almost_ working [0].

While it's certainly still a WIP, I wanted to lay out my goals here, so 
we can discuss them, and use them to drive the rest of this work.


# For users

What I want users to be able to do is to just grab a template 
`docker-compose.yml`, and do a `docker-compose up` to have a 
ready-to-use, non-lazy, MG instance.

By default, it would store config, DB, and media in an FS volume mounted 
from the host into the container. Importantly, the container would 
create all those by default on first run if missing. Another important 
feature is that the container should migrate the DB whenever needed 
[e.g., like the Paperless-NG containers 1], so an MG upgrade would 
simply be a `docker-compose pull; docker-compose down; docker-compose 
up`.

I'm building the MG image itself to start the lazy-server by default, 
but the docker-compose should ultimately start two containers from the 
same image, for paste and celery. It would also start a rabbimq 
container from upstream images.

At this stage, the single-container docker image seems pretty good, but 
the docker-compose bit is still not done. That single-container image is 
probably something we'd want to push to Docker hub every release.


# For developers

On the build side, I tried to move as much of the logic as I could 
outside of the Makefile, and straight into the Python setup, including 
deps that historically were provided by the system, such as lxml.
This means that MG is a lot more like a normal Python app (which is part 
of the ultimate goal, maybe even pushing it to PyPI). Unfortunately, due 
to GStreamer and GObject, I don't think it is entirely feasible to build 
and run in a stock Python container, so there still is some 
(in-container) Debian-magic to install those dependencies from the 
distro.

The idea is also to make the MG image as small as possible, so not 
shipping builds dependencies (particularly JS ones) seems like a good 
idea. For those build dependencies (mainly node.js, bower and 
node_modules), I have a separate build stage which processes the assets, 
and make them available for Python to package afterwards (this bit 
works, but it is not thoroughly tested yet).

As Mediagoblin is a GNU project, my understanding is that we are 
required to have a `./configure` script. While I want to have most of 
the build out of the Makefile, I think we'll have to keep simple 
versions of those around. I think it's ok, as it make it easier for 
people to get going from source. I think they will need to be greatly 
simplified, though.

Essentially, the configure script should either check that Docker is 
available (and nothing else, as a default behaviour), or alternatively, 
check for Python, GStreamer, Bower and other dependencies for a 
non-Docker build. Ideally, the generated Makefile should just have short 
calls to either Docker or the Python build (maybe I should make all the 
asset-building with bower into a Python build step, too).


I'll be pretty happy when I reach a point where all of what's described 
above is ready. I'm happy to receive comments and/or suggestions on this 
goal, but will otherwise keep working towards it, module revisions.


# Not addressed

A couple of things that will need to be fixed down the line, but that I 
don't intend to touch immediately are

* the Git submodules: we should probably have a better way to bring them 
  in, that doesn't depend on Git. For the moment, the bootstrap script 
  is still needed to be run before the configure or any other build 
  steps, Docker or not.

* the plugins: at the moment, I package Python dependencies for all of 
  them within the image (though I have separate `extra` sections in the 
  `setup.cfg` for better modularity, maybe to be supported via the 
  configure script in the future). I'm not sure how to support 
  external/custom plugins, but I think it would make sense to modularise 
  the MG code so it can load plugins from custom locations. A user could 
  then provide their own plugins in the Docker volume along with 
  configuration/DB/media. This is similar to how Home Assistant [2] does 
  it for `custom_components`. I'm not sure how to handle additional 
  Python deps, but I guess they could be installed there in the same 
  way.


So anyway, let me know what you think, and if there is any way to 
improve on that goal while I work on it.


[0] https://git.sr.ht/~shtrom/mediagoblin/log/refactor-build
[1] https://github.com/jonaswinkler/paperless-ng
[2] https://github.com/home-assistant/docker

--
Olivier Mehani <shtrom+mediagoblin@ssji.net>
PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655
Confidentiality cannot be guaranteed on emails sent or received unencrypted.