Re: [GMG-Devel] Help extract documentation from my brain

[Christopher Allan Webber]

Okay!  Thanks everyone for your feedback on what's useful to have
documented.  I'm kicking it off with database migrations.

Database Migrations
===================

What are migrations? 
--------------------

Sometimes the way we store data changes.  We might add a new field or
deprecate an old field.  Even though MongoDB allows us to store things
very "flexibly", it's important that as we change our schema (add new
fields, remove fields, rename fields, restructure some data) that our
database is updated to have things stored in the same form that we
expect to access them in our codebase.

How to run 
----------

Pretty simple!  Just run:

./bin/gmg migrate

How to add a new migration 
--------------------------

Migrations are handled in:

mediagoblin/db/migrations.py

Migrations aren't too complex!  Basically they're just python
arguments that take a pymongo database as their sole argument.

Note, this is a pymongo database and NOT a mongokit database!  The
reason for this is we don't want people to use the ORM to avoida a
chicken and egg type paradox: our ORM might have tools that expect
things to be at our current schema

So we force people to use the simple pymongo database API, which is
itself not too hard!

Let's look at one from the unit tests:


  @RegisterMigration(1, TEST_MIGRATION_REGISTRY)
  def creature_add_magical_powers(database):
      """
      Add lists of magical powers.
  
      This defaults to [], an empty list.  Since we haven't declared any
      magical powers, all existing monsters should
      """
      database['creatures'].update(
          {'magical_powers': {'$exists': False}},
          {'$set': {'magical_powers': []}},
          multi=True)

This is a fairly simple example where we're using the update command
to find all instances where there aren't any magical powers, and we
set the magical powers thusly to an empty list.  For more on update
commands and mongodb, see:

[http://www.mongodb.org/display/DOCS/Updating]

You'll notice that preceding the migration is a decorator called
RegisterMigration.  This takes two arguments: the number of this
migration (increment this by one from whatever the last migration was)
and the "migration registry" we'll be storing it in.  (Probably
MIGRATIONS).  This decorator will thusly record your migration in the
registry as being this version.  (This is compared against the
"current_migration"'s value stored in your database's 'app_metadata'
collection in the '_id': 'mediagoblin' document so it knows what new
migrations must be run.)

Another example, from the real migrations in migrations.py:


  @RegisterMigration(1)
  def user_add_bio_html(database):
      """
      Users now have richtext bios via Markdown, reflect appropriately.
      """
      collection = database['users']
  
      target = collection.find(
          {'bio_html': {'$exists': False}})
  
      for document in target:
          document['bio_html'] = cleaned_markdown_conversion(
              document['bio'])
          collection.save(document)

This one is slightly more compliated, but still not too hard.  It's
just taking the value from the key 'bio', running it through the
cleaned_markdown_conversion, and storing that result in bio_html.

(Just one more note, if you're using the update method, for now please
don't use the '$rename' modifier as it isn't supported in versions of
mongodb used in most current stable distributions.)