Re: [GMG-Devel] Plugins plans

[Christopher Allan Webber]

Hi all,

So I did even further thinking on this based on what we said on this
thread and also looking at the previous plugin design document:

http://wiki.mediagoblin.org/PluginSystemDesignDocument

Reading that, as well as the rest of the thread, probably gives better
context for the rest of the things I'm writing here.  It looks like we
need the following things still:


====================================================
Hooks that loop all the way through, hooks that halt
====================================================

We can handle notification and transform via the "callable hooks"
system, but we don't have a "handler" that is the last one to get it
and then things stop.

One way to handle this: have a series of method calling handlers
that do things in different ways, like:

 - callable_hook_runone: runs them all till they stop
   - which order?  Does the last or the first get to claim it?
 - callable_hook_runall: runs every one of the hooks
   - likewise... which order?

.. both of those would expect *args, **kwargs and pass those keyword
arguments onto the hooks.

It might also be possible to make some callable classes like this,
like:

   callable_hook = CallableHookRunone(
      'some_hookname', accrue=True)  # accrues the results into a list
   results = callable_hook(arg1, arg2, kwarg="foo")


======================
"meta-decorator hooks"
======================

This is an incredibly evil and flexible idea that, like decorators
itself, is probably elegant, hard to understand, powerful, and yes, a
little bit evil.

So this is an idea inspired from emacs lisp, which allows you to pass
in functions to wrap a function that already exists.  Actually, hooks
in emacs look a heck of a lot like decorators, so actually...

Say you have this function:

  @meta_decorator_hook('some_function_hook')
  def some_function(foo):
     bla bla

Now, you want to wrap this function!  But how do you wrap it?  It's
already defined, you can't just wrap arbitrary things around it!
... or can you?

In this meta_decorator_hook, the meta_decorator_hook returns a
lazy-loaded method set to actually construct a chain of wrapped
methods.  So if we did something like:

  pluginapi.wrap_function('some_function_hook', our_function)

.. it would return, basically a WrappedFunction() that, the first
time it is executed, walks through all the decorator like methods
that have been pushed onto that hook and wraps them in each other.
That meta-decorated method is cached, and eventually, executed.

Sounds evil, right?  :)

This could be super powerful though.  You could do crazy things like
double-check the results of a view before returning it, or totally
override the view and decide not to call it at all for your plugin,
call it both before and after, or set up special-case exception
handling for hooks that are called within the function itself.

I'm sure I didn't make that sound any less evil, but I think it'd be
really great.


============================================================
How do we "halt" a function if a plugin needs to tell us to?
============================================================

Exceptions probably... but how to make this clean?

Okay, let me explain a bit more.  Say someone implements a special
hook that sets up something special (I'm having a hard time thinking
of what) but notices that something is terribly wrong.  It needs to
immediately halt the execution of the function.  A reasonable thing
to do is raise an exception.  But we don't just want it to show up as
a programming error and hit the logs.  How to handle it?  One of two
ways:

We could already be expecting certain types of exceptions to possibly
be raised like the following:

  try:
     pluginapi.callable_hook_runall('some_hook', arg, blarg)
  except ExpectingThisException:
     something something
     return Response('oh snap')

Okay, but what if the view designers weren't able to anticipate the
kind of issue you have... how to handle this?

The answer is the meta-decorators solution above I think.  In your
own meta-decorator around the view that calls the hook above, you
could set yourself up to catch the exception.

Tricky, but should work! :)


============================
WTForms form extension hooks
============================

This has been discussed previously.  But basically I wonder if it's
possible to do something like this:

  class BlaBlaForm(pluginapi.HookForm):
    _hook_name = 'blabla'
    bla_bla = wtforms.BlaBlaField()

This would require some fancy metaclassing but if it worked, it
should be able to allow to attach fields to the form either before or
after like the following:

  pluginapi.add_form_field(
     wtforms.SomeFormField(),
     position=before)

Is this possible?  Not totally sure.  Would be great if it did.
Would also involve evil metaclassing.  But what part of plugin
systems don't involve a bit of evil ;)


==============================
Interface implementation hooks
==============================

Sometimes we want people to supply an alternate interface to "swap out"
certain things.  For example, they might want a totally different user
authentication system, and maybe we'd have some sort of user
authentication api.

Altenately, the storage interface is a good example.

In this method we set up a minimal interface and a default
implementation.  Users then have the option to swap out this
implementation.

I think the way storage interfaces look are already currently pretty
good, so swapping out the interface could probably be like:

  pluginapi.set_component('public_storage', BagOfHolding())


=============================
Static files and staticdirect
=============================

Plugins need a way to add their own static files, like CSS, images, etc.
They also need ways for them to be linked and statically served.  Then
they also need a way to include those files in templates.

The docs make some mention of this... the assetlink command we have can
be extended for this, and I think we can both extend staticdirect to
take a "domain" like:

  {{ request.staticdirect('/images/foo.jpg', domain='plugin:someplugin') }}
  {{ request.staticdirect('/images/foo.jpg', domain='theme') }}

This could also have fallback support.


================================
Namespacing the hook repository?
================================

We already have callable hooks and we have template hooks.  I wonder if
we should provide a generic API for fetching and storing them?  I guess
maybe this won't work with all hook types... not sure.


====================================
Version checking against MediaGoblin
====================================

Plugins should be allowed to make sure they work with the current
MediaGoblin and barf if otherwise.

Question though: what about working with old plugins in devmode for the
new release?


==========
Unit tests
==========

Some way or another we might want the ability to run unit tests for
certain plugins and not others.  What's the nicest way to do this?

See also: http://issues.mediagoblin.org/ticket/614


============================
Deciding where to put hooks!
============================

So we are putting hooks all over.  But where?  We should try to think
intelligently about what parts need hooks where!


==================================
The changing MediaGoblin landscape
==================================

Relatedly, MediaGoblin will change over time, and so our "hook API"
won't be stable-ish for some time.  I think we should make that clear.
Eventually we can take a different stance on this, maybe around 1.0?


======================
Things we already have
======================

 - callable hooks
 - ability to add new or override templates
 - template hooks for extending sections
 - database support