Re: [GMG-Devel] Plugins plans

[Christopher Allan Webber]

Heya all,

I'm keeping the email I sent with the big plugins plans stuff below.
But since writing this, I've had a conversation with Will Kahn-Greene
about what he implemented for plugins, as well as what I had written
below.  It turns out that Will had actually accounted for much of the
stuff below, but in a way I hadn't understood correctly at the time.

With Will's permission, I am going to actually just copy his email into
here:

> I re-read the plugin system design document I wrote. Then I read your 
> hook email.
> 
> Two things that jump out at me:
> 
> 1. Your hook email seems pretty divergent from the plugin design 
> document. You're not using the same language and basic units. That's 
> fine, but it makes it hard to advise you.
> 
> 2. Seems like you're focusing on structures of things rather than 
> operations. The hook system I was talking is all functional programming. 
> It's essentially map, filter, first, ... All of which operate on a list 
> of functions that are registered with the hook. The hook itself is just 
> a name and a list of functions.
> 
> In the design document, I talk about three operations we need to be able 
> to do:
> 
> * notification: every function in the hook list gets called with the 
> same arguments
> 
> * transform: the output of the previous function is the input of the 
> next function in the hook list--the final result is used
> 
> * handler: the functions in the list get called with the same arguments 
> until one handles things and returns a value that indicates it's handled 
> it. You could do this by throwing exceptions, but it's cheaper to just 
> check a return value.
> 
> Those three kinds of operations (and a variety of others) were 
> implemented in Pyblosxom in a single function that took different 
> operator arguments:
> 
> https://github.com/pyblosxom/pyblosxom/blob/master/Pyblosxom/tools.py#L745
> 
> Here's an example of usage:
> 
> https://github.com/pyblosxom/pyblosxom/blob/master/Pyblosxom/tools.py#L636
> 
> I think that gets you pretty far and covers a lot of the things in your 
> email.
> 
> The metadecorator hook sounds great, but it seems like it's a callback 
> chain implicitly structured in closures.
> 
> I think that covers everything I'm thinking. Hope that helps!
> 
> /will

The reason I think things got so out of sync is I was operating under
the assumption that the dictionary which stored functions in the plugin
api was mostly a dicitonary for storing a specific set of things:
callable functions.  Having viewed it as a "thing to store things", I
then started thinking about "different buckets to store things for
different components of MediaGoblin", and utilities that would act
differently for storing things in those different buckets.

But in the universe Will laid out, we aren't storing specific things,
we're always storing functions, even when those functions are themselves
just wrapping around returning a thing.  This means we have one simple
API that can do pretty much everything.  And if you look at the two
links Will put above:
 - https://github.com/pyblosxom/pyblosxom/blob/master/Pyblosxom/tools.py#L745
 - https://github.com/pyblosxom/pyblosxom/blob/master/Pyblosxom/tools.py#L636

You'll see that just by providing different functions to specify
behavior we can implement pretty much most if not all the below.  And
that's really compelling to me.

But that still leaves some questions:
 - What about namespacing?  I guess I am still somewhat in the bucket
   era of thinking, but it felt to me like storing things that are
   basically function calls in the same place of things as interfaces or
   form modifications seems somewhat surprising to me.  I suppose
   if hooks are super generic and can do basically anything, namespacing
   might not really be needed and might be arbitrary.  I'm not sure.
 - I think that maybe the right route forward is to take this approach,
   and then also try to provide as many "assistants" or utilities to the
   most common patterns: pulling down or installing an interface hook,
   modifying a form, etc?

Hm, are there other questions?

I think I am going to leave template hooks as the exception they are
currently to this design, though maybe will refactor them
later... that's the one space where I feel like a separate namespacing
is really needed for sure.  Plus that code works.

In the meanwhile, I guess I should start implementing the below with the
vision of the above?  One thing's for sure... I need to get moving fast
given the next release's timeline!

Thanks for tolerating yet another braindump,
 - Chris


Christopher Allan Webber writes:

> Hi all,
>
> So I did even further thinking on this based on what we said on this
> thread and also looking at the previous plugin design document:
>
> http://wiki.mediagoblin.org/PluginSystemDesignDocument
>
> Reading that, as well as the rest of the thread, probably gives better
> context for the rest of the things I'm writing here.  It looks like we
> need the following things still:
>
>
> ====================================================
> Hooks that loop all the way through, hooks that halt
> ====================================================
>
> We can handle notification and transform via the "callable hooks"
> system, but we don't have a "handler" that is the last one to get it
> and then things stop.
>
> One way to handle this: have a series of method calling handlers
> that do things in different ways, like:
>
>  - callable_hook_runone: runs them all till they stop
>    - which order?  Does the last or the first get to claim it?
>  - callable_hook_runall: runs every one of the hooks
>    - likewise... which order?
>
> .. both of those would expect *args, **kwargs and pass those keyword
> arguments onto the hooks.
>
> It might also be possible to make some callable classes like this,
> like:
>
>    callable_hook = CallableHookRunone(
>       'some_hookname', accrue=True)  # accrues the results into a list
>    results = callable_hook(arg1, arg2, kwarg="foo")
>
>
> ======================
> "meta-decorator hooks"
> ======================
>
> This is an incredibly evil and flexible idea that, like decorators
> itself, is probably elegant, hard to understand, powerful, and yes, a
> little bit evil.
>
> So this is an idea inspired from emacs lisp, which allows you to pass
> in functions to wrap a function that already exists.  Actually, hooks
> in emacs look a heck of a lot like decorators, so actually...
>
> Say you have this function:
>
>   @meta_decorator_hook('some_function_hook')
>   def some_function(foo):
>      bla bla
>
> Now, you want to wrap this function!  But how do you wrap it?  It's
> already defined, you can't just wrap arbitrary things around it!
> ... or can you?
>
> In this meta_decorator_hook, the meta_decorator_hook returns a
> lazy-loaded method set to actually construct a chain of wrapped
> methods.  So if we did something like:
>
>   pluginapi.wrap_function('some_function_hook', our_function)
>
> .. it would return, basically a WrappedFunction() that, the first
> time it is executed, walks through all the decorator like methods
> that have been pushed onto that hook and wraps them in each other.
> That meta-decorated method is cached, and eventually, executed.
>
> Sounds evil, right?  :)
>
> This could be super powerful though.  You could do crazy things like
> double-check the results of a view before returning it, or totally
> override the view and decide not to call it at all for your plugin,
> call it both before and after, or set up special-case exception
> handling for hooks that are called within the function itself.
>
> I'm sure I didn't make that sound any less evil, but I think it'd be
> really great.
>
>
> ============================================================
> How do we "halt" a function if a plugin needs to tell us to?
> ============================================================
>
> Exceptions probably... but how to make this clean?
>
> Okay, let me explain a bit more.  Say someone implements a special
> hook that sets up something special (I'm having a hard time thinking
> of what) but notices that something is terribly wrong.  It needs to
> immediately halt the execution of the function.  A reasonable thing
> to do is raise an exception.  But we don't just want it to show up as
> a programming error and hit the logs.  How to handle it?  One of two
> ways:
>
> We could already be expecting certain types of exceptions to possibly
> be raised like the following:
>
>   try:
>      pluginapi.callable_hook_runall('some_hook', arg, blarg)
>   except ExpectingThisException:
>      something something
>      return Response('oh snap')
>
> Okay, but what if the view designers weren't able to anticipate the
> kind of issue you have... how to handle this?
>
> The answer is the meta-decorators solution above I think.  In your
> own meta-decorator around the view that calls the hook above, you
> could set yourself up to catch the exception.
>
> Tricky, but should work! :)
>
>
> ============================
> WTForms form extension hooks
> ============================
>
> This has been discussed previously.  But basically I wonder if it's
> possible to do something like this:
>
>   class BlaBlaForm(pluginapi.HookForm):
>     _hook_name = 'blabla'
>     bla_bla = wtforms.BlaBlaField()
>
> This would require some fancy metaclassing but if it worked, it
> should be able to allow to attach fields to the form either before or
> after like the following:
>
>   pluginapi.add_form_field(
>      wtforms.SomeFormField(),
>      position=before)
>
> Is this possible?  Not totally sure.  Would be great if it did.
> Would also involve evil metaclassing.  But what part of plugin
> systems don't involve a bit of evil ;)
>
>
> ==============================
> Interface implementation hooks
> ==============================
>
> Sometimes we want people to supply an alternate interface to "swap out"
> certain things.  For example, they might want a totally different user
> authentication system, and maybe we'd have some sort of user
> authentication api.
>
> Altenately, the storage interface is a good example.
>
> In this method we set up a minimal interface and a default
> implementation.  Users then have the option to swap out this
> implementation.
>
> I think the way storage interfaces look are already currently pretty
> good, so swapping out the interface could probably be like:
>
>   pluginapi.set_component('public_storage', BagOfHolding())
>
>
> =============================
> Static files and staticdirect
> =============================
>
> Plugins need a way to add their own static files, like CSS, images, etc.
> They also need ways for them to be linked and statically served.  Then
> they also need a way to include those files in templates.
>
> The docs make some mention of this... the assetlink command we have can
> be extended for this, and I think we can both extend staticdirect to
> take a "domain" like:
>
>   {{ request.staticdirect('/images/foo.jpg', domain='plugin:someplugin') }}
>   {{ request.staticdirect('/images/foo.jpg', domain='theme') }}
>
> This could also have fallback support.
>
>
> ================================
> Namespacing the hook repository?
> ================================
>
> We already have callable hooks and we have template hooks.  I wonder if
> we should provide a generic API for fetching and storing them?  I guess
> maybe this won't work with all hook types... not sure.
>
>
> ====================================
> Version checking against MediaGoblin
> ====================================
>
> Plugins should be allowed to make sure they work with the current
> MediaGoblin and barf if otherwise.
>
> Question though: what about working with old plugins in devmode for the
> new release?
>
>
> ==========
> Unit tests
> ==========
>
> Some way or another we might want the ability to run unit tests for
> certain plugins and not others.  What's the nicest way to do this?
>
> See also: http://issues.mediagoblin.org/ticket/614
>
>
> ============================
> Deciding where to put hooks!
> ============================
>
> So we are putting hooks all over.  But where?  We should try to think
> intelligently about what parts need hooks where!
>
>
> ==================================
> The changing MediaGoblin landscape
> ==================================
>
> Relatedly, MediaGoblin will change over time, and so our "hook API"
> won't be stable-ish for some time.  I think we should make that clear.
> Eventually we can take a different stance on this, maybe around 1.0?
>
>
> ======================
> Things we already have
> ======================
>
>  - callable hooks
>  - ability to add new or override templates
>  - template hooks for extending sections
>  - database support