Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Hi,

On 5 April 2012 23:27, David Boyce <address@hidden> wrote:
> A few years ago I suggested a plugin architecture much like this (but
> I didn't supply a patch - crucial difference), to allow a plugin to
> make the "up-to-date" determination, replacing the hardwired timestamp
> system. That could in theory be quite useful as it would allow an MD5
> or similar signature to be used without major mods to make internals
> which is what's defeated such attempts in the past. Any idea how hard
> it would be to handle this with a plugin?

This is a serious issue indeed - I've wanted md5 timestamps for ages.
I think this is much more complicated than adding functions though.

I don't know how make evaluates prerequisites in detail and if I
naively stuck an md5 check in place of a call to stat() for a check
then the file might get read and stamped many times repeatedly which
could be very slow.

It also might be great until I was doing a check on a 650MB iso or
something like that.

So for a feature like this I'd be looking at heuristics and
complicated tricks to avoid the pitfalls and I'm actually fairly sure
it would be easier to do this without plugins first and then
generalise it afterwards to allow people to choose different checksum
algorithms and to set the tradeoffs about what kinds of files to use
each method on etc - but I'd do that after I had written a hardcoded
version and understood all the problems.

> On an unrelated note, how would you anticipate plugins being loaded in
> real life? I think expecting  people tp type "make
> --plugin=tests/plugins/test-plugin.so ..." is too unwieldy for daily
> use. Environment variable? Auto-discovery of plugins in a conventional
> location? Is it possible a .PLUGIN special target could be embedded in
> a makefile that requires a given plugin, or would that be too late?
> Maybe a MAKE_PLUGIN_PATH env var?

:-) that's a good point. I tend to use it from a script and I hadn't
thought about the convenience of it. Good suggestion for an
environment variable.

I am initialising the plugins quite early but I don't think that it's
critical to do so - hence adding syntax is quite possible.  I was
trying to avoid doing that because it's always so controversial but
since Paul already has a plan for syntax...

Regards,

Tim

> -David Boyce
>
> On Thu, Apr 5, 2012 at 1:12 PM, Tim Murphy <address@hidden> wrote:
>> Hi,
>>
>> On 5 April 2012 20:52, Lawrence Ibarria <address@hidden> wrote:
>>> I like this idea quite a bit.
>>>
>>> I see this as still work in progress to define what type of functions the 
>>> plugins can have.
>>> Maybe they can even create or change make variables.
>>
>> At this point, plugins are naughty and are kind of able to access
>> anything in make that they have the prototype for (using -rdynamic to
>> make all the symbols available).  This is definitely "dodgy" since
>> there's nothing stopping a plugin from trying to access something that
>> might be intended as fairly volatile within make.
>>
>>> In the case of variable_buffer_output, I'd suggest that each plugin has an 
>>> 'initialization function'. This is a handshake where make provides the 
>>> plugin with function pointers (avoids the extern function) and the plugin 
>>> can provide its name, versions, and the function table. That simplifies the 
>>> code that right now tries to get every single variable from the dl_handle.
>>>
>>> -- Lawrence
>>
>> Ok, good idea - I'll add something to look for an initialiser and call
>> it and as you say, pass it a table. Returning values is critical and I
>> think that looking up variable values at the least is also probably
>> important.  Another thing I've wanted for ages is to be able to know
>> if some target was declared already - I want plugins to have the
>> ability to look at make's target database if possible although not
>> necessarily modify it.
>>
>> Regards,
>>
>> Tim
>>
>>
>>> -----Original Message-----
>>> From: address@hidden [mailto:address@hidden On Behalf Of Tim Murphy
>>> Sent: Thursday, April 05, 2012 2:51 AM
>>> To: address@hidden
>>> Subject: Patch to allow make to load plugins that add new functions.
>>>
>>> Hi,
>>>
>>> I am between jobs which made me realise that I am absolutely free to
>>> contribute to make for about 10 days :-)
>>>
>>> The one thing I have wanted the most and the longest is a way to add
>>> new functions without having to rebuild and look after a custom
>>> version of make.  Essentially this should allow people to extend make
>>> without necessarily having to maintain a custom version of it.
>>>
>>> So here is an attempt at a plugin system for your interest. Currently
>>> it ought to work on systems with dlopen.  LoadLibrary on windows
>>> should do the job admirably but I haven't had time to have a go at
>>> that yet.  So the feature is optional (configure --with-plugins).  I
>>> am looking for feedback indicating interest or disagreement or
>>> whatever and mostly I just want to put some code into the open so that
>>> it's there.
>>>
>>> My test plugin adds the $(equal ($X),$(Y)) function which returns $(X)
>>> if the variables X and Y match as strings and returns the empty string
>>> if not. I have often wanted this for use with $(if) and have a
>>> horrible and inefficient macro for doing it now but it's something I
>>> feel really ought to be built in.
>>>
>>> Here's how you tell make to use a plugin:
>>>
>>> ./make --plugin=tests/plugins/test-plugin.so -f tests/plugins/plugintest1.mk
>>>
>>>
>>> Here's the code for this simple test plugin:
>>>
>>> #include "plugin.h"
>>> #include <string.h>
>>>
>>> /* The next line is a cheat  to save from having to include all the
>>> make headers and possibly end up linking to all sorts of make object
>>> files which defeats the purpose.
>>>   Don't like it and  will have to think of something better: */
>>> extern char *variable_buffer_output (char *ptr, const char *string,
>>> unsigned int length);
>>>
>>> int gnumake_plugin_api_version = 1; /* what api the plugin expects
>>> make to provide - just a way for make to know if it's loading a plugin
>>> that should work  */
>>>
>>> int gnumake_plugin_major_version = 1; /* The version of this plugin -
>>> might be useful for problem reports */
>>> char gnumake_plugin_name[] = "test_plugin";  /* also the name of the
>>> feature added to .FEATURES */
>>>
>>> /* $(equal astring1,astring2) returns blank, $(equal
>>> astring1,astring1) returns "astring1"
>>>  * The purpose of this function is to allow equality tests in $(if)
>>> functions e.g. $(if $(equal $(X),$(Y)),<something>,<else>)
>>>  * */
>>> static char *function_equal(char *o, char **argv, const char *funcname)
>>> {
>>>  if (argv[0] && argv[1] && strcmp(argv[0],argv[1]) == 0)
>>>    {
>>>          o = variable_buffer_output (o, argv[0], strlen (argv[0]));
>>>    }
>>>  else
>>>    {
>>>          o = variable_buffer_output (o, "", 0);
>>>    }
>>>
>>> return o;
>>> }
>>>
>>> padded_plugin_entry gnumake_plugin_function_table[] = {
>>>        {"equal", 2, 2, 1, function_equal}, /* 2 args, expand them first */
>>>             {"", 0, 0, 0, (void *)0}
>>> };
>>>
>>> /* end ---------------------- */
>>>
>>>
>>>
>>> So a plugin is fairly easy to write and it's not hard to build either.
>>>
>>> I have tried to set it all up with autoconf and so on but I'm not an
>>> expert at that.  There is also nothing stopping you from loading the
>>> same plugin twice at the moment and it needs more tests and some
>>> documentation so I realise that this patch is not right yet.
>>>
>>> There is one set of modifications to main.c to add calls to
>>> load_plugin() which is in a new file, manage-plugins.c, and there is a
>>> new commandline option ("--plugin=") which can be used multiple times.
>>>
>>> That's it!
>>>
>>> Regards,
>>>
>>> Tim
>>>
>>> --
>>> You could help some brave and decent people to have access to
>>> uncensored news by making a donation at:
>>>
>>> http://www.thezimbabwean.co.uk/friends/
>>> -----------------------------------------------------------------------------------
>>> This email message is for the sole use of the intended recipient(s) and may 
>>> contain
>>> confidential information.  Any unauthorized review, use, disclosure or 
>>> distribution
>>> is prohibited.  If you are not the intended recipient, please contact the 
>>> sender by
>>> reply email and destroy all copies of the original message.
>>> -----------------------------------------------------------------------------------
>>
>>
>>
>> --
>> You could help some brave and decent people to have access to
>> uncensored news by making a donation at:
>>
>> http://www.thezimbabwean.co.uk/friends/
>>
>> _______________________________________________
>> Bug-make mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/bug-make



-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/