Re: Dynamic loading progress

[Daniel Colascione]

On 02/16/2015 11:29 AM, Eli Zaretskii wrote:
>> Date: Mon, 16 Feb 2015 10:22:18 -0800
>> From: Daniel Colascione <address@hidden>
>> CC: address@hidden, address@hidden
>>
>>>> The `size' member tells modules how long the emacs_runtime structure
>>>> is. (It's better to use size than an explicit version field: this way,
>>>> .size = sizeof(struct emacs_runtime) is always correct.)
>>>
>>> This approach requires us to change the size each time we change the
>>> layout, even if the change itself leaves the size intact.  Using a
>>> version field doesn't have this disadvantage.
>>
>> Emacs is allowed to pass a new version of the runtime and context
>> structures to modules expecting old versions. That can only work if the
>> structures are append-only, so `size' is adequate here.
> 
> I understand that, but appending members does not necessarily increase
> the size of the struct, due to alignment, bit fields, etc.

If it's just a table of function pointers and everything has natural
alignment, adding members will increase the size.

>>> You say "we don't
>>> lock ourselves into conservative stack-scanning GC", which I interpret
>>> as saying you don't want to rely on stack scanning to avoid a
>>> destructive GC in this case.  But if we don't rely on that, where's
>>> the guarantee that such emacs_value will survive GC?
>>
>> Emacs stores emacs_value values in a local reference table before
>> handing them to module code.
> 
> That's something that wasn't in your original writeup.  Sounds like a
> significant part of the design.

I thought I'd made that clear. I'm glad I've been able to clarify it.
Stefan doesn't seem to think the indirection is necessary, but I'm not
sure I agree with him, especially because his scheme has a space and
speed cost on 32-bit platforms.

>>>> We'll represent all Lisp values as an opaque pointer typedef
>>>> emacs_value.
>>>
>>> This doesn't play well with --with-wide-int, where a value can be
>>> wider than a pointer.  I think we should instead go with intmax_t or
>>> inptr_t, whichever is wider on the host.
>>
>> emacs_value objects are not literally Lisp_Object values. They're
>> indirected through a reference table (either local or global) so that we
>> can GC them. A pointer value can address all possible memory locations,
>> so it's fine here. (In the wide-int case, the pointed-to table entries
>> are wider than pointers, but modules don't need to know that.)
> 
> How this will work when some Lisp calls a primitive implemented by a
> module.  From the Lisp interpreter POV, it's just a call to a
> primitive, so how will this special handling come into existence?
> (And why? just to save us from picking up a wider data type?)

I think there's some confusion about how function calls work.

From the Lisp interpreter perspective, we're calling a regular Lisp
function. That Lisp function closes over the information needed to call
a module function and then calls into an Emacs core function that
actually makes the module call.

Say a module wants to provide a function "frobricate" taking two
arguments. It defines a C-level function (call it do_frobricate) with a
signature matching emacs_subr. It then calls `make_function' like this:

  emacs_value frobricate = env->make_function(env, 2, 2, do_frobricate)
  if (frobricate == NULL) { FAILURE(); }

Now `frobricate' is a Lisp value usable in Lisp funcall; if we bind
this value to the function slot of some symbol, Lisp code can call
frobricate like any other function.

Say do_frobricate has address 0xA3B41234. Implementation-wise, the value
`frobricate` that `make_function' returned is a Lisp closure that
essentially looks like this:

  (lambda (&rest args) (%do-module-call frobricate-module 0xA3B41234 2 2
args))

%do-module-call then clears the pending-error information, packs its
arguments into a C array, and calls 0xA3B41234 like this:

  emacs_subr module_function = (module_function) 0xA3B41234;
  result = module_function(some_env, args, nargs);

(I've left out the part where we transform emacs_value to Lisp_Object
and vice versa.)

>>> What about the doc string?
>>
>> We'll set that at the lisp level.
> 
> I'm not following: what Lisp level?  What I had in mind was a module
> that implements a primitive.  Are you saying that every primitive a
> module wants to implement will need a Lisp part and a C part?  That
> sounds cumbersome, just to save us from passing yet another argument
> to make_function.

No: that's silly. I'm saying that creating a globally-named function in
Emacs has two parts: 1) create the actual _function object_. 2) bind
that function object to a name by setting some symbol's function slot to
that function object. We can do step #2 entirely in Lisp code that the
module calls, and we can set a documentation property at this time. It's
only step #1 that requires a magical incantation.

I've only described the magical, ABI-relevant bits of the interface. In
order to make a module actually usable, we'll need glue, but since we
can implement this glue in Lisp and call it from module code using the
narrow interface I've described here, I'm not worried about it.

>>>>     emacs_value (*funcall)(
>>>>       emacs_env* env,
>>>>       emacs_value function,
>>>>       int nargs,
>>>>       emacs_value args[]);
>>>
>>> Shouldn't funcall use emacs_subr?
>>
>> Why?
> 
> Because otherwise I see no point in having emacs_subr, it's almost
> unused.

emacs_subr is a typedef for a pointer to a module-provided function.
Every module-provided function is an emacs_subr.

>>>> If Lisp signals or throws, `funcall' returns NULL.
>>>
>>> I suggest some other value or indication of that.  NULL is a valid
>>> return value, so usurping it for errors might be too harsh.
>>
>> No it isn't. Qnil is distinct from NULL in this model because
>> emacs_value is not a Lisp_Object in disguise. Qnil is not special here.
> 
> I wasn't thinking about Qnil.  I simply don't see why it is a good
> idea to forbid funcall from returning NULL in a normal case.

We're not preventing functions from returning NULL in the general case.
We're deciding that NULL is not a valid value of emacs_value type. This
property makes NULL a good candidate for an error sentinel return value
for functions that would otherwise return valid emacs_value values.

>>>> `intern' also does the obvious thing.
>>>
>>> Do we need 'unintern' as well?
>>
>> Modules can call unintern through Lisp.
> 
> Then why provide 'intern'?  It, too, can be called from Lisp.

Modules have no way other than `intern' of getting an emacs_value that
corresponds to a known symbol. They can call `unintern' by first
interning the symbol "unintern", then calling it.

>>>>     emacs_value (*type_of)(
>>>>       emacs_env* env,
>>>>       emacs_value value);
>>>>
>>>> Like Lisp type-of: returns a symbol.
>>>
>>> What is a "symbol", from the module's C code POV?
>>
>> It's an emacs_value. You can compare symbols (by calling `eq'), call
>> them as functions, or use them as function arguments. That's sufficient.
>>
>> Come to think of it, though, we do need a C-level `eq'.
> 
> Why not simply return a C enumeration type?

Why make the set of elisp primitive types part of the ABI? We already
have a way to refer to symbols. There's no need to an enum, which would
just be a redundant way of naming types.

>>>  . direct access to buffer text (using buffer-substring means consing
>>>    a lot of strings)
>>
>> Is that really a problem for the use cases we want to support?
> 
> It could be a problem for a module that processes buffer text.
> 
>>>  . not sure how will a module "provide" its feature
>>
>> It doesn't. You don't require a module: you load it. Most modules will
>> come with a small Lisp wrapper, just like Python modules.
> 
> So do we _require_ a Lisp wrapper?  Otherwise, how would Emacs know
> whether the module is loaded or not?

By path, I imagine. Why don't you have this problem with (load
"myfile.el")? I'm not opposed to hooking modules into the
provide-and-require machinery, but I don't see a particular need either.

>>> One thing that's inconvenient is the need to drag the environment
>>> pointer through all the calls.  Why exactly is that needed?
>>
>> Modules have no other way of accessing Emacs internals. Dragging one
>> parameter through the system isn't that bad and lets us sanely track
>> which module is doing what.
> 
> The environment must be known to the module, that's for sure.  But why
> does it have to be plugged into every call to every interface
> function?

How else are the interface functions supposed to know the module for
which they're being called?

>>> Also, the buffer returned by find-file when it returns normally is
>>> lost here, isn't it?
>>
>> Sure. It's just a tiny example, and the returned buffer itself is still
>> GCed.
> 
> That's not what bothered me.  I don't understand how will the module
> receive that return value in the first place.  An additional argument
> to funcall, perhaps?

funcall returns an emacs_value corresponding to the value the called
function returned. Here's an alternate implementation of emacs_find_file:

emacs_value
emacs_find_file(emacs_env* env, const char* filename)
{
  emacs_value e_filename = env->make_string(env, filename);
  if(env->error_check(env)) return NULL;
  emacs_value e_find_file = env->intern(env, "find-file");
  if(env->error_check(env)) return NULL;
  return env->funcall(env, e_find_file, &e_filename, 1);
}

Now emacs_find_file returns NULL on error (with error information set in
the environment) and an emacs_value on success. The returned value
corresponds to whatever find-file returned, so in this case, it's a buffer.

signature.asc
Description: OpenPGP digital signature