Re: Packages/modules

[Urs Liska]

Am Montag, den 20.01.2020, 23:45 +0100 schrieb David Kastrup:
> Urs Liska <address@hidden> writes:
> 
> > OK. The *current* behaviour of oll-core is:
> > 
> > * loaded packages and modules (let's for now keep the existing
> > names)
> >   are accounted for in an alist.
> > * if the requested package/module is already loaded:
> >   * if options are passed, try setting them (overriding defaults
> >     and/or values set by an earlier loading)
> >     => This behaviour has to be discussed
> 
> Sounds to me like overriding defaults is perfectly reasonable (else
> specifying options wouldn't work at all) 

Yes, of course that's the idea behind it.

> and overriding values set by an
> earlier loading should flag an error.

That's probably right. Sometimes giving users too much controls and to
many alternatives makes the interface convoluted and as a result
confusing.

> 
> Maybe packages should have two ways of overriding a default: set a
> default and require a value.
> 
> A request in a user document is always treated as a requirement, two
> packages setting defaults leads to an error (or a warning and a
> revert
> to the original default?) unless some package (or the user) requires
> a
> particular setting which takes priority.  And of course different
> requirements cause an error.
> 
> Or is this too contrived, and only requirements should be allowed?

I don't find this too contrived in general, but there's one thing where
you may be too generic in the description.

There are two basic use cases for overriding default values: Loading a
package with options and setting an option explicitly after loading the
package. The other complication is in packages implicitly loading other
packages.

The simple case is when a user explicitly sets an option with (current
syntax) \setOption package.module.some.option-leaf value. This can
happen to change the global behaviour or to change something at some
point in the score/music expression (depending on how the value is
used). This is what you called a requirement and should always be
respected. If *that* triggers an error or produces unwanted results
it's clearly the user's responsibility.

The other case is when a package depends on another one and loads it,
which may result in the depended-on package being loaded more than
once, possibly with conflicting options.

Say you have a helper package [A] providing a data structure for some
use case (for example there is the `breaks` package that provides some
alists and accessor functionality to maintain sets of line/page
breaks).
Such a package may expose an option with a default value (currently the
syntax is \registerOption package.option.path default-value (specifying
types is on my wish-list)).
A client package [B] may just load that while a client package [C]
might load it overriding the value, for example because it wants [A] to
work in a specific mode. Another client package [D] may override the
option with a different value or pass the choice along to the user.
This is where conflicts may occur, and where it is not in our reach to
know at design time (designing the system, i.e. now) whether these
conflicts will be harmless or not. So erroring may be the safest
choice. Actually that would be the situation in LaTeX when you have to
know that two packages "don't work well" together.

After having written this I have the impression that the spot to
address your distinction between "setting" defaults and "requiring"
values would be:

 * A package "knows" whether a current value is at its default value or
   not (either with a "changed" flag or by on-the-fly comparison)
 * When to the *loading* of a package an option is passed that has
   already been changed it is considered illegal, and a warning or
   error is raised (to be decided, probably an error).
 * When an option is modified explicitly from user code it is simply
   done.

I will raise the issue of how options are handled and stored in a
separate thread, we should keep that out of the current discussion.

> >>> ...
> 
> > > And of course Guile has modules.  Do we already have a thought
> > > about
> > > how
> > > to relate to the module system?  
> > 
> > Yes, that's right! \loadModule is definitely a bad name.
> 
> I was not as much worrying about the name, actually, 

OK, but still that name should be discarded.

> but about the
> semantics.  Seems to me like defaulting to a separate module might be
> a
> reasonable thing.  It would require exporting whatever you want to
> use
> from outside the module.

OK, I *think* I see. I'll comment on that together with the other stuff
at the end below.

> 
> > In LilyPond there's a fundamental difference between \include and 
> > #(use-modules, which is not the case in oll-core. There "modules"
> > are
> > essentially the same as packages, just used to organize packages in
> > a
> > hierarchical fashion:
> > 
> > * scholarLY includes modules:
> >   * annotate
> >   * choice
> >   * staff-cancellation
> >   * ...
> > * snippets can be addressed like
> >   \loadModule oll-misc.layout.horizontal-spacing
> > 
> > Actually it would make transition smoother if we choose completely
> > new
> > names for the functions.
> > 
> > So:
> > * I suggest not to discern between "use/load" and "require",
> >   just have one command that behaves like require.
> > * (caveat: handling of config options when given to a
> >   secondary call)
> > * Is the "\load" prefix the best choice?
> >   * \loadPackage
> >   * \usepackage
> >   * \use
> >   * \package
> >   ?
> 
> LaTeX has \usepackage and \RequirePackage either of which don't
> really
> match LilyPond naming conventions.  A bit of a pity: I'd have opted
> for
> \usepackage otherwise.
> 
> > * Do we need a word for the (current) "module" at all?
> >   What about
> >   \use scholarly.annotate
> >   or
> >   \use \with { subs = annotate.choice } scholarly
> >   ?
> > 

I also like \usepackage, and having different semantics from LaTeX is
not necessarily a showstopper if it's properly communicated. Every
programming language has its own approach to their module systems, and
that's OK. One probably doesn't like the opposite behaviour of Python
and Lua with regard to global scopes, and one typically stumbles over
it when switching back and forth. But that's not something you'd blame
the languages for.

I suggested \use because it doesn't discriminate between packages and
subpackages (or "modules"), but it *feels* a little bit too generic as
a name. OTOH it may not be a problem to refer to both items as packages
and use the hierarchy only as an organizational means:

 * scholarly is a top-level package, which is stored in its own Git
   repository
 * it includes a number of components (which might be a good label for
   the concept), such as annotate, choice, or source.
 * one can load/use/whatever scholarly as a package (with or without a
   selection of its components) or any component directly.
 * The differences between loading scholarly or scholarly.annotate are:
   
    * loading scholarly *may* include components or not
    * loading scholarly.annotate will implicitly load scholarly's code
      if it hasn't been loaded already.
 * But I think these differences can be hidden from the user, simply by
   determining if we're addressing a symbol (or single-element symbol-
   list) or a list with more than one element.

> > 
> > > With regard to namespaces, there may be
> > > something to be said for requiring explicit export in the long
> > > run?
> > > 
> > 
> > Although I know this is important I don't feel comfortable having
> > an
> > opinion about this type of issue.
> 
> Ok.  One thing to think about is that we want package files to be
> contributed by "ordinary" users.  But something like
> 
> \exportSymbols transposeSequence,instrumentGroup,scratchMyBack
> 
> would be perfectly feasible syntactical sugar.
> 

I'll be more verbose than probably necessary, just to make sure we're
talking about the same thing.

Given a compiled file
  
  document.ly

which \include-s

  library.ily
  styles.ily

any variables defined in library.ily are visible in document.ly, and
also in styles.ily. So library.ily makes functionality available to all
LilyPond code that is parsed later, right?

A Scheme module in

  scheme-lib.scm

which is included in library.ily through #(use-modules ...) will only
see variables from scheme-lib.scm which have been explicitly exported
there. And these variables are not visible in document.ly or
styles.ily, right?

So the question at hand is how scholarly/package.ily should behave when
loaded through \usepackage or whatever we'll choose to name it.

In the current implementation openLilyLib packages behave just like
regular \include-d LilyPond files, i.e. their variables are exposed to
all later code.

If I got you right then from my experience with openLilyLib and
creating project environments (which basically are the same as
packages), I would say:

 * I'm all for hiding names in packages by default and having to
   explicitly expose/export the package's interface
 * But unlike Scheme modules that exposed interface should be publicly
   visible to later code:
    * library.ily loads \usepackage scholarly.annotate
    * scholarly.annotate exposes \criticalRemark
    * I *think* I want \criticalRemark to be available everywhere
      without having to do \usepackage scholarly.annotate in every file
      where I want to use it.
 * One could argue that it would be cleaner to explicitly state a
   package's use at the top of each file (like e.g. Python imports).
 * OTOH having it public is how LilyPond was always used: Creating a
   variable/tool just makes it available for all.

I'm leaning to one side, but I'm not too sure about it and can be
convinced by arguments. However:

 * I'd assume that making \usepackage behave differently than the
   existing mechanisms of file includes (currently ly:parser-parse-
   string is used in 
   
https://github.com/openlilylib/oll-core/blob/master/scheme/oll-core/internal/file-handling.scm#L44-L51
   ) would require work on the parser side that I can't do?
 * If we go for the approach of having the user explicitly use a
   package in each file, we should discuss if that change shouldn't be
   considered as a *general* change to scoping in order to have a
   cleaner approach to namespacing.
 * BUT: this would of course affect *any* existing score and can *not*
   be fixed by a convert-ly rule (how should convert-ly know where any
   used variable or command name is defined? We can even look up a
   parser-variable whose name is generated on-the-fly, so no chance for
   convert-ly here ...)
 * The very least that would be necessary here (from a user's
   perspective) would be a switch (like e.g. for point-and-click) to
   preserve the previous behaviour. However, that is surely an
   unreasonable effort to support in parallel. Plus: While it would
   probably be OK to do something in 2.21.1 or so, i.e. at the very
   beginning of a new devel cycle, there will eventually come the
   moment when "stable" users have to switch from 2.20 to 2.22.
 * From that perspective: Maybe it wouldn't be a bad idea to enforce
   explicit import, but not for regular files, only for the new concept
   of packages. (well, what about \import, or even \from scholarly
   \import annotate,choice,source ?)

Urs