Re: Do we really offer the future?

[Gilles]

Hello.

On Thu, 23 Apr 2015 12:09:29 +0200, Urs Liska wrote:
> Hi all,
>
> I had to leave this alone for a while, otherwise I wouldn't have been
> able to reply calmly to a number of the first responses on this
> thread. It's only lately that the discussion has reached a level of
> constructivity that may be helpful. In the meantime the discussion 
> has
> split up to a number of threads and sub-topics, all going in the
> direction of how to improve LilyPond, and I'm glad this came up. 
> Let's
> hope it will eventually trigger some real improvements...
>
> However, I think I have been grossly misunderstood by some of the
> early commenters, as the invocation of certain magic words (where
> "publishing house", particularly in combination with "big" is one of
> the foremost, ranking directly after "copyright") triggers near
> Pavlovian responses that may not really take into account what was
> originally expressed. As a consequence some of my main points (or
> not-explicitly-named consequences of them) haven't been part of the
> discussion at all, and I think these should be picked up too.
>
> It doesn't make any sense to reply to all of the messages
> individually, so I'll try to do it in a more general reply to my
> initial post.
>
> First of all:
> I have _not_ asked "the LilyPond team" to spend any resources for
> whatever.

First of all, nobody wrote that you did.

> What my post was asking for is a discussion in the _user
> community_.

And we have it; and the "early commenters" explicitly thanked you
starting it.

> Obviously this discussion arose, and in the meantime it
> has reached a fruitful direction.

Perhaps, in part, because _choosing_ a direction implies that
several views have been expressed.

> Second:
> The intention of my post is _not_ to find ways to help publishers do
> their business, to help them increase their profit, to "please" them,
> to sell LilyPond's soul to the "sharks" or whatever some commenters
> wanted to read from it. I must admit it's somewhat disappointing to 
> be
> misunderstood that way,

Perhaps your initial post offered the possibility of misinterpretation.

> and I somehow feel attacked below the belt by
> such comments.

Nobody attacked you.  [Not even if you actually held the idea of
defending "big publishers" profits (which would be your right).]

I'm sorry that you misinterpreted an honest response to your honest
request for comment as a treacherous personal attack.

> ###
>
> > To whom LilyPond should strive to "offer the future"?
> > IMHO, certainly not to the "[...] big house[s] with traditions,
> > regulations and limitations".
>
> > What's for the LilyPond team in spending resources trying to work 
> > around
> > those self-inflicted limitations?
>
> and
>
> > "Can you tell me why we should be interested in helping music 
> > publishers exactly?"
>
> I don't inherently care about the fate of publishing houses. But I
> think there are a few "self-inflicted limitations" in LilyPond's
> attitude that should be overcome to assist LilyPond to survive or at
> least to prevent it from eventually being doomed to insignificance.

I assure you that I understand your frustration (from personal
experience in another project).
As much as I agree with the risk, prediction of doom is not a working
argument (from personal experience in another project...).

From an "external" (i.e. someone not on the "dev" ML) oberver's
viewpoint, LilyPond has been in this state since... its beginnings (?).
Astonishingly, some projects don't die easily. :-)

> The main point here is not about helping publishers do their business
> but (and that is a point that hasn't been discussed at all in this
> thread) to help LilyPond evolving through an increased user and
> developer base.

I do agree with that goal, and I did mention the idea of a project
towards that goal (albeit using another direction that would probably
displease the "big publishers").

> LilyPond's developer base is too thin, to a dramatic extent in my
> opinion. As it stands, the withdrawal of *any* core developer would
> have a noticeable impact, and there are a number of persons whose
> withdrawal (for whatever reason) could trigger an existential crisis.
> LilyPond development is much too slow IMO (I don't know if it has
> decreased or if it was always like this). There are so many brilliant
> ideas around to make LilyPond even better, and there are so many
> complaints around about things that LilyPond should do better to be
> really usable - but much of it just doesn't happen because there are
> too few man-hours available to do something about it. The current
> discussion is a very good example for that, and I don't know if the
> resources are available to do it better this time.

I agree as I said above (that lacking resources is a risk).
I understood your original post as doing more with less, in the sense
that "we" (community) should (in order to solve the problem) work
towards satisfying the "limitations" of "big publishers".
I might have misinterpreted, but you did not give explicit examples
beyond the existence of problems with "big publishers" (or IOW "big
publishers having a problem with LilyPond).

IIUC, your idea is that the user base would grow if LilyPond were used
by big publishers.  This I can perhaps imagine. [Although from what I
read on this list, a "SCORE" program exists that is/was used by
publishers and yet does not seem to have a large user base.]
But I and others highlighted that they might not want to use LilyPond
for the foreseeable future whatever amount of work you put in 
overcoming
their (or LilyPond's) limitations.

> And LilyPond's user base is too small. The results of my recent
> survey make me believe that one cannot wholeheartedly offer LilyPond
> services on professional level, as it simply isn't sufficiently clear
> that there will be enough capable people around for a long time. It 
> is
> a sad irony that you can claim a fundamental superiority of open
> standards and plain text file formats, but in reality this doesn't
> help anyone because you can't rely on the availability of human
> resources.

I agree on the irony.  But is it better in the proprietary world?
The Sibelius case has demonstrated that it is not.
A free software cannot die on the whim of a few individuals. That's
a serious advantage.  But "big publishers" don't want to contribute
to make LilyPond up to their business processes...

> Therefore:
>
>  * Anything that increases the developer base is an asset for 
> LilyPond
>    as a project.

Yes.

>  * Anything that increases the user base raises the chance to 
> increase
>    the developer base.

Yes.

>  * Paid projects, particularly big ones with long-term perspectives, 
> do
>    both of the above.

Yes/Maybe

>  * Such projects are very likely to directly contribute back to 
> LilyPond.

Maybe/Not expected (dixit Kieren, somewhere up the thread).


> ###
>
> The fact that it is practically impossible today to deliver LilyPond
> files to commercial publishers rules out a whole category of 
> potential
> users: (highly professional) music engravers who do (want to and/or
> need to) work for commercial publishers.
>
> I can see that a GNU project is not interested in this kind of work
> and explicitly won't endorse it. And that it doesn't consider "market
> share" a goal to be pursued.

But it did consider it (as viewed from an "external" oberver):
Contributions were requested to allow a highly skilled developer to 
work
on developing LilyPond.
Big publishers, which one might imagine know the existence of a 
software
in their field that produces high quality output, could have stepped in
and request enhancements in whathever direction they may have seen fit.
Did they?

> To some extent I have even accepted that
> as a fact. But I am sure (and partially know) that a lot of LilyPond
> users would be happy to work for commercial publishers - and 
> currently
> they only have the choice to use other programs or just not to get
> these commissions.

Sure, it would be great that LilyPond indirectly creates jobs.

> As long as the GPL explicitly permits the use of GPLed software to
> produce commercial content such use is legitimate, and users pursuing
> that line of work shouldn't be treated like heretics.

They are not, and I did not read that they were treated as such in
this thread.

> However, if people feel strongly about it, perhaps make the
> recommendation that LilyPond be placed under a license that prohibits
> its use for proprietary documents, and the wider community can 
> discuss
> that proposal.

(IANAL)
GPL protects the use and availability of the sotware source.
The license of what it produces is an orthogonal concern.

> Opening the door to the publishing business would open the door for
> existing users to distribute their work in more places. It would open
> the door for said category of potential users to use LilyPond. As
> others have stated the fact that LilyPond is non-existent in the
> publishing business also causes it to remain non-existent in other
> "markets", namely the educational - and I'm talking of schools and
> music schools as well as academic musicology. Which in turn would
> "produce" new users.

A fact, it is.
Its cause is not related to the capacities (existing or potential)
of LilyPond.
One cause is very likely the lack of marketing power (that was the
case for much functionality that existed long before a marketing
power made a "shiny" app that suddenly made it exist in the eyes
of the wider public).

Again, I agree with the goal.  I just have doubts on the means.

> Opening the door to the publishing business would also open the door
> to academic musicology. I think one of the most prominent evolutions
> of the decade is the exploration of digital edition concepts. This is
> also interesting for LilyPond because a) engraving tools compiling
> text are natural choices to process text-encoded content (as is
> generally the case) and b) it is a strong tendency (if not a 
> completed
> transition) to think in terms of free software and content there. (In
> Germany you will only get public money for research projects in the
> "digital" domain if all resulting material is done with open,
> accessible standards.) Currently edition projects and institutes 
> don't
> see using LilyPond as a viable goal because it will require
> substantial work to be able to do that, and because they can only
> judge it from the perspective of their individual project - and for 
> an
> individual project the necessary effort would clearly be
> inappropriate.

That's very concrete and interesting.
Why not explore what it would take to make that effort worth it?

More explicitly: why does a public/research institution find it more
efficient to use non-free rather than free tools?
How did they reach the "not viable" conclusion?
Can we argue against it?

> This is the reason why I see it as an important consideration to
> adopt additional encoding standards like MEI (or even MusicXML) more
> vigorously than LilyPond does so far (I think one could consider
> LilyPond a classical case of a vendor lock-in situation).

No!
Everything (source code and input format) is free and open: a
publisher doesn't even need to ask for permission to develop a
converter, and use it to get nice output from its wealth of its
otherwise encoded music contents.

> ###
>
> It has been brought up that adoption of LilyPond by publishers would
> restrict access to culture by "creating" editor's copyright on music
> that should be in the public domain.
>
> There are (at least) two fundamental flaws in this argument:
> First:
> When a publisher releases a new edition of an old work its legal
> status is in no way affected by the tool used. So you can't take that
> as an argument against endorsing the use of free tools for commercial
> products. However, when such an edition runs out of copyright, or the
> publishers should change their minds the editions are already
> accessible in open formats and for free tools. So using LilyPond for
> commercial editions can be seen as an (albeit hypothetical) 
> advantage.

?

I think that you misunderstood my argument.
I _never_ said or implied that free software should not be used to
create copyrighted editions.

Quite the contrary, I of course agree that having _any_ edition use
LilyPond for engraving will increase awareness.

> Second:
> A commercial edition of a public domain work does not restrict access
> to the work in any way. It just doesn't provide improved free access.

That argument I indeed made.
Improved and increased access to free contents (Creative Commons) 
created
by LilyPond is another way to increase awareness of it.

> So from the perspective of free culture is simply doesn't matter if
> the publisher produces this edition or not.

The point is not to try and prevent publishers to do whatever they 
want.
It is rather what _we_ (or at least some of the users of free software)
would like to foster (creating free edition of public domain contents),
independently of other, equally valid and respectable, endeavours.

> Consequently this is
> completely irrelevant to the question whether publishers should be
> encouraged to use LilyPond.

Agreed (so what?).

> Additionally, new editions often *do* improve access to the
> compositions, by means of an improved editorial quality. As a
> professional musician, I'd nearly always prefer paying for a
> commercial edition over printing an outdated 19th century edition 
> from
> the internet.

And wouldn't you prefer even more that a fellow LilyPonder had
encoded it for you to enjoy?

> It has been stated that commercial publishers produce editions that
> suck for their editorial quality and that therefore these publishers
> are not worth being convinced of using LilyPond. Sorry, but this
> really not very helpful. I didn't perform a qualified comparison, but
> I can't imagine that the average and overall editorial quality of
> "free" editions comes even close to what commercial publishers have
> achieved over the centuries.

Apples and oranges.
Unless I'm mistaken, the correct argument is a comparison between
softwares. And automatic creation of beautiful music: LilyPond tries
to achieve the quality which _humans_ built up over the centuries.

Again, free and commercial engravers can use the same tool to achieve
whatever quality level those tools allow.

Again, it would be great if commercial engravers embraced LilyPond.
They did not want to, they still don't want to.
I'd be interested to know the reasons. [E.g. "We rely on the
long-term maintenance of commercial software"... like "Sibelius".]

> This is also true for projects like IMSLP
> or Mutopia. These don't provide any inherent mechanism for editorial
> quality control. The vast majority of scores to be found there are
> simply copied from existing editions, without any standardized way to
> guarantee their correctness.

The implementation of the Mutopia has (serious) shortcomings (discussed
in another thread).  Does it mean that the idea is bad?

What about Wikipedia?  Should that idea have been dismissed under the
assumption that it would never have the same level of editorial quality
control as the "Encyclopedia Britannica"?

> And did anybody think about the fact that
> nearly all of these "model" editions have simply run out of copyright
> and wouldn't exist if they hadn't been produced by commercial
> publishers in the first place?

IIRC, it has been mentioned on this ML (quite some time ago) that new
editions with minute changes are published to ever extend the copyright
limit.
It is good (IMO) that at some point, public domain contents really
becomes free, even if it originates from the work of an engraver who
lived 100 years ago and did live from his engraving work.

> Truly Free editions would require
> people going to the archives and producing new editions from the
> original sources. Then we can start talking about *their* quality ...

That's a valid concern.

If the editor does a bad job, it's a bad job.
But, with a free edition, anybody can improve it (if some part of
work is worth using it) and share the now good edition.

Not so with a "new" commercial edition of "old" public stuff.

> In general I only use these free resources to get a first impression
> - which is of course an example use case for the free distribution of
> culture. But edition quality is an aspect that is somewhat at odds
> with the free distribution, and I would not accept sacrifying this
> part of the equation.

"quality is [...] at odds with free distribution"
???

First, people can be paid to produce free contents. It's not really
"gratis" if the money comes public funds (it was paid for, through
taxes).

Second, it is not fair, especially here, to indirectly associate
"money" with "quality" and "benevolent" with "crap".

It's foremost a question of policy. Commercial products are not
necessarily good because you pay for them...

> ###
>
> Now to the aspect of "convincing" publishers.
>
> > If they are such corporate dinosaurs that do not recognise the 
> > benefits of advanced lilypond technology,
> > open source and open systems, of what concern is it to the community 
> > of lilypond engravers?
>
> I have outlined above why this should be a concern to the LilyPond
> community. But the first part of that statement is too short-sighted
> IMO.
>
> I have talked with a few "corporate dinosaurs", and it quickly became
> clear that there is no sense in continuing my efforts, presumably
> until the respective persons have retired. One bluntly told me that
> plain text is outdated and that he had thought this had been overcome
> decades ago. Another one only slightly concealed (under politeness)
> that his company will only consider any technological change when 
> they
> can't prevent it because everybody else has already jumped the
> bandwagon.

That's another piece of information which would help preventing what
you deemed a misinterpretation of your original post.

> But most others I have talked with have been seriously interested and
> grasped quite clearly what I am talking about. They asked exactly the
> right questions and saw potentials and problems quite clearly too.
>
> But that's not enough to consider a change.
> When a business is proposed a new technology I think there are
> basically three considerations:
> a) The suggestion promises solutions for problems the business is
> actually and significantly suffering from
> b) The suggestion promises (mostly financial) benefits that outweigh
> the investment.
> c) The risk of failure seems overseeable.
>
> ad a)
> Most people clearly see the advantages.
> Notably engraving quality is one of the least in this regard. Most
> publishing houses are satisfied with their output because they have
> either invested quite some effort in it or they simply don't care.

So: "commercial" and "don't care [about the quality]".

From a long time subscriber's to this list, it seems that engraving
quality was _primarily_ why users of other softwares switched to
LilyPond.  [Making them willing to overcome their possible initial
wondering at the textual input.]

> There is one point that can be made here, and that is the
> out-of-the-box quality, which can be of interest when it comes to
> producing performance material on short notice and only later refine
> that to publication quality.
> The most compelling points seem to be the potentials of project
> management/documentation through version control and the option of
> distributing work over arbitrary numbers of collaborators. The Grid
> approach made quite some impression particularly.
> As mentioned I have the impression that the overall issues of data
> integrity and the risks of relying on commercial software have become
> more present in publishers' minds by now.

Imagine your suffering if the Sibelius case did not happen... :-)

> b) and c) are the problematic parts. And here I think the "risk" part
> is more of an issue thant the "investment". People can't really
> oversee how the new technology can fit into their existing
> infrastructure, particularly given the fact that they do and will
> always have contributions by numerous people from outside that have 
> to
> be integrated. Add to this the question if we can reliably enough
> convince anybody that "we" will be around in a few years and that
> there is a sufficient user base to guarantee that they will always
> find someone to get "tech support" from.

The problem is that they don't grasp what "free software" is.
If they want to, "tech support" will be there, thanks to them.
You cannot promise that it will be there, if they don't become
involved in and concerned about the (evolving) "community".

And as noted earlier, they have demonstrably wrong expectations
that tech support for commercial will always be there!

> So finally I'm back at the beginning, namely my original post's
> question, preparing a convincing set of facts, arguments and
> "promises" that help to overcome the reservations with regard to b)
> and c) of the above list. I have the impression I'll be more or less
> alone with that, so I'll try to put my things together on my own.
> Having triggered quite a bunch of different (and hopefully fruitful)
> discussions is something too ...

Back to thanking you for starting this discussion. And your 
involvement.
And your work. And your projects.
Maybe there will be a little slot for trying to take other paths into
consideration for increasing LilyPond user base (or publish what you 
tried
already and how/why it failed).

Thanks,
Gilles


> Best
> Urs
>
> Am 17.04.2015 um 15:03 schrieb Urs Liska:
> > Just one more of the fundamental questions I took home from the 
> > Musikmesse ...
> >
> > The question can be asked somewhat less pretentious then in this 
> > message's subject line, but I think it actually boils down to no less 
> > than that.
> >
> > You know that I have again been at the Frankfurt Musikmesse this 
> > week, and again I had the opportunity to talk with various people from 
> > publishing houses (names only privately ...), and I was (unpleasantly) 
> > surprised that I didn't always have fully satisfactory answers ready.
> >
> > The questions came in various variants of "why should a publishing 
> > house use LilyPond?" And despite all the reasoning and writing I have 
> > produced over the last years I didn't always find "the" striking key 
> > features that were convincing in the concrete situation.
> >
> > I think  I have always taken a perspective that was focused slightly 
> > beside the point, namely the perspective of an individual editor or 
> > the team of editors. This is perfectly transferable to a publishing 
> > house starting from scratch, but not to a big house with traditions, 
> > regulations and limitations.
> >
> > Compared to last year I have the impression that many people have 
> > become more aware of the basic questions about longevity of binary and 
> > textual data formats and data processing. The question has become much 
> > less "why should we consider dropping Finale and Sibelius, it's 
> > working, heh?" and more "OK, we see that we need an alternative 
> > approach, but how do you convince me that LilyPond has to be it?"
> >
> > We always say that text based tools are superior because they are 
> > much less prone to become unusable. This, and the potentials that come 
> > from version control, make quite some impression, but again this is 
> > only relevant to new material and doesn't take into account two 
> > important issues:
> >
> > 1)
> > Publishers receive heterogenuous data material from various sources 
> > (editors, composers, engravers), and these are mostly done using 
> > Finale or Sibelius (and in some cases Score). It is completely out of 
> > question to requre all these people to switch to LilyPond.
> >
> > 2)
> > One major question publishing houses consider today is how to carry 
> > their (digitally) existing material to the future. By now they have 
> > realized that simply using the latest version of the mainstream 
> > programs is calling for disaster, that's good for us. But again, the 
> > question is: Can we really offer "the" solution for this?
> >
> > The immediate idea would be to go some route via MusicXML and offer 
> > hassle-free workflows to convert existing editions or said received 
> > data material) to LilyPond. But firstly I don't think we can already 
> > guarantee this. And secondly, the question is natural to pop up: If 
> > one already uses MusicXML as the permanent and future-oriented storage 
> > format, where is then the need to consider LilyPond for this?
> > I think it is reasonable to expect that there will always be 
> > mainstream programs that can work with MusicXML files, and their user 
> > base will probably always be larger than ours.
> >
> > ###
> > So, now I've somewhat laid out why we are actually facing this 
> > question about "offering the future".
> > I can't give convincing answers to that, but of course I have a few 
> > ideas, and I would be happy about a constructive discussion. This is 
> > not a pipe dream discussion as I will (have to) pick up the 
> > communication with the publishers soon. And it will probably make a 
> > good impression if I can show that we have taken their concerns 
> > seriously, especially if I can come with some promising suggestions.
> >
> > The first asset is the fact that plain text tools allow highly 
> > sophisticated workflows and adaptation of the programs' functionality. 
> > The biggest impression I could make was probably our "grid" approach 
> > (of course only backed up by the fact that we have successfully 
> > realized it in a real project), and - to some extent - the prospect of 
> > maintaining the whole edition process within one single context (the 
> > \annotation functionality in the ScholarLY library). But the latter 
> > was somewhat less striking because it seems most publishing houses 
> > don't really care anymore about the editorial process, and they have 
> > the impression that this could actually create more overhead than they 
> > have currently.
> >
> > The second asset I see is that we can (principally, in the 
> > real-world it isn't completely mature yet) completely separate content 
> > from representation, which should be stressed very much when it comes 
> > to the questions of long-term data storage and of repurposing content.
> >
> > In theory, LilyPond (as LaTeX) is better suited to process textual 
> > data than binary tools, simply because it's their natural appraoch. 
> > But I don't know what I would answer to the question "well, yes, I 
> > see, but what *impact* does this really have?"
> >
> > There is one road that I could see as the "golden bridge".
> > I think MusicXML isn't the best solution for long-term management of 
> > editions. It's just too much focused on data exchanged between music 
> > software - and mostly pop music oriented tools. Looking for a 
> > fundamental solution to migrate the whole data base of a publishing 
> > house I would think that MEI is a much better solution because it's 
> > inherently more comprehensive, and because it has been originally 
> > conceived from an editor's perspective rather than music production 
> > tools.
> > Currently there doesn't exist *any* straightforward way to render 
> > MEI encoded data with a professional engraving program, so this could 
> > be the key feature for avoiding the question "so why should we prefer 
> > LilyPond over Sibelius or the new Steinberg app?".
> > IF we could come up with a promising path to let LilyPond work with 
> > MEI data (that is firstly: use MEI as input to LilyPond and/or convert 
> > MEI data to LilyPond files, and secondly: Be able to convert to both 
> > directions so one can also edit scores as LilyPond and convert them 
> > back to MEI for storage) that _could_ be the satisfactory answer I 
> > claimed as missing above. Publishers have more than once thought about 
> > a concerted effort with regard to data management. Of course that 
> > would also imply a platform for _distributing_ music, so the 
> > option/request to provide *digital* scores is also ubiquituous. 
> > Actually this was again suggested this week in one of my meetings, and 
> > I would love to pick that up with a more or less concrete but at least 
> > convincing outline.
> > At the same time the prospect of publishers (or "the" publishers in 
> > a common effort) would consider MEI this would motivate the MEI 
> > community, and if it would be somehow connected with LilyPond it would 
> > raise the chance that some of them would actually step out and start 
> > something worthwile (I know there is a latent interst in LilyPond 
> > within the MEI community, but there's no sufficient "market" for it so 
> > nobody actually came up with a solution so far.
> >
> >
> > ####
> > ####
> > To conclude:
> >
> > - most people in the business have moved away from taken the status 
> > quo
> >   with Finale and Sibelius for granted.
> > - they know that they *have* to find new answers.
> > - many (except a few die-hard reactionists) see that LilyPond and 
> > friends *can* offer answers to their questions
> > - but they also see that these are maybe not the only possible 
> > answers and
> > - that we (currently) can't guarantee straightforward migration 
> > paths.
> >
> > Market is hard, and everything is moving quite slowly, of course.
> > But IF we should be able to come up with convincing solutions or at 
> > least roadmaps I see that we now have better chances than ever to get 
> > LilyPond a foot in the door with the publishing business in general.
> >
> > Sorry for that elaborate text, but I think it is important and 
> > hopefully fruitful.
> >
> > Best
> > Urs
> >
> >
> >
> > _______________________________________________
> > lilypond-user mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/lilypond-user