Re: Linux-libre 5.8 and beyond

[Alexandre Oliva]

Hello, Mark,

Apologies for the delay in responding.  It's been an "interesting" week.

I'm breaking up what turned out to be a very very long reply into
multiple posts, so as to address the various issues in separate posts,
that might very well turn into separate subthreads.


On Aug 15, 2020, Mark H Weaver <mhw@netris.org> wrote:

> Alexandre Oliva <lxoliva@fsfla.org> wrote:

>> No.  There are much better (faster and less risky) ways to tend to that
>> requirement, see #bisecting below.
> [...]
>> #bisecting
>> 
>> You can even take one of our releases and apply the patches that went
>> into the next upstream stable release, and check that what you get
>> matches our own corresponding release.  Some 98% of the time, they will
>> be exact matches.  Occasionally, there will be a difference, and then
>> you'll likely find a corresponding change in the deblobbing scripts, or
>> a preexisting pattern that caused the change.  We do this for every
>> release, as part of our pre-release checks, and you're welcome to do so
>> as well, and to use the resulting tree to bisect problems.

> I agree that this would be faster, but I fail to see how it's "less
> risky" than running the deblob scripts meant for Linux-libre X.Y.Z on a
> git checkout of the upstream stable git repo between X.Y.(Z-1) and
> X.Y.Z.

You're right.  My mistake was failing to mention the need to compare
between X.Y.Z-gnu* and the result of rebasing the X.Y.(Z-1)..X.Y.Z
patches onto X.Y.(Z-1)-gnu* to enjoy the safety I had in mind.

If the changes made to both ends are the same, then it's not entirely
unreasonable to assume that all intermediate commits would also be
properly cleaned up with the same set of changes, assuming the history
in between is linear (i.e., only cherry-picks, not merges that could
take a bisect to much earlier commits).

Even with linear history, you might admittedly still be surprised if an
intervening patch introduces something undesirable and a subsequent one
reverses it.  Running through deblob-check each version of each modified
file that matches neither boundary would mechanically avoid nearly all
such surprises.


> More importantly, it's a much less straightforward thing to implement.

It really isn't.  Once you remove the deblobbing and point the recipe
directly at the linux-libre git repo, users will be able to do the above
rebasing on a local repo, and build reasonably quickly any of the
commits that the local git bisect tells them to try.

> In the current implementation, we get the ability to deblob arbitrary
> git commits from the same stable branch essentially for free.

Taking arbitrary commits from a known non-Free repo is really not
something to be encouraged, given the odds of hitting freedom problems.
When using the latest scripts for a stable series, odds are the
procedure you suggest would work more or less reliably within that
stable series, but we've recently had an example in the 5.7-gnu series
in which it wouldn't.

> I guess you're suggesting that I should implement a radically different
> mechanism specifically for this purpose, that extracts the individual
> patches from the upstream stable git repository, attempt to apply them
> to the base Linux-libre release, compare that to the next Linux-libre
> release, and then implement my own bisection functionality.

git rebase --onto libre/vX.Y.(Z-1)-gnu nonfree/vX.Y.(Z-1) nonfree/vX.Y.Z
git diff libre/vX.Y.Z-gnu
git bisect start HEAD libre/vX.Y.(Z-1)-gnu

> If I were to implement this, what would you suggest I do if the patches
> fail to apply

Look at the conflict presented by the rebase, and resolve the likely
freedom issue introduced at that point.

> if the result fails to match the next Linux-libre release?

Identify the intervening commit where code that got cleaned up
differently was introduced or removed and make the change to the
deblobbing at that point; rinse and repeat.

If differences remain that are not caused by the patches, it's something
that changed in the scripts, possibly improving or correcting an earlier
deblobbing error, e.g., something cleaned up that was found to be Free,
or something that was missed in deblobbing, or a different way to clean
it up.  Such differences will likely be noticeable in the scripts.  It's
probably best to turn the difference in cleaning up into a separate
commit for the purposes of the bisection, just in case it is the source
of the issue being investigated.


-- 
Alexandre Oliva, happy hacker
https://FSFLA.org/blogs/lxo/
Free Software Activist
GNU Toolchain Engineer