RFC: Grub project management

[Glenn Washburn]

Hi fellow GRUB developers,

I want to start by recognizing that most people who are involved with
the project maintenance are doing so on a voluntary basis (or at least
that's my assumption) and people have busy lives. Its a mostly
thankless job, so thank you guys for all the time spent keeping this
essential project alive. I hope that the following will not be construed
as judging anyone, but merely taking account of where we've been, where
we are, and how to move forward.

TLDR; Check out https://gitlab.com/gnu-grub/grub and do merge requests
with changes rebased on to feature/gitlab-testing against
feature/gitlab-testing to get automatic CI testing.

I believe I speak for more than just myself when I say that the current
development process leaves much to be desired. GRUB has been in a
feature freeze since last March, with the release being pushed
rescheduled several times and now looks like its projected to be in
March (assuming its not pushed back again). I get the impression that
Daniel is overloaded with non-GRUB responsibilities. I also wonder if
part of the issue is concerned about making a release which he believes
hasn't had sufficient testing.

Daniel, could you please clarify what needs to be done to get the
release out the door and what the community can do to help expedite
this process?

In the meantime, patches are being sent to the list which will not be
considered for acceptance until the release is out and feature freeze
lifted. Does anyone believe that someone is going to go back to last
March and check all the submitted patches to see about including them?
I judge the answer to be no. So patches will be continually falling
into a blackhole. What we need is an issue/merge management system so
things don't get lost, or can easily be found.

GRUB currently uses the Savannah GNU project hosting site. And while I
love GNU philosophy and the role that Savannah has played for free
software projects, it is showing its age and not ideal for a modern
development process. My two big pain points is that it does not do
merge requests and the bug tracker is an unpleasant experience when
compared to modern ones. If you've ever tried to search for a bug,
you'll know what I mean. So unfortunately, I think its time to say
goodbye Savannah.

Another issue, as I see it, is the testing. The only testing that I'm
aware of is the travis CI, which is only a build test, and Adrian's
testing for debian, which does do the make check tests but is missing
filesystem testing and qemu tests for most architectures. Ideally, the
make check tests should be done within travis, but we're not there yet.
This process feels closed and non-transparent to me, which I find
undesirable in general, but especially so for a free software project
which is the corner-stone of the linux community. And I'm not saying
that that is intentional on the part of those working on those tests.
So, for one, it would be nice to have publicly accessible the status of
Travis runs. And may be it is, but why isn't it obvious?

Based on what I've gathered from the list and private conversations
(which isn't much), the travis is only really run as a prep for making
a release. If that's true, the process fails to use the main point of a
continuous integration system. I would like GRUB to get to a place
where every commit is run through CI (at least the build test). In the
short term, it would be nice to kick off a CI job every time master
changes.

It would be even more nice if we could integrate the merge tracking
system with the CI system so that merge requests automatically kick off
a CI job to see if the proposed changes pass both the build test and
functional make check tests.

I believe, and my sincere hope, is that by resolving these issues we
can speed up the development cycle. I have a lot of changes that I've
been waiting months to submit to the list. I don't think it should take
a mountain of patience to get changes accepted. The GRUB project and
its users are worse off for it.

It is in consideration of all the above that I've setup a system on
GitLab that addresses all these concerns and more. I also believe that
GitLab was founded as a more ethical company that would be
more compatible with a GNU software project than most other project
hosting sites. I chose to use GitLab's builtin CI because it has what
appears to be unlimited free resources (although resource constrained
to an extent) for open source accounts.

I've setup an unofficial GNU GRUB group and project repository at
https://gitlab.com/gnu-grub/grub. It is currently configured to mirror
the official repository. I have created a few merge requests that are
real examples to show what the process would look like.

The most important merge request is the one adding the GitLab CI
feature (https://gitlab.com/gnu-grub/grub/-/merge_requests/2). This is
a merge request created from a branch in the repo, as opposed to one
from a fork. Because it has the .gitlab-ci.yml file configured to run
the CI on merge requests, a "Detached merge request pipeline" is kicked
off. The results of that can be seen on the merge request page with a
link to the pipeline and jobs so that failing jobs can be quickly found
and debugged. The last job in the pipeline outputs a summary of
architectures for which "make check" tests were run and different kinds
of test failures, for a quick overview. Importantly, the GitLab CI
config runs "make check" tests successfully for 8 targets: arm-efi,
arm64-efi, i386-efi, i386-pc, i386-qemu, powerpc-ieee1275,
sparc64-ieee1275, and x86_64-efi.

Another merge request is titled "Add basic support for xHCI USB
controllers and non root xHCI hubs." As noted in the description, this
is a patch series I took off the list authored by Patrick Rudolph. The
merge request pipeline fails for the x86_64-efi build, but all the
others succeed. I suspect that the gcc being used (10.1.0) is newer than
the one developed on and this gcc may be grumpier. These errors can be
fixed, the pushed to the same branch and another CI job will be started
to test the updated merge request automatically.

Until the GitLab CI feature is merge into master, merge requests should
be rebased onto the feature/gitlab-testing branch and merged requested
against that branch (not master). This is needed so that the merge
request does not show the GitLab CI commits as part of the merge
request. Once the GitLab CI feature has been merged into master, merge
requests can be off master and against master, as one normally would.
One can see that this has been done for all the current merge requests.

For patches that people don't want to disappear on the list, I think a
merge request can help mitigate that. Also since the merge request is
effectively a whole commit tree, instead of some free floating diffs,
we can know precisely which commit its based off of (master? or last
stable?).

This is not intended to change the current patch submission
requirements for the project. Patches will still need to be sent to the
list and will be reviewed on the list. I think for non-trivial patches
it should be required to make a merge request as well so that the CI
passing is a requirement. I hope that by enforcing a passing CI that
maintainer and reviewer time is saved by ensuring that patch series
pass the sniff test.

This is currently not officially part of the GRUB project, so creating
issues, for instance, should still be done on Savannah. I'll be sending
out a patch series to the list soon with the changes used to integrate
GRUB into GitLab's CI.

If you've made it this far, congratulations! I hope we can put this to
good use and help speed up GRUB's development cycle.

Glenn