Re: [lmi] Using Git submodules for the dependencies

[Greg Chicares]

On 2019-09-21 21:48, Vadim Zeitlin wrote:
> On Fri, 20 Sep 2019 18:45:06 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> Okay, I guess, though I'm not sure I understand the ramifications.
> GC> Does this simply mean that we'd now add all the wx etc. sources to
> GC> lmi's savannah repository? And that then, at our leisure, we could
> GC> adjust makefiles and scripts to use those newly local sources? But
> GC> that for now nothing would break--i.e., we could continue building
> GC> as in the past, and it would still work?
> 
>  There are indeed 2 steps:
> 
> 1. Adding submodules (note that this doesn't necessarily add them to
>    Savannah repository, it just adds the pointers to whichever repository
>    really contains them -- but it could, and probably is going to, be
>    Savannah too, as per the discussion below).

I think we need copies rather than pointers. Our corporate server blocks
almost everything except github. (For today's lmi build system, that
doesn't matter as long as all tarballs in /cache_for_lmi/downloads/
are copied over, manually; but I've done that once, and don't ever want
to have to do it again, e.g., if we ever move to a different server.)

> 2. Starting to use them instead of whatever we do now (i.e. clone remote
>    Git repositories or downloading tarballs from SourceForge etc).
> 
>  Usually it makes sense to do both together because there is not much point
> in having (1) if the submodules are never used. But if you'd like, we could
> do this in 2 different steps. (1) will definitely be helpful to us, anyhow,
> as the upcoming install_linux.sh could use the submodules, even if
> install_msw.sh does not.

I'm just leery of destabilizing anything right now. A major release
will be distributed on Halloween, according to the latest plan, and
I don't want to take any risks until that has gone out--so delaying
(2) temporarily seems wise.

> GC> >  Similarly, I wonder where would you want the submodule URLs to point 
> to.
> GC> > We could just continue using https://github.com/wxWidgets/wxWidgets.git 
> and
> GC> > https://github.com/vadz/wxpdfdoc.git as we do now in install_wx*.sh, but
> GC> > perhaps you want to use some repositories on Savannah instead? This can 
> be
> GC> > changed later, but if you already know which repositories do you want to
> GC> > reference, it would make sense to start by using them immediately. Of
> GC> > course, if you don't, then you don't really need to waste time thinking
> GC> > about it and I can just use the same URLs as now.

To avoid overly-aggressive corporate network filters, I think we need
to have a copy of anything that isn't on github. If, say, we wanted to
build our own 'make', then pointing to its repository on savannah seems
perfect, but would turn out not to work, I'm afraid: the server allows
only github, so if the github mirror of lmi contains a pointer to the
'make' repository on savannah, that pointer can't be dereferenced.

Corollary: wx and wxpdfdoc are already okay: pointers into github from
savannah on the main lmi site become pointers into github on lmi's
github mirror, so we don't need deep copies of those libraries.

> GC> I don't know. Maybe the version we have on redhat does support
> GC> submodules, but its failure on
> GC>   git clone jobs=
> GC> caused me to question that.
> 
>  According to
> 
> https://github.blog/2016-03-28-git-2-8-has-been-released/#parallel-fetches-of-submodules
> 
> "--jobs" parameter of git-clone was added in Git 2.8, while Git version in
> RedHat 7 is 1.8, which is almost 7 years old.

Yes, what we have is git-1.8.3.1 .

> But you have to install SCL
> to get any reasonably recent versions of development tools under RedHat
> anyhow, and there is rh-git218 which contains a version which is perfectly
> usable with submodules.

According to
  https://www.softwarecollections.org/en/scls/rhscl/rh-git218/
it should be as easy as this:

sudo yum-config-manager --enable rhel-server-rhscl-7-rpms
sudo yum install rh-git218
scl enable rh-git218 bash

But is that likely to break anything else? IOW, would we just
be yum-ing git from some alternative place, like grabbing a
single package from debian-testing--with no effect on any
other package?

> GC> > Of course, there is always the "when all you have is
> GC> > Debian, any Linux looks like an OS where you can install it in a chroot"
> GC> > approach, i.e. you could use debootstrap to create a chroot of Buster on
> GC> > this machine and just do everything inside it with minimal drawbacks (a 
> few
> GC> > hundred of extra MiB of disk consumption).
> GC> 
> GC> Is that feasible? If so, it sounds like an excellent idea.
> 
>  I think it is... I've never done it in this direction, only the other
> way round, but there is this package
> 
> https://rpmfind.net/linux/RPM/epel/7/x86_64/Packages/d/debootstrap-1.0.109-2.el7.noarch.html
> 
> and I guess it's at least supposed to work, if it exists. Should I try
> creating a Debian chroot inside my (just installed) CentOS chroot?

Yes, please. If that works, then it resolves an entire category
of problems, some of which I may not yet even have encountered.

> GC> If my debian system had no GUI; and forced me to use a
> GC> little rubber keyboard with no Ins key at all (while many
> GC> other keys require Fn, like Fn+UpArrow = PgUp), and a
> GC> tiny screen, and a touchpad; and terminated any session
> GC> after 15 minutes of inactivity--then I'd find debian
> GC> pretty hard to use, too.
> 
>  Sorry if I'm just being obtuse, but why can't you ssh into this system
> from a machine with a larger screen and better pointing device?

I'm not sure that's possible. I think it may be accessible
only on an intranet, through a VPN.

> GC> Perhaps I should do the same. After actually installing
> GC> fedora (before finding out that's not very close to redhat),
> GC> for multi-booting (before you explained that there are
> GC> better ways), I looked into installing a centos chroot,
> GC> but it seemed like too much work for me, for too little gain.
> GC> 
> GC> But if you're going to do that anyway, maybe you could share
> GC> your exact steps and I can just ride your coattails.
> 
>  I've documented the steps I used for creating my CentOS 7.6 chroot at
> 
> http://www.tt-solutions.com/en/articles/install_centos_in_debian_chroot
> 
> Some of this might be unnecessary, e.g. I guess you could live without
> pseudo-terminals if you always used [s]chroot to enter the chroot, but as I
> have a setup in which I can ssh myusername-chrootname@localhost to enter a
> chroot as if it were a remote machine, I do really need them (and handling
> chroots and remote machines in the same way is convenient).

Thanks. I'll give that a try.

>  And also please let me know if you'd like me to create the mirrors for
> everything we use at Savannah.

Yes, please.