[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Guix mirror: sourceware discussion report

From: zimoun
Subject: Guix mirror: sourceware discussion report
Date: Fri, 13 Dec 2019 18:46:54 +0100


Currently "guix pull" from Savannah and issues can arise. As we
recently experimented. Tobias and Ricardo recently discussed how to
mirror the repo. IMHO, it is a good idea to mirror but not a good idea
to locate it on Ricardo infrastructure, again. :-)

Well, I propose to see if we can mirror on which are
GNU friends. ;-)

I took the liberty to contact them on IRC. See below.

What I see is: we still "guix pull" from Savannah as usual.
Then if Savannah is down, we catch the error and we try (transparently
for the user) the sourceware url.

As a first contact, they agree on the principle. Even they ask numbers
about traffic etc. :-)

What do you think?

All the best,


[Fri Dec 13 2019]
<zimoun> I am working with Guix and we discussed mirroring the GNU Guix repo
         on sourceware.  [17:56]
<zimoun> The best before doing plan is to ask if it should be possible, on the
         principle.  [17:57]
<fche> is this a source code repo or somethign else?
<fche> (sorry, not familiar)
<cgf> Also, what would be the reason for a mirror on sourceware if you already
      have a repo?  [17:58]
<zimoun> To be more concrete, and if you are not familiar with Guix, there is
         command named "guix pull" which basically "git pull". Currently, the
         main repo is on Savannah.
<zimoun> And everything is fine with Savannah. :-)
<zimoun> However, when Savannah is down; which happens, really rarely, but
         happens, then all Guix is kind-of broken.  [17:59]
<fche> so how big is this repo, what form is it, ?  [18:00]
<zimoun> Broken means that the users cannot update their system. And it is not
         related to development but related on normal usage. Well, "guix pull"
         is kind-of equivalent to "apt-get update".
<fche> accessed via git or what?
<fche> we're not set up to host large binaries e.g.  [18:01]
<zimoun> Yes, via Git. And it is only for source.
<cgf> Maybe hold off on this until new sourceware is online?
<zimoun> The mirror of
<zimoun> And if it is too big, maybe even only the master branch  [18:02]
<fche> hm, I can't git-clone that bad boy here
<fche> ah never mind starting  [18:03]
<zimoun> What I have in mind, and I speak for myself and not on the behalf of
         the Guix project.
<fche> not that big apparently, some 170MB of content, that's not a problem
<zimoun> sorry, my browser returns me the wrong URL  [18:04]
<fche> yeah
<fche> do you have a personal sourceware presence already?
<zimoun> Just to be on the same wavelength, currently, each time an user run
         "guix pull" then they "git pull" from However, when Savannah is
         down which rarely happens, everybody is annoyed. So I would like that
         "guix pull" catch the error and then try to use a mirror. Which
         means, that will be the default
         mirror. So the traffic pull will be very low (when Savannah is
<zimoun> down or unreachable) and the traffic push a bit more.
<mjw> guix is awesome! But savannah often is overloaded :{
<zimoun> No I do not have a personnal sourceware presence already.  [18:12]
* fche has no objection
<fche> how do you plan to make the local git repo a fresh enough mirror?
<zimoun> Then, I can ask to Savannah admin to have a hook to automatically
         mirror to sourceware
<fche> or you can run a cron job on sourceware to auto git-pull periodically
<zimoun> yes, I have not think too much about the details. Because I would
         like first if it was even possible. :-)
<fche> do you expect the repo to seriously grow over time ?  [18:14]
<zimoun> No. But I can do some stats with Git to answer you more
         precisely. Currently, Guix is around 13000 commits per
         years. However, I do not know how many addition lines it represents
<mjw> It guix.git is ~124MB currently. It normally sees multiple commits a
      day, but most of them are fairly small (a couple of lines to a handful
      of files).  [18:18]
<fche> I mean it's not going to grow from 120MB to 1GB or 10G in the
       forseeable future?
<fche> those are the levels at which we might get concerned
<zimoun> Who knows ;-) But I do not think so. And in case, you can say: stop,
         it is too much for our infrastructure.  [18:20]
<zimoun> Well, if we have a first agreement on an hostage (mirror), can I save
         this conversation and post it on the mailing list guix-devel. Collect
         feedback; especially from the maintainers. Then come back to you with
         more concrete details.  [18:21]
<fche> 'forseeable' doesn't ask for infinite prophecy powers :)
<zimoun> :-)
<fche> yeah, I think we can do it.
<zimoun> thank you for your help.
<segher> but how much load on sourceware will this cause?
<zimoun> do you want to be CC of the email?
<mjw> guix is very much pure GNU, and very much sources should produce
      reproducible builds. You can expect it to stay text based forever and
      never see any binary blobs.
<fche> segher, it's a git repo similar in size to valgrind  [18:22]
<zimoun> Almost no load on sourceware. As I tried to explain above. It would
         be a feedback when Savannah is down -- which happens rarely from my
<segher> fche: no...  how many pulls per seconds are asked from it  [18:23]
<segher> zimoun: that doesn't change the worst case at all
<zimoun> segher: I have no idea. I think it is hard to count.
<fche> segher, I expect nothing noticeable, but zimoun will let us know if
       there's reason to worry  [18:24]
<fche> ie a few downloads a day is nothing; a hundred git updates an hour is
<segher> and a simple gcc pull is almost 20m
<mjw> zimoun, I can also provide a backup mirror. Not as fancy as sourceware,
      but if you need something to fallback on.  [18:25]
<zimoun> segher: yes, it is just a first contact to see if we can agree. I
         totally understand your limitation.
<segher> a fresh clone, i mean
<fche> the gcc git repo is weird/svny and much larger
<segher> yes
<segher> my point is that we already are overloaded  [18:26]
<fche> not with git clones
<segher> the server, in general
<mjw> zimoun, it would go under (I
      cannot guarantee an SLA though, but I think neither can sourceware)
<fche> and definitely not with git updates
<segher> if it is actually light weight, then who cares ;-)  [18:27]
<cgf> overloaded sourceware is why I suggested waiting until the new server
      comes online.
<segher> right
<cgf> There's also the issue of outdated git.  [18:28]
<segher> it already is super annoying (to GCC) that half of the time (or
         whatever; it feels very often) the problem is caused by cygwin, not
         by us
<cgf> Probably more like 70% of the time.  :-)  [18:29]
<zimoun> mjw: thank you. And thank you for the hard work you did on the gnu
         list moderation.
<segher> cgf: it feels that way, yes
<mjw> zimoun, haha, yeah about that... o boy.
<cgf> I may be wrong but I don't think there are any other projects like
      cygwin on sourceware.  cygwin is a source code repository (lightweight)
      and a binary distribution mechanism (imo, heavy weight)  [18:30]
<cgf> but I digress...
<zimoun> mjw: as you are also on #guix maybe you can suggest the idea of
         wildebeest. Did you subscribe to the guix-devel mailing list?  [18:31]
<segher> cgf: right, and sharing resources with a project with different usage
         patterns of those resources is a bit problematic
<fche> segher, let us know of problems when they appear, and we'll keep
       tweaking things  [18:32]
<fche> there are a lot of knobs to turn
<segher> fche: i can tell you things were bad yesterday  [18:33]
<segher> not too useful, eh :-)
<zimoun> My point coming here and asking is: establish a first contact to see
         if we can go further together. Or if it is already a no go. Because I
         understand you need numbers and details. And I do not have them,
         right now. But I would like to open the discussion on this topic of
<fche> segher, exactly :)
<mjw> And on Monday gcc will switch to git too!  [18:34]
<fche> zimoun, we can put something like that up in a matter of minutes
<segher> (gcc bugzilla gave ISEs most of the time)
<mjw> well as a mirror and then Dec 31! finally git all the way!
<fche> mjw, that should help our loads, svn is much less efficient than git
<zimoun> fche: thank you for the offer but I am not maintainer and I speak on
         my own voice. I would like to hear the civodul/Ludo's opinion :-)
<mjw> civodul did suggest sourceware too, I did tell him to join here. But
      does he listen to me...  [18:36]
<zimoun> mjw: yeah! I know... that's why I am doing ;-)
<zimoun> but you know, french diplomacy :-)  [18:37]

reply via email to

[Prev in Thread] Current Thread [Next in Thread]