Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"

[Daniel P . Berrangé]

On Tue, Sep 10, 2019 at 04:16:58PM -0400, Cleber Rosa wrote:
> On Tue, Sep 10, 2019 at 04:43:00PM -0300, Eduardo Habkost wrote:
> > On Tue, Sep 10, 2019 at 08:29:24AM -0400, Cleber Rosa wrote:
> > > On Tue, Sep 10, 2019 at 09:41:20AM +0100, Daniel P. Berrangé wrote:
> > > > On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> > > > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > > > > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > > > > the "vmlinuz" file.  The reason for the failure is that the Fedora
> > > > > project retires older versions from the "dl.fedoraproject.org" URL,
> > > > > and keeps them in "archives.fedoraproject.org".  As an added note,
> > > > > that test uses a Fedora 28 image, because of the specific Linux kernel
> > > > > version requirements of the test.
> > > > > 
> > > > > For the sake of stability, let's use URLs from the archived and
> > > > > supposedely ever stable URLs.  The good news is that the currently
> > > > > supported versions are also hosted on the later.  This change limits
> > > > > itself to change the URLs, while keeping the fetched files the same
> > > > > (as can be evidenced by the unchanged hashes).
> > > > 
> > > > The download.fedoraproject.org site we're (mostly) currently using
> > > > is serviced by the Fedora mirrors which is very desirable as it
> > > > spreads the load.
> > > > 
> > > > The archive.fedoraproject.org site is the master Fedora hosting
> > > > server(s). dl.fedoraproject.org is the same master hosting service
> > > > that is intended for use by the mirror sites to sync their content
> > > > from.  Projects really shouldn't use either of these URLs for getting
> > > > any content which is available via the mirror service as it places
> > > > uncessary load on the Fedora master servers.
> > > > 
> > [...]
> > > > > @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> > > > >          :avocado: tags=arch:ppc64
> > > > >          :avocado: tags=machine:pseries
> > > > >          """
> > > > > -        kernel_url = 
> > > > > ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > > > > -                      
> > > > > 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> > > > > +        kernel_url = 
> > > > > ('https://archives.fedoraproject.org/pub/archive'
> > > > > +                      
> > > > > '/fedora-secondary/releases/29/Everything/ppc64le/os'
> > > > > +                      '/ppc/ppc64/vmlinuz')
> > > > >          kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> > > > >          kernel_path = self.fetch_asset(kernel_url, 
> > > > > asset_hash=kernel_hash)
> > > > 
> > > > Don't change these URLs. We can update to Fedora 30 though if you want
> > > > to.
> > > > 
> > > 
> > > I believe your suggestion to not change the URLs is based solely on your
> > > perspective on the load on the "archives" server, right?
> > > 
> > > You should know that there's a number of counter points.  One is that
> > > these tests are used on environments, which are maintained for a much
> > > longer time than the lifespan of the currently maintained Fedora
> > > versions.  We really need to make it stable and reliable, and IMO it
> > > should be done upstream, for the benefit of all.
> > 
> > We're trying to offload the costs of long term hosting for our
> > test cases to a third party (Fedora Project), but we need to know
> > if that's really acceptable usage of archives.fedoraproject.org.
> > 
> > -- 
> > Eduardo
> 
> I'd argue that we're boosting archives.fedoraproject.org morale.  As a
> public Internet service, it begs to better be used! :)

Not really, in general we want users to keep updated with latest
supported Fedora releases and not use EOL versions from archive. 
The critical reason for archives.fp.org to exist is license,
beyond that its largely just a historical record, but we digress
off topic here...

> Now, more seriously, I agree that we should coordinate with them.  I
> have the feeling that it shouldn't be a problem of resources per se,
> given that we're redirecting traffic.  But, it may be a matter of
> letting them better allocate these resources.
> 
> Daniel,
> 
> You seem to have some much better than average knowledge about the
> mirror structure.  Does that mean you know who to contact about this?

I spoke with some folks I know who didn't have an answer on the impact
or policy around using archive.fp.org, but did point out that there's
a way to avoid breaking

The Fedora mirror manager software provides a URL to query what
sites currently host a given version

eg to find Fedora 16 for x86_64 fetch this

  https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-16&arch=x86_64

whereupon it reports something like this:

# repo = fedora-16 arch = x86_64 country = DK country = DE country = UA country 
= global 
https://mirrors.dotsrc.org/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
https://fedora-archive.ip-connect.info/fedora/linux/releases/16/Everything/x86_64/os/
https://ftp-stud.hs-esslingen.de/pub/Mirrors/archive.fedoraproject.org/fedora/linux/releases/16/Everything/x86_64/os/
https://fedora-archive.ip-connect.vn.ua/fedora/linux/releases/16/Everything/x86_64/os/
http://ftp.cuhk.edu.hk/pub/linux/fedora-archive/fedora/linux/releases/16/Everything/x86_64/os/
http://fedora-archive.mirror.liquidtelecom.com/archive/fedora/linux/releases/16/Everything/x86_64/os/
http://mirrors.kernel.org/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
http://mirror.math.princeton.edu/pub/fedora-archive/fedora/linux/releases/16/Everything/x86_64/os/
https://pubmirror2.math.uh.edu/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
https://pubmirror1.math.uh.edu/fedora-buffet/archive/fedora/linux/releases/16/Everything/x86_64/os/
https://dl.fedoraproject.org/pub/archive/fedora/linux/releases/16/Everything/x86_64/os/

any one of those returned URLs should serve the content we want. It is
interesting to see that there's several mirrors still carrying ancient
software, despite 'download.fp.org' returning 404

Admittedly this extra redirection is a bit more tedious for the test
suite, but we could hide that behind a common helper function.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|