[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Bug-wget] Updating a mirror

From: Luc Van der Veken
Subject: [Bug-wget] Updating a mirror
Date: Tue, 30 Dec 2008 10:31:41 +0100

I'm trying to mirror a site (http://www.gamingcommission.fgov.be) and
update the mirror at regular intervals.

This is primarily meant to detect new and changed pages and files.  While
at it, I set it up as a local mirror so I get the added benefit of that
(the site has a tendency of going down a LOT).

I'm using wget 1.10.2 on Ubuntu Server 7.10, apparently they don't have
1.11.x in the repository yet (just scanned for and installed updates today
to make sure).
Should I have asked somewhere else in this case?

The initial local copy works perfectly, it is fully browsable, etc.

Then when I run wget again with the exact same command line arguments, this
is what happens:

1) It checks only the top page (index.html) and direct descendants.  If
there are no changes there, it just stops.  In other words, it only detects
a change in a page if the parent of that page (in the link tree) has also

2) It messes up links in the pages it re-checks.  Any link to a page that
isn't downloaded again (because it hasn't changed) is replaced by an
absolute link to the original server, but using the filename of the local

An example of an URL mutilated by (2):

Mirror, initial copy:
Mirror, after one update:

I am using these command line parameters:
   --w 2
   -e robots=off

Is there anything wrong with my parameters?
Is it something in the site?
Is it a bug?  Fixed in 1.11.4?

BTW, in case someone wants to test it out locally: total size of the mirror
is 116MB.  You can significantly reduce that by skipping the .pdf and .doc

reply via email to

[Prev in Thread] Current Thread [Next in Thread]