[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] HTTP quota bug
Re: [Bug-wget] HTTP quota bug
Sun, 03 May 2009 12:39:32 +0300
Icedove 184.108.40.206eol (X11/20080724)
Micah Cowan wrote:
I don't think that changing this may break someone's script. Most likely
current wget's behaviour may cause enexpected results and can occupy
match more disk space than it was expected.
As it described in documentation, when --quota option is used "download
will be aborted when the quota is exceeded". But HTTP code has no
corresponding lines to break download unlike FTP code. So, if some file
is downloaded via HTTP, it will be fully downloaded and stored (no
matter is --quota option used or not) but with additional warning when
quota is exceeded: "Download quota (... bytes) EXCEEDED!"
What documentation are you talking about? This is what I see:
Specify download quota for automatic retrievals. The value can be
specified in bytes (default), kilobytes (with `k' suffix), or
megabytes (with `m' suffix).
Note that quota will never affect downloading a single file. So
if you specify `wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz',
all of the `ls-lR.gz' will be downloaded. The same goes even when
several URLs are specified on the command-line. However, quota is
respected when retrieving either recursively, or from an input
file. Thus you may safely type `wget -Q2m -i sites'--download
will be aborted when the quota is exceeded.
Setting quota to 0 or to `inf' unlimits the download quota.
Which is exactly the case, whether you're talking FTP, or HTTP. It
doesn't break download in the middle of a file.
Which, yeah, I agree is counter-intuitive. But with a program like wget,
I can never be sure that changing this won't break someone's script
somewhere. Not that we shouldn't do proper quotas, but we most likely
need to add that feature as a different option.
Let's assume that someone write script and set quota to 1G.
As it described in manual "Thus you may safely type `wget -Q2m -i
sites'--download will be aborted when the quota is exceeded". So, user
will expect that command `wget -Q1000m -i sites' will be executed safely
and will be aborted as soon as quota is exceed. But, if user download
file which size is 5GB (or mirror any site that contain similar file) it
will be used 5-6Gb of disk space after script execution.
So, in this case I can't see any safe wget execution with --quota
option. At the same time I see that if free space at selected partition
was 4,5GB, wget can't download all files (because they are need more
disk space) but it can break normal system functionality (if come
processes are working with selected partition and will not find free
space after wget execution)
I think that manual page should be appended by explanation that "quota
is checked only when file download is finished and it doesn't break
download in the middle of a file" or something similar. Also, I hope
that additional option will be added to future versions of wget to
provide breaking file download as soon as quota is exceed.
Trifle Co., Ltd.