|
From: | Andrey Semenchuk |
Subject: | Re: [Bug-wget] HTTP quota bug |
Date: | Sun, 03 May 2009 12:39:32 +0300 |
User-agent: | Icedove 1.5.0.14eol (X11/20080724) |
Micah Cowan wrote:
I don't think that changing this may break someone's script. Most likely current wget's behaviour may cause enexpected results and can occupy match more disk space than it was expected.As it described in documentation, when --quota option is used "download will be aborted when the quota is exceeded". But HTTP code has no corresponding lines to break download unlike FTP code. So, if some file is downloaded via HTTP, it will be fully downloaded and stored (no matter is --quota option used or not) but with additional warning when quota is exceeded: "Download quota (... bytes) EXCEEDED!"What documentation are you talking about? This is what I see: `-Q QUOTA' `--quota=QUOTA' Specify download quota for automatic retrievals. The value can be specified in bytes (default), kilobytes (with `k' suffix), or megabytes (with `m' suffix). Note that quota will never affect downloading a single file. So if you specify `wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz', all of the `ls-lR.gz' will be downloaded. The same goes even when several URLs are specified on the command-line. However, quota is respected when retrieving either recursively, or from an input file. Thus you may safely type `wget -Q2m -i sites'--download will be aborted when the quota is exceeded. Setting quota to 0 or to `inf' unlimits the download quota. Which is exactly the case, whether you're talking FTP, or HTTP. It doesn't break download in the middle of a file. Which, yeah, I agree is counter-intuitive. But with a program like wget, I can never be sure that changing this won't break someone's script somewhere. Not that we shouldn't do proper quotas, but we most likely need to add that feature as a different option.
Let's assume that someone write script and set quota to 1G.As it described in manual "Thus you may safely type `wget -Q2m -i sites'--download will be aborted when the quota is exceeded". So, user will expect that command `wget -Q1000m -i sites' will be executed safely and will be aborted as soon as quota is exceed. But, if user download file which size is 5GB (or mirror any site that contain similar file) it will be used 5-6Gb of disk space after script execution.
So, in this case I can't see any safe wget execution with --quota option. At the same time I see that if free space at selected partition was 4,5GB, wget can't download all files (because they are need more disk space) but it can break normal system functionality (if come processes are working with selected partition and will not find free space after wget execution)
I think that manual page should be appended by explanation that "quota is checked only when file download is finished and it doesn't break download in the middle of a file" or something similar. Also, I hope that additional option will be added to future versions of wget to provide breaking file download as soon as quota is exceed.
-- Best wishes, Andrey Semenchuk Trifle Co., Ltd.
[Prev in Thread] | Current Thread | [Next in Thread] |