[Bug-wget] Re: HTTP quota bug (Matthew Woehlke)

[Matthew Woehlke]

Andrey Semenchuk wrote:
> Matthew Woehlke wrote:
> > Micah Cowan wrote:
> > > It doesn't break download in the middle of a file.
> > >
> > > Which, yeah, I agree is counter-intuitive. But with a program like wget,
> > > I can never be sure that changing this won't break someone's script
> > > somewhere.
> >
> > I have a script that would be broken by such a change :-). (Well, 
> > except that at this point, it usually finishes the download before it 
> > would run into the quota.)
>
> Matthew if your script finishes the download before it would run into 
> the quota - you have no reason to use a quota at all. But even in those 
> cases when script is running enough to run into the quota I have no idea 
> how download breaking in the middle of a file may broke your script. Can 
> you explain what really your script doing and how it can be broken in 
> such situation?

Oops, I guess I should have finished reading the /list/ :-). Please 
thread your message correctly next time ;-).

Well I already replied privately, but since you're asking the whole 
list, I guess I shouldn't leave everyone else in the dark.

It's quite simple, actually. I'm using quota (and rate limiting) to 
limit bandwidth usage when mirroring a site (for the sake of the server, 
not because my own bandwidth is an issue). I *did* hit the quota when I 
initially mirrored the site. I'd hit it again if there is a large amount 
of change since my last update; I want this to happen. In practice I've 
been resyncing often enough that I usually don't hit the quota, but that 
doesn't mean I want to remove it.

Now... why would my script break? I don't care about disk usage, so I'm 
not using it the way you are thinking. Since I'm not using download 
resuming, a behavior change would translate into "corrupt" files if I 
hit the quota.

(Besides, what happens if you're trying to download a single file larger 
than the quota and the server doesn't support resuming? I realize such 
servers should be increasingly few and far between, but...)

Now, I'm all for improving the doc to make it more clear how the current 
behavior works. I'm all for adding an option to interrupt downloads on 
exceeding the quota, even (probably as a modifier of the current 
behavior rather than a separate quota). But I like the current behavior, 
and feel Micah is right that changing it (in the absence of a new option 
to request different behavior) would be the wrong thing to do.

--
Matthew
Please do not quote my e-mail address unobfuscated in message bodies.
--
"Any customer can have a car painted any color that he wants...
...so long as it is black." -- Henry Ford