Re: [Bug-wget] Timestamping vs incomplete downloads

[Tim Rühsen]

On 10/23/18 9:07 AM, Darshit Shah wrote:
> On October 22, 2018 11:49:12 PM UTC, Dave Warren <address@hidden> wrote:
>> Currently when a download with timestamping enabled gets interrupted, 
>> the timestamp of the resulting file ends up being the current time and 
>> when wget is re-executed after connectivity is restored the local file 
>> is then seen as newer and skipped.
>>
>> robocopy handles this a little differently, by setting a date far in
>> the 
>> past as a way of ensuring that on a subsequent execution the transfer 
>> can be resumed.
>>
>> Is there a better way to handle this situation in wget? A way to force 
>> an old date on the file? I'd be happy with a fixed "in the past" date, 
>> the service supplied date minus a second, etc. Or some way to detect 
>> that the file is incomplete (too small) on a subsequent run?
> 
> I haven't tested it but what you say indeed sounds like a valid bug.
> 
> The cleanest approach, IMO, is to use the extended file attributes in modern 
> systems to store this time at the very beginning and look for it on 
> continuation. Setting the time in the past doesn't work since every packet 
> that is written will once again update the last modified time. Setting the 
> time after each write() is not a feasible solution. What you suggest can only 
> work when the client gets a clean exit in the face of an interruption and 
> this isn't always the case. 

There is an option to skip if-modified-since. With it wget has to make
an extra HEAD request - and that not only returns a file timestamp but
also the length of the file.

@Dave Could you give us an example of a command line where the issue
occurs, resp. could you test with --no-if-modified-since ?

Regards, Tim

signature.asc
Description: OpenPGP digital signature