Re: [Bug-wget] [Bug-Wget] Issues with Metalink support

[L Walsh]

Random Coder wrote:
> On Sat, Apr 5, 2014 at 4:09 PM, L Walsh <address@hidden 
> <mailto:address@hidden>> wrote:
>
>     I.e. IE will download things to a tmp dir (usually
>     under the user's home dir on windows), then
>     move it into place when it is done.  This prevents partly
>     transfered files from appearing in the destination.
>
>
> IE does not download to a tmp folder.
---

        It depends on timing, what version of IE, and probably
the phase of the moon, but here's a abbreviated trace of me downloading
the linux kernel into C:\tmp\download.  I annotate what's going on in
the left column... you can see almost 50% of the file was downloaded
into a tmp file, then switched to final destination and only
wrote 1M chunks instead of previous 4-12K chunks.

6:17:13,IEXPLORE,CreateFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N","Desired Access: 
Read Attributes, OpenResult: Opened"
6:17:13,IEXPLORE,CreateFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Desired 
Access: Generic Write, Read Attributes, OpenResult: Created"
6:17:13,IEXPLORE,SetAllocationInformationFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","AllocationSize: 
78,399,152"
6:17:13,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
0, Length: 704, Priority: Normal"
6:17:13,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
704, Length: 1,944"
6:17:13,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
2,648, Length: 8,192"
6:17:13,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
10,840, Length: 4,096"
...
6:17:23,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
36,207,192, Length: 4,096"
6:17:23,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
36,211,288, Length: 4,096"
6:17:23,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
36,215,384, Length: 16,384"

I've typed in the save pathname now:
6:17:23,explorer,808","CreateFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz","Desired Access: Read Attributes, 
OpenResult: Opened"
6:17:23,explorer,808","CloseFile",OK ,"C:\tmp\download\linux-3.14.tar.xz",""
6:17:23,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
36,231,768, Length: 4,096"
...
6:17:23,IEXPLORE,WriteFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
36,461,144, Length: 4,096"
...

opens "partial file in same directory":

6:17:23,IEXPLORE,CreateFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Desired Access: Generic 
Write, OpenResult: Opened"

copies from 1st tmp to final location tmp, but in 1MB increments
6:17:23,IEXPLORE,ReadFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
0, Length: 1,048,576, Priority: Normal"
6:17:23,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 0, Length: 
1,048,576, Priority: Normal"
...
6:17:23,IEXPLORE,ReadFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
35,651,584, Length: 817,752"
6:17:23,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 35,651,584, 
Length: 817,752"
6:17:23,IEXPLORE,ReadFile","END OF 
FILE","C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Offset: 
36,469,336, Length: 1,048,576"

deletes first tmp, and now saved directly to "patial" at destination:
6:17:23,IEXPLORE,SetDispositionInformationFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz","Delete: 
True"
6:17:23,IEXPLORE,CloseFile",OK 
,"C:<user>AppData\Local\MS\Win\<tmp-internet-IEfiles>\BNZE234N\linux-3.14.tar[1].xz",""

only 1M writes:
6:17:23,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 36,469,336, 
Length: 1,048,576"
6:17:24,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 37,517,912, 
Length: 1,048,576"
6:17:24,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 38,566,488, 
Length: 1,048,576"
6:17:24,explorer,QueryDirectory",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Filter: 
linux-3.14.tar.xz.w5aj0r5.partial, 1: linux-3.14.tar.xz.w5aj0r5.partial"


final output being created:
6:17:24,explorer,CreateFile",OK ,"C:\tmp\download\linux-3.14.tar.xz","Desired 
Access: Read Attributes, Read Control, OpenResult: Opened"

more writes to partial:
6:17:25,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 40,663,640, 
Length: 1,048,576"
6:17:25,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 41,712,216, 
Length: 1,048,576"
6:17:25,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 42,760,792, 
Length: 1,048,576"
6:17:25,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 43,809,368, 
Length: 1,048,576"
6:17:26,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 44,857,944, 
Length: 1,048,576"
6:17:26,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 45,906,520, 
Length: 1,048,576"
....
6:17:34,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 75,266,648, 
Length: 1,048,576"
6:17:34,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 76,315,224, 
Length: 1,048,576"

Last write to partial then close:
6:17:34,IEXPLORE,WriteFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","Offset: 77,363,800, 
Length: 1,035,352"
6:17:34,IEXPLORE,CloseFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial",""
6:17:34,IEXPLORE,CloseFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial",""
6:17:34,IEXPLORE,Thread Exit",OK ,"","Thread ID: 2888, User Time: 0.3900025, 
Kernel Time: 0.4524029"

rename from partial to final:
6:17:34,iexplore,SetRenameInformationFile",OK 
,"C:\tmp\download\linux-3.14.tar.xz.w5aj0r5.partial","ReplaceIfExists: True, 
FileName: C:\tmp\download\linux-3.14.tar.xz"
6:17:35,explorer,CloseFile",OK ,"C:\tmp\download\linux-3.14.tar.xz",""


---------------------------------------------------------------------

(trace from process monitor)..

> I'm not convinced trying to pre-optimize for disk fragmentation is 
> useful here.  If the user is concerned about such things, they're free 
> to copy the download after it's done and delete the original.  Or run an 
> defragmenter.

----
Well it appears they only write 1M chunks to the destination
dir.  Before that, in the tmp location, they used small 4K chunks mostsly.


I'm not "convinced" about anything myself.  I'm stating possible
reasons for downloading a 'slowish' download into a tmp location
first.  The kernel downloads pretty quickly, so maybe it detects
the speed of the download and that figures into when to switch over.
But it only does larger writes at the destination.

Even utilities like winzip and 7zip will extract file to the user's tmp
dir before copying or moving them into the final location.