[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] performace issues when saving a http iptv stream on a SD

From: Bykov Aleksey
Subject: Re: [Bug-wget] performace issues when saving a http iptv stream on a SD car=
Date: Sat, 18 May 2013 23:00:57 +0300
User-agent: Opera Mail/12.14 (Win32)

Greetings, Hrvoje Niksic

Yes, at least my Windows is misconfigured (but  write buffer is surely
Yes, this problem exist with all programs, that tries to write on flash
drive without themself buffering (at least on Win98, WinXP, Win Vista).

Can You please help with advise?

Results of checking at Win Vista SP2 x86 and Win XP SP3 x86 with three
different flash drives (16 Gb with 8 Kb cluster, 128 Mb and 2 Gb with 4 kb
cluster) is almost same. Sorry, no *nix system in access.
Test file size - 200 Mb
With fflush - 20-40 kb/s. Write it by variable size chunks (1,5 - 12 kb)
(checked by procmon). First Mb at full speed, then quick drop.
Without fflush - 300-450 kb/s. Write it by 4 kb chunks
Without fflush with additional 8Mb buffer - 600-650 kb/s. Write it by 256
kb chunk.
As default in MinGW BUFSIZ = 512.

Current variants, that i see (at current data set. Need more data from all
0. Leave it as is.
1. Simple add Windows to exception (to not fflush) (as currently VMS).
2a. Add user option to disable flush. As default - current behavior. In
documentation notate, that possible data loss and delays.
2b. Add user options to determine buffer size. 8Mb buffer - is overkill,
it's enough 8-64 Kb.
Side effect of second variant - converting links during breaking recursive
downloads by Ctrl-C ;-)

What more variants to solve this inconvenience is exist?
What variant brings more advantages?

Thanks in advance.

Best regars, Alex

I am really curious what the rationale behind the fflush was.

The rationale was the data is not of much use to anyone while lingering
in Wget's stdio buffer.  No one else sees it, and if you only press ^C,
you lose it permanently.  Wget is not asking for the data to be
immediately written out to disk, it is merely handing it off to the
operating system, so other programs (be it tail -f or a locally running
browser) can see it and put it to good use.  fflush != fsync.

Plus, typical stdio buffer is smaller than Wget's network read buffer,
so most of the data would have been flushed anyhow.  The fflush() only
makes sure that by the time Wget announces that it has downloaded (say)
123,456 bytes of data, your file contains no *less* than 123,456 bytes.

If some misconfigured system were really neglecting to cache filesystem
writes, then it would have problems with most programs out there, not
just Wget.  After all, BUFSIZ on a typical system is only 4-8K.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]