[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Warn that --compression=gzip is not enforced

From: Tim Rühsen
Subject: Re: Warn that --compression=gzip is not enforced
Date: Sat, 31 Jul 2021 13:25:10 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0

On 30.07.21 03:30, 積丹尼 Dan Jacobson wrote:
The man page should mention for --compression=gzip,
the website can still ignore it and send the whole file uncompressed.
And wget is not picky, and still will accept it. So better have extra
disk space ready. Tested with
https://www.ncc.gov.tw/chinese/files/opendata/radio.csv .

This is because the client just tells the server what it supports / accepts (not what it desperately wants).

The issue here is that the "identity" encoding is always implicit - there is no way to disable it.

So a
  Accept-Encoding: gzip
means "hey server, send me the content either not encoded or gzip encoded - whatever you like :-)"

If you know that the server sends uncompressed content, you can compress it yourself on-the-fly to avoid excessive disk space usage. At least it works for single files:
  wget -O- <URL> | gzip > radio.csv.gz

Printing a warning may help some users, but others will be swamped, e.g. when doing recursive downloads.

Regards, Tim

Attachment: OpenPGP_signature
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]