bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Please use gzip/gunzip when fetching webpages


From: itstheworm
Subject: Please use gzip/gunzip when fetching webpages
Date: Wed, 1 Feb 2023 18:05:48 +0100 (CET)

More often than not I try recursively downloading a webpage using wget, only to 
have it download a single `index.html.gz` then stop. Obviously wget can't read 
gzipped files so it fails to find any links for recursive downloading... I 
ended up using a wget fork[1] that was last updated 10 years ago and it works 
fine, however I find it odd that such a basic feature never made it into 
mainline wget.

Please add a feature for automatically detecting and uncompressing gzipped 
webpages before crawling them.

[1] https://github.com/ptolts/wget-with-gzip-compression


reply via email to

[Prev in Thread] Current Thread [Next in Thread]