|
| From: | Tim Ruehsen |
| Subject: | Re: [Bug-wget] wget downlaod the robots.txt file even with '-R txt' |
| Date: | Thu, 08 Aug 2013 10:22:31 +0200 |
| User-agent: | KMail/4.10.5 (Linux/3.10-1-amd64; KDE/4.10.5; x86_64; ; ) |
On Thursday 08 August 2013 05:51:08 atar wrote:
> wget download the robots.txt file although the '-R txt' option was
> specified.
wget -e robots=no <url>
or to switch if off forever put
robots=no
into /etc/wgetrc or ~/.wgetrc.
Regards, Tim
| [Prev in Thread] | Current Thread | [Next in Thread] |