[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] -e robots=off problem
From: |
Micah Cowan |
Subject: |
Re: [Bug-wget] -e robots=off problem |
Date: |
Mon, 20 Jun 2011 11:20:57 -0700 |
User-agent: |
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110516 Thunderbird/3.1.10 |
On 06/12/2011 06:34 AM, david chou wrote:
> If there is a robots file ,
> how could I retrieve file from
> that site? I 've used the -e robots=off instruction ,
> but it still can not mirror a web site.
Use the --debug flag to diagnose, and compare the output with
http://wget.addictivecode.org/FrequentlyAskedQuestions#Why_isn.27t_Wget_downloading_all_the_links.3F_I_have_recursive_mode_set
--
Micah J. Cowan
http://micah.cowan.name/