[Bug-wget] [bug #30999] wget should respect robots.txt directive crawl-delay

[Miquel Llobet]

Follow-up Comment #5, bug #30999 (project wget):

I have read the robots.txt spec thoroughly and found no way to set crawl-delay
for a specific file. If someone could look into it that would be nice. 
Otherwise I think the best solution is to set --wait to the matching
crawl-delay if the user hasn't set --wait already.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?30999>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/