[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: LYNX-DEV URL grabber for lynx

From: David Woolley
Subject: Re: LYNX-DEV URL grabber for lynx
Date: Fri, 10 Oct 1997 08:44:39 +0100 (BST)

> I've just written a small C program to filter URLs from stdin, sort

Wouldn't it have been easier to borrow the routine from wget (Gnu sites)
that does this (assuming that you mean filter them from an HTML source)?
(You are allowed to do this providing the result is licensed only under
the GPL.)

> batch or at) to retrieve all found URLs using "lynx -dump" (or

For -dump, and with recursion allowed, Lynx already does this.

> However, I'm hesitating about publishing it in due to possible abuse.
> It is very easy to retrieve 1000 and more URLs at once, so before
> publishing I would like to hear your opinion.

wget has options to insert delays between each fetch to avoid overloading
the server and also respects robots.txt to avoid fetching private or
dynamic information.
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]