Re: lynx-dev Web robots

[C.J.LAWSON]

Surprising you never bothered to find out what I wanted to do with the web
robot ...
        Any how, thanks for the advise

        Jonathan

On Tue, 8 Sep 1998, David Woolley wrote:

> >    Please does anyone know about web robots and how to write/implement
> > them?
> 
> Lynx can act as a web robot.
> 
> You need to be aware that many sites have found the web robot features in
> MSIE 4 and third party products to be highly undesirable, by hitting large
> numbers of pages, costing them bandwidth, that are never then read.
> A number of sites are taking counter measures, although several crawlers
> take counter counter measures by failing to provide a distinguishable user
> agent string.  Lynx could be at risk of being blacklist if it were implicated
> in significant numbers of crawling incidents with its standard user agent
> string.
> 

--
Jonathan Lawson 
Thermal Processes Unit 
Department of Applied Energy and Optical Diagnostics 
School of Mechanical Engineering, 
Cranfield  University, 
Cranfield, Bedford. UK.  
email address@hidden


                                        'They came forth from unholy darknesses 
...
                                         and were driven back by the rage of 
Angels'