[Savannah-hackers-public] [gnu.org #328648] lists.gnu.org and search engines

[Joshua Ginsberg via RT]

If my institutional memory serves me correctly, we disallowed robots on
lists.gnu.org because the frequency and intensity with which the spiders
crawled lists made mail and web service on the box slow to a crawl --
the disks were hammered and began to fry from all of the I/O. We made
the decision that, since our archives are available and googleable from
the numerous mirrors out there, given the choice between faster mail
service and search-engine indexed archives, we'd prefer the former.

Lists is still under a relatively constant load. I'm attaching the load
average graph for lists for the last 24 hours, for your viewing pleasure.

If there is not a googleable service out there that fits your needs,
we'll look into caching mechanisms and distributed load to try to bring
the spiders back to lists. Otherwise, this request may have to wait
until we migrate lists to the 4-core Opteron machine.

-jag

> [beuc - Thu Jan 25 11:13:41 2007]:
> 
> Hi,
> 
> Search engines do not index lists.gnu.org (and lists.nongnu.org) at
> all, probably due to the current robots.txt:
> 
>  User-agent: *
>  Disallow: /
> 
> This is blocking when I try to search across several mailing lists.
> 
> Can you allow indexing of public archives?
> 
> Thanks,
> 
> 
> 
-- 
Joshua Ginsberg <address@hidden>
Free Software Foundation - Senior Systems Administrator