Re: [Bug-wget] Wget Starting Questions

[Micah Cowan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jason Todd Slack-Moehrle wrote:
> Hi All,
> 
> I have some starting Wget questions that I am hoping to gain insight about.
> 
> I want to start at Dmoz.org and follow links for entertainment (like
> concerts, art gallery events, etc) and examine the link to see if I
> should get data back about it and from it.
> 
> My questions:
> 
> 1. Can Wget start at a given URL and examine every link (based upon my
> criteria)? (obviously I can write Case or If/Else or While to do this)

There's -A and -R, which let you decide whether to follow a link based
on wildcard matching; however, it doesn't match against the query-string
portion of the URL (whatever follows a ? in the URL). Also, any link
with a filename ending in .htm or .html will always be downloaded
irrespective of -A or -R (to check for further links to recurse; the
file itself will be deleted later if it doesn't match the accept/reject
rules).

> 2. If I find a link that has certain keywords that I find of interest,
> can I hit that link of interest and get information from that page?

It's not clear to me what you mean by that... if you mean, can wget
respond based on page content, then no.

> 3. How do I get the information about the link of interest and its
> content of interest into a MySQL database? (I know ColdFusion and MySQL
> and PHP). I think what I am asking is how do I get back to my database
> from a crawler?

Wget won't be much help in achieving that. Post-download content (and
log) scanning might.

> 4. I bought Webbots, spiders and screen scrapers in PHP and so far it is
> interesting, but I am wondering what best practices are..

I can't speak much to that myself.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
Maintainer of GNU Wget and GNU Teseq
http://micah.cowan.name/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknrdmIACgkQ7M8hyUobTrFdBwCff8nhEJrw/4w4+XMSfBxMbEaE
AeIAn1bNtHZJNiIVrqDAd7PjnL91UB1x
=rCco
-----END PGP SIGNATURE-----