guile-user
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: salutations and web scraping

From: Andy Wingo
Subject: Re: salutations and web scraping
Date: Tue, 10 Jan 2012 22:46:12 +0100
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) 
Hi Catonano,

On Fri 30 Dec 2011 23:58, Catonano <address@hidden> writes:

> I´m a beginner, I never wrote a single line of LISP or Scheme in my life
> and I´m here for asking for directions and suggestions.

Welcome! :-)

> I´m mumbling about a pet project. I would like to scrape the web site of
> a comunitarian radio station and grab the flash streamed content they
> publish. The license the material is published under is Creative Common 
> so what I´m planning is not illegal.

Sounds like fun.

> my boldness is such that I´d ask you to write for me an example
> skeleton code.

Hey, it's fair, I think; that is a new part of Guile, and there is not a
lot of example code.

Generally, we figure out how to solve problems at the REPL, so fire up
your Guile:

  $ guile
  ...
  scheme@(guile-user)> 

(Here I'm assuming you have guile 2.0.3.)

Use the web modules.  Let's assume we're grabbing http://www.gnu.org/,
for simplicity:

  > (use-modules (web client) (web uri))
  > (http-get (string->uri "http://www.gnu.org/software/guile/";))
  [here the text of the web page gets printed out]

Actually there are two return values: the response object, corresponding
to the headers, and the body.  If you scroll your terminal up, you'll
see that they get labels like $1 and $2.

Now you need to parse the HTML.  The best way to do this is with the
pragmatic HTML parser, htmlprag.  It's part of guile-lib.  So download
and install guile-lib (it's at http://www.non-gnu.org/guile-lib/), and
then, assuming the html is in $2:

  > (use-modules (htmlprag))
  > (define the-web-page (html->sxml $2))

That parses the web page to s-expressions.  You can print the result
nicely:

  > ,pretty-print the-web-page

Now you need to get something out of the web page.  The hackiest way to
do it is just to match against the entire page.  Maybe someone else can
come up with an example, but I'm short on time, so I'll proceed to The
Right Thing -- the problem is that whitespace is significant, and maybe
all you want is the contents of "the <title> in the <head> in the
<html>."

So in XML you'd use XPATH.  In SXML you'd use SXPATH.  It's hard to use
right now; we really need to steal
http://www.neilvandyke.org/webscraperhelper/ from Neil van Dyke.  But
you can see from his docs that the thing would be

  > (use-modules (sxml xpath))
  > (define matcher (sxpath '(// html head title)))
  > (matcher the-web-page)
  $3 = ((title "GNU Guile (About Guile)"))

Et voila.

I don't do much web scraping these days, but I know some others do.  So
if those others would chime in with better ways to do things, that is
very welcome.

Happy hacking,

Andy
-- 
http://wingolog.org/
reply via email to
[Prev in Thread] Current Thread [Next in Thread]