lynx-dev Lynx.trace clued howto use curl to download yahoo messages

[Bob]

clemensF wrote:

> > Bob:
>
> > > would curl http://curl.haxx.se/ do the job?  it handles cookies, even SSL.

Thanks so much.

I installed the debian binary package of curl and it worked the first time.

http://curl.haxx.se/download.html


URL="http://groups.yahoo.com/group/AkhaWeeklyJournal/message/120";
ANON="anonymous:address@hidden"

curl -u $ANON -L --url $URL -O

>From lynx's -trace (Lynx.trace file), I learned that one yahoo
obstacle is a cookie used for authorization to get a messageURL.
Also curl's -L switch is absolutely necessary to get over the ad
page pseudoURL and a six "Aren't you supposed to be somewhere
else?" messages in Lynx.trace. Yahoo is triggering an error
condition in its own list browsing code, which if you join a yahoo
group you can see if you go to groups.yahoo.com in browser and
login. Even members who are logged in have to look at these
ad pages which Lynx.trace reveals to be standard responses to
an error condition. The way past the error page(the ad page),
which in my haste I always thought of ;-) as an error page, is
either to click the link to request the same url again, from lynx
or any browser on the ad/error page, or use curl which keeps
trying things when it gets the "Aren't you supposed to be
somewhere else?" error--lynx retries six times, but curl
finds a way and lynx doesn't.

#Lynx.trace  6 times: "Aren't you supposed to be somewhere else?"

Location: 
http://groups.yahoo.com/group/AkhaWeeklyJournal/auth?done=%2Fgroup%2FAkhaWeeklyJournal%2Fmessage%2F120
Connection: close
Content-Type: text/plain

Aren't you supposed to be somewhere else?

Server: Apache/1.3.3 (Unix)
Cache-control: no-cache, no-cache="Set-Cookie", private
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Pragma: no-cache
Set-Cookie:
G=v=6&data=c0NPVPvMlS5I_HO8VBzv2VJrmLI5t8pw5cOEg-Na73BzmSmYjydpsbCTqYQI1zP3OtqwZLKlulblDsoEETrfVl9rDfY5I7CUzEK-Tvis0tIfTG3KOcMFQpAOhoXHFAQcsFBxcYTyVQbSgNUEvzSyZfGoJPpsLfd8LvTSDOPMO5HKeYsPTznFHVoioTtVknKO59O93mnj4CcxadQKZgI6I08KmfrfPSMslGk2Ry7G9HXrHYl--3hcvuuQopZhCyI4zXDXSpATCP3ETqgf0ne8_Qd9SrKtL7M-cf7GloubvUYy3Bm5y2judTUJFxDBApGetrCfpx7Gk4_D_7tkqdJv32TB6EFgm5LAncYvOYxwm2NXBX4OQveBS-3t_bZ139675cn1t5KuN9qxbCWO_1LjfG3MoyYNkXLTGdfIFWFSvYARYmJZr1LBZdVejzU&n=17;
path=/group/AkhaWeeklyJournal; domain=groups.yahoo.com
Location: 
http://groups.yahoo.com/group/AkhaWeeklyJournal/auth?check=G&done=%2Fgroup%2FAkhaWeeklyJournal%2Fmessage%2F120

Connection: close
Content-Type: text/plain

Aren't you supposed to be somewhere else?

Location: http://groups.yahoo.com/group/AkhaWeeklyJournal/message/120
Connection: close
Content-Type: text/plain

Aren't you supposed to be somewhere else?

Server: Apache/1.3.3 (Unix)
Cache-control: no-cache, no-cache="Set-Cookie", private
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Pragma: no-cache
Location:
http://groups.yahoo.com/group/AkhaWeeklyJournal/interrupt?st=1&ln=AkhaWeeklyJournal&h=120&m=1&done=%2Fgroup%2FAkhaWeeklyJournal%2Fmessage%2F120

Connection: close
Content-Type: text/plain

Aren't you supposed to be somewhere else?

Content-Type: text/html
Pragma: no-cache
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Cache-control: no-cache, must-revalidate, no-cache="Set-Cookie", private

Server: Apache/1.3.3 (Unix)
Cache-control: no-cache, no-cache="Set-Cookie", private
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Pragma: no-cache
Set-Cookie: interrupt=1&1017042077&120; path=/; domain=groups.yahoo.com
Location:
http://groups.yahoo.com/group/AkhaWeeklyJournal/interrupt?st=2&ln=AkhaWeeklyJournal&m=1&done=%2Fgroup%2FAkhaWeeklyJournal%2Fmessage%2F120

Connection: close
Content-Type: text/plain

Aren't you supposed to be somewhere else?

-Bob


; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to address@hidden