Re: lynx-dev Automate Login to set Cookie or Lynxing from a Script

[Klaus Weide]

On Mon, 6 Sep 1999, bjm wrote:

> I am new to lynx and have a problem which seems related to Lynx's cookie 
> support.
> 
> I am using Lynx Version 2.8.1pre.9 (10 Oct 1998) on Redhat Linux 5.2.

First, if you seriously want to tinker around with this, you should 
upgrade to 2.8.3 or, better yet, to the most recent development code
from <http://sol.slcc.edu/lynx/current/>.  There have been some changes
in cookie handling, and some replies you get may not apply to the older
code.  At a minimum, the comments in the latest lynx.cfg are a bit more
explicit (more than 2.8.2 I think).

> I use ebay alot. When I want to ask a seller a question, I have to login with 
> my ebay
> username/password. A cookie is set in my browser which makes subsequent pages 
> that
> contain other ebay usernames also contain their email address. In other words,
> the first time you go to this URL:
> http://cgi3.ebay.com/aw-cgi/eBayISAPI.dll?ReturnUserEmail&requested=bmurphee
> 
> you are prompted to login. After you login and do a refresh on the above URL
> (in the same Netscape window, ie, session), the page will yield the email
> address I want. 

I don't know about those specifics with ebay, so I assume you know
what you are doing.

> I am trying to write a script that will automate the login and setting of that
> cookie so that my script can go on to check on the data from my auctions.
> 
> Ebay says its cookies are one session only. Lynx documentation I have read 
> says
> cookies it supports are one session only. 

If lynx documentation still says that (even in the latest code), it is a
bit obsolete.  But session-only is still the default behavior afaik unless
you enable PERSISTENT_COOKIES in lynx.cfg.

The definition of a "session" lynx uses is the normal one: one invocation
of a lynx process.  Lynx can keep cookies persistent across sessions by
storing them in a file (see COOKIE_FILE in lynx.cfg).  But it only does
that for cookies that are marked to expire at some point in the future,
i.e. NOT for "session cookies".

Storing "session only" cookies in the file would be a violation of the
specs (altough I guess one could argue around that by redefining what
constitutes a "session").

> How can I make that cookie information
> permanent so that I dont have to login each time I go to those web pages to
> get the email addresses?  
> 
> I am an extreme beginner to shell scripting using awk, sed and perl. 
> Question: If I write
> a script that includes a command like 
> 
> lynx -dump URL
> 
> and then grab data from the output to create something like
> 
> lynx -dump newupdatedURL
> 
> have I just created two lynx sessions? 

Yes.

> Will cookie information carry over to the second
> instance of the lynx -dump command?

Only if you have configured lynx to use persistent cookies, and never
for session-only cookies.

By the way for persistent cookies (if storing new values is important)
you should not run those lynx sessions at the same time (whether
interactive or not), otherwise cookie updates received in one session
(the one that terminates erlier) can be lost as far as the cookie file
is concerned.  (Or somehow specified different cookie files for
separate concurrent sessions.)

> If not, if I can't automate the login and setting of the cookie and have it 
> carry over to
> other instances of lynx -dump or -get_data in the script, can anyone suggest 
> something
> better.....should I use a Perl library or something like that to better 
> interact with the
> logins and setting of the cookies?

You may be able to trick lynx into believing that a cookie is
persistent when it really was meant as session-only.  If you
want to try that, start with looking at the Cookie Jar Page (^K),
and at the .lynx_cookies file after accepting some (persistent)
cookies.  You may be able to cut-and-paste cookies from the Jar page
into the file, with appropriate modifications, or something like
that, but that's going to be tricky (e.g. observe TAB vs. spaces
formatting).  Another way to grab cookies "in transit" would be
using "junkbuster" (I'm sure there is a redhat package for it).
Most likely you can even force junkbuster to _send_ arbitrary (fixed?)
cookies added to each request.

There must be a Perl library for doing all this more flexibly (isn't
there for everything?), but I don't know specifics.

> Any assistance on this is greatly appreciated. Anything--resources I can 
> read, things
> I can try, scripts you have written to do similar things that I can look 
> at--all would
> be appreciated. I am a beginner so any help is greatly appreciated. Oh, also, 
> I am not
> a member of this list but if solid advice comes from this query, I would 
> probably
> consider joining. In the meantime, please reply to my address: address@hidden

Check out the list archives while you are not subscribed, or you
are going to miss some replies.

Btw. please reduce your text line width!

   Klaus