[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] Fwd: Regarding wget to download webpage
|
From: |
Henrik Holst |
|
Subject: |
Re: [Bug-wget] Fwd: Regarding wget to download webpage |
|
Date: |
Tue, 24 Jan 2012 10:50:34 +0100 |
I looked at your web site and it does not perform standard http
authentication so --username and --password cannot be used to logon to
that page.
You have to supply the username and password using --post-data there
aswell. If you had followed my advice to use the Live Headers extension
with Firefox you would have seen exactly what to do, please use that
tool aswell as learn some http basics and you will soon learn how to
perform what you want. Because since we do not have access to that site
of yours (no username or password) we as a community will have quite
some hard time to tell you exactly how to proceed since we cannot test
things at our end.
Anyways, as I wrote I tested to perform a logon attempt with the site
and with Live HTTP Headers extension active I could see that the
authentication should be performed like this:
wget --post-data "detour=https%3A%2F%2Fwww.collabnet.timeinc.net%
2F&loginID=username&password=password&Login=Login" --save-cookies
cookies.txt "https://www.collabnet.timeinc.net/servlets/TLogin";
Replace the "username" and "password" in the post data with your account
details.
However since I have no account on that site of yours, I do not know if
this really works and whether the detour=xxx thing is really needed and
whether you have to also add a Referer: header or not to the request.
So if I where you:
1. Install Live HTTPS Headers extension in Firefox
2. Open it's capture window
3. Perform a logon and a sql query
4. Study the results in the capture window and try to mimic what you see
with wget.
/HH
tis 2012-01-24 klockan 10:11 +0530 skrev Bhargavi N:
> Hello Henrik,
>
>
> I am saving the log on details in the cookies same way as you have
> mentioned.
>
>
> But cookie file looks empty..
>
>
> Next i loaded the cookie.txt with the
> URL: https://www.collabnet.timeinc.net/servlets/AdHocQuery
>
>
> I saved the results page and then opened in firefox, i get the same
> logon page to ollabnet instead of results page.
>
>
> Steps to run the query:
>
>
> 1) I need to logon to collabnet website:
> https://www.collabnet.timeinc.net
>
>
> 2) In the collabnet website, we have an option to run the query:
>
>
> https://www.collabnet.timeinc.net/servlets/AdHocQuery
>
>
> I need to go to this page and then need to provide SQL query in the
> query text area.
>
>
> Then need to submit the button "Run Query", which will submit the
> query.
>
>
> 3) Next need to download the results page to my local directoy on
> LINUX box.
>
>
> Please help me regarding this.
>
>
> Thanks !
>
>
> Regards,
> Bhargavi
>
>
> On Mon, Jan 23, 2012 at 12:43 PM, Henrik Holst
> <address@hidden> wrote:
> Ok,
>
> most probably the first site (where you logon) returns a
> cookie which you must present to the other site (where you
> perform the sql-query). So
>
> wget --username xx --password yy "https://collabnet website"
> -O /dev/null --save-cookies cookies.txt
>
> That will logon and save the resulting cookie in the
> cookies.txt file. Next you have to send this cookie and your
> query to the other site:
>
> whet --load-cookies cookies.txt "https://the query site"
> --post-data "the query" -O result
>
> And the result should be in the result file.
>
> Now as Angel Gonzales wrote you probably have to send the
> query using the form like --post-data "query=Select%20*%20From
> %20table" where the sql query in question has to be
> url-encoded (something that wget cannot do for you) which is
> mostly replave all spaces with %20 and also all occurences of
> +, &, " and ' with their respective %hex code.
>
> A easy way to see exactly which formfield to use in the
> post-data is to install the Live Headers extension to Firefox
> and open it's capture window and peform the query on the site
> using Firefox, you'll then get to see the exact query, what
> the GET string is, what the post-data is etc. It's a very good
> start to go from.
>
>
> /HH
>
> 2012/1/20 Bhargavi N <address@hidden>
> I meant i am unable to logon with to the website.
>
>
> Sorry, to confuse you with lot of questions.
>
>
> I have to logon to collabnet website first, providing
> username and passwd.
>
>
> Next i want to go to servlet page where i can provide
> SQL query in the textarea of the form.
>
>
> Then i need to submit the form and download the
> results page to my local directory in linux box.
>
>
> I will be running wget command on linux box.
>
>
> Please help me regarding this.
>
>
> Thanks !
>
>
> Regards,
> Bhargavi
>
>
>
> On Fri, Jan 20, 2012 at 7:20 PM, Bhargavi N
> <address@hidden> wrote:
> Hi Henrik,
>
>
> Thanks for the reply.
>
>
> But when i run as you have suggested me, then
> i get the login page of the collabnet
> website..
>
>
> instead of the servlet page where i can enter
> sql.
>
>
> Please help me !
>
>
> I am unable to logon to that page 1st.
>
>
> Thanks!
>
>
> Regards,
> Bhargavi
>
>
> On Fri, Jan 20, 2012 at 4:44 PM, Henrik Holst
> <address@hidden> wrote:
> Well, it normally should work with
> this:
> wget
> "https://collabnet.net/servlets/adhocquery";
> --user="username" --password="password" --post-data "select * from emp" -O
> output_filename
>
> But then according to the man page,
> wget only supports post-data this is
> in urlencoded format like
> "key=value&key2=value2" so wget might
> not support sql queries at all.
> Perhaps cURL works better for your use
> case.
>
> /HH
>
>
> 2012/1/20 Bhargavi N
> <address@hidden>
> Including the team to this
> mail chain..
>
> ---------- Forwarded message
> ----------
> From: Bhargavi N
> <address@hidden>
> Date: Fri, Jan 20, 2012 at
> 4:30 PM
> Subject: Re: [Bug-wget]
> Regarding wget to download
> webpage
> To: Dayanand Kumbhar
> <address@hidden>,
> address@hidden,
> address@hidden
>
>
>
> Hello Everyone,
>
> I am extremely thankful to
> all of you for the help
> regarding wget. But
> still i am unable to get the
> right page downloaded.
>
> Requirement:
>
> I need to run adhoc sql query
> on remote web page. ie;
> collabnet. The
> webpage has text area where i
> can enter sql query and then
> click submit.
>
> Once i submit sql runs and
> results are displayed in the
> page.
>
> This REMOTE page is call to
> servlet ie;
>
> https://collabnet.net/servlets/adhocquery for example.
>
> I want to pass SQL query as
> form data through POST/GET
> method and get the
> sql results page downloaded to
> my local directory on LINUX
> box. I am
> running wget on LINUX host on
> commandline / shell script.
>
> I will invoke wget with the
> URL
>
> https://collabnet.net/servlets/adhocquery and
> formdata ie; SQL as
> --post-data "select * from
> emp". Finally i want to
> download the sql query results
> page to my local directory.
>
> I tried all the options that
> you suggested me, but still
> nable to get the
> right page. Every time it is
> displaying me the index page
> for collabnet,
> which i do not need. I need to
> run the servelet page with the
> formdata
> passed to it.
>
> Then i want to get the results
> page downloaded to my local
> directory on my
> linux host.
>
> Please help me regarding this.
>
> Again thanking all of you for
> your reply and helping me out.
>
> Thanks !
>
> Regards,
> Bhargavi
>
> On Wed, Jan 18, 2012 at 12:40
> PM, Dayanand Kumbhar
> <address@hidden>wrote:
>
> > Hi
> >
> > i am assuming you have to
> post the query string
> > You can use this
> >
> > wget -qO-
> http://example.com/page.php
> --post-data "SQL QUERY" | cat
> >
> >
> > On Wed, Jan 18, 2012 at
> 12:24 PM, Bhargavi N
> <address@hidden> wrote:
> >
> >> Hello Team,
> >>
> >> My name is Bhargavi and i
> am working on UNIX for over
> 7years.
> >>
> >> I work on project, that
> needs to download results from
> remote web page.
> >> This web page has adhoc sql
> querying option. I need to run
> this sql query
> >> on webpage and get the
> results downloaded using wget.
> >>
> >> How can i do that ?
> >>
> >> Please suggest me way to
> download the results page
> using wget.
> >>
> >> Thanks !
> >>
> >> Warm Regards,
> >> Bhargavi
> >>
> >
> >
> >
> > --
> > Dayanand Kumbhar
> >
>
>
>
>
>
>
>
>
>
>
Message not available