[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] wget doesnt work but curl works !
From: |
Vishwanath Reddy Beemidi |
Subject: |
[Bug-wget] wget doesnt work but curl works ! |
Date: |
Fri, 14 Oct 2011 15:24:39 -0700 |
Hi,
I have trouble getting wget to work when downloading a file using http.curl
works fine for the same URL.
Both the commands are being run from the same server at the command line, OS
: RH Linux mdc1pr012 2.6.18-238.9.1.el5
Following are the commands and the debug messages, any insights into what
the problem could be are appreciated.
address@hidden wget -d -S http://www.preprod.abc.com/tools/90067660.csv
Setting --server-response (serverresponse) to 1
DEBUG output created by Wget 1.11.4 Red Hat modified on linux-gnu.
--2011-10-14 18:00:58-- http://www.preprod.abc.com/tools/90067660.csv
Resolving www.preprod.abc.com... 184.31.131.61
Caching www.preprod.abc.com => 184.31.131.61
Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.
Created socket 3.
Releasing 0x0000000007683bc0 (new refcount 1).
---request begin---
GET /tools/90067660.csv HTTP/1.0
User-Agent: Wget/1.11.4 Red Hat modified
Accept: */*
Host: www.preprod.abc.com
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 302 Found
Location:
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Connection: close
Content-Length: 925
---response end---
HTTP/1.1 302 Found
Location:
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Connection: close
Content-Length: 925
Location:
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=[following]
Closed fd 3
--2011-10-14 18:00:58--
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
Resolving fd000xnchegrn02... 11.48.43.72
Caching fd000xnchegrn02 => 11.48.43.72
Connecting to fd000xnchegrn02|11.48.43.72|:80... connected.
Created socket 3.
Releasing 0x00000000076806e0 (new refcount 1).
---request begin---
GET
/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
HTTP/1.0
User-Agent: Wget/1.11.4 Red Hat modified
Accept: */*
Host: fd000xnchegrn02
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Pragma: no-cache
WWW-Authenticate: NTLM
WWW-Authenticate: BASIC realm="Federated_Department_Stores"
Content-Type: text/html; charset=utf-8
Proxy-Connection: close
Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/
Connection: close
Content-Length: 1114
---response end---
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Pragma: no-cache
WWW-Authenticate: NTLM
WWW-Authenticate: BASIC realm="Federated_Department_Stores"
Content-Type: text/html; charset=utf-8
Proxy-Connection: close
Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/
Connection: close
Content-Length: 1114
Closed fd 3
Authorization failed.
>>>>>>>>>>>>>>> Following is the curl trace info for the same URL
address@hidden ~]$ curl --trace-ascii tr.out -o out.dat
http://www.preprodmacys.fds.com/tools/90067660.csv
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 155k 100 155k 0 0 449k 0 --:--:-- --:--:-- --:--:--
550k
address@hidden ~]$ more tr.out
== Info: About to connect() to www.preprod.abc.com port 80
== Info: Trying 184.31.131.61... == Info: connected
== Info: Connected to www.preprod.abc.com (184.31.131.61) port 80
=> Send header, 186 bytes (0xba)
0000: GET /tools/90067660.csv HTTP/1.1
0022: User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
0062: OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
008b: Host: www.preprod.abc.com
00ab: Accept: */*
00b8:
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 25 bytes (0x19)
0000: Server: IBM_HTTP_Server
<= Recv header, 46 bytes (0x2e)
0000: Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
<= Recv header, 30 bytes (0x1e)
0000: ETag: "b19b9-26f41-7f2b4580"
<= Recv header, 22 bytes (0x16)
0000: Accept-Ranges: bytes
<= Recv header, 26 bytes (0x1a)
0000: Content-Type: text/plain
<= Recv header, 37 bytes (0x25)
0000: Date: Fri, 14 Oct 2011 22:06:15 GMT
<= Recv header, 24 bytes (0x18)
0000: Content-Length: 159553
<= Recv header, 24 bytes (0x18)
0000: Connection: Keep-Alive
<= Recv header, 127 bytes (0x7f)
0000: Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=4117DB; Domain=.prep
0040: rodmacys.fds.com; expires=Fri, 14-Oct-11 22:36:15 GMT; Path=/
<= Recv header, 70 bytes (0x46)
0000: Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc
0040: .com
<= Recv header, 104 bytes (0x68)
0000: Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:36:15 GMT
0040: ; path=/; domain=.preprod.abc.com
<= Recv data, 5078 bytes (0x13d6)
0000: 90067660,po-xx-xx-xx,popup,
001d: 90067660,ex-xx-xx-xx,external sites,
0043: 90067660,im-xx-xx-xx,imports,
.........MORE LINES..............
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Also when we run wget from another server against the same URL it works
fine...the version on the other server is 1.10.4. So we got that version and
installed it on the server mentioned in the above example and tried but we
still got the same error message.
Following is the debug log from the successful run on the another server :
address@hidden /home/scripts(0)$ wget -d -S
http://www.preprod.abc.com/tools/90067660.csv
Setting --server-response (serverresponse) to 1
DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
--15:03:42-- http://www.preprod.abc.com/tools/90067660.csv
=> `90067660.csv.3'
Resolving www.preprod.abc.com... 184.31.131.61
Caching www.preprod.abc.com => 184.31.131.61
Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.
Created socket 3.
Releasing 0x000000000063b390 (new refcount 1).
---request begin---
GET /tools/90067660.csv HTTP/1.0
User-Agent: Wget/1.10.2 (Red Hat modified)
Accept: */*
Host: www.preprod.abc.com
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Server: IBM_HTTP_Server
Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
ETag: "b19b9-26f41-7f2b4580"
Accept-Ranges: bytes
Content-Type: text/plain
Content-Length: 159553
Date: Fri, 14 Oct 2011 22:03:42 GMT
Connection: keep-alive
Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.preprod.abc.com;
expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/
Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com
Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
domain=.preprod.abc.com
---response end---
HTTP/1.0 200 OK
Server: IBM_HTTP_Server
Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
ETag: "b19b9-26f41-7f2b4580"
Accept-Ranges: bytes
Content-Type: text/plain
Content-Length: 159553
Date: Fri, 14 Oct 2011 22:03:42 GMT
Connection: keep-alive
Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.
preprod.abc.com; expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/
Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com
Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
domain=.preprod.abc.com
Registered socket 3 for persistent reuse.
cdm: 1 2 3 4 5 6 7 8
Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
2011-10-14 15:33:42] BC_HA_7039BD33D34B38D1_70737B63 41174D
cdm: 1 2 3 4 5 6 7 8
Stored cookie preprod.abc.com -1 (ANY) / <session> <insecure> [expiry none]
shippingCountry US
cdm: 1 2 3 4 5 6 7 8
Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
2011-10-14 15:33:42] stop_mobi yes
Length: 159,553 (156K) [text/plain]
100%[=================================================================================================================================================================================================>]
159,553 105.31K/s
15:03:44 (105.15 KB/s) - `90067660.csv.3' saved [159553/159553]
PS : I changed the domain names mentioned in the logs above
Thanks,
Vishwanath
- [Bug-wget] wget doesnt work but curl works !,
Vishwanath Reddy Beemidi <=