Re: [Bug-wget] Trying to wget files from IEEE802 document server

[Robert Moskowitz]

On 05/20/2012 04:34 PM, Ángel González wrote:
> On 20/05/12 19:45, Robert Moskowitz wrote:
> > On 05/20/2012 12:34 PM, Ángel González wrote:
> > > The problem you get when using --no-check-certificate is due to
> > > https://mentor.ieee.org/802.18/dcn/12/ not containing a file list. TRy
> > > going to https://mentor.ieee.org/802.18/dcn/12/ in your browser: "Not
> > > Found The page requested was not found or is not a functioning properly"
> > So I see.  Problem with assuming that the URL seen here:
> >
> > https://mentor.ieee.org/802.18/documents
> >
> > Would work off the higher level.
> >
> > The nice page listed above gives a nice user interface into a group's
> > documents, hiding which year the rev0 doc was posted.  I like to
> > maintain the by year orientation of wg/dcn/yr/.
> You can use something like
>   wget --no-check-certificate -m -np https://mentor.ieee.org/802.18/documents

There are reasons I do not want to use this high of a level of wget, 
principally there are old years that don't have any updates and wget is 
rather inefficient (compared to rsync, but that is not an option here) 
in checking directories for changed files.

But I figured it was a good place to start.  So I ran it and it was 
working ok.

The directory structure this creates is ../mentor.ieee.org/802.18/

There are 3 directories that got created: bp, dcn, and file.  bp has a 
startpage html and file has year 09 (strange). dcn had year 12 that came 
down ok.  But when it got to 11 under dcn and the problems started.  I 
see the following:

--2012-05-20 17:18:36--  
https://mentor.ieee.org/802.18/dcn/11/18-11-0085-00-0000-rr-tag-agenda-november-2011.ppt
Reusing existing connection to mentor.ieee.org:443.
HTTP request sent, awaiting response... No data received.
Retrying.

--2012-05-20 17:18:37--  (try: 2)  
https://mentor.ieee.org/802.18/dcn/11/18-11-0085-00-0000-rr-tag-agenda-november-2011.ppt
Connecting to mentor.ieee.org|140.98.193.36|:443... connected.
WARNING: cannot verify mentor.ieee.org’s certificate, issued by 
“/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at 
https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA 
- G3”:
  Unable to locally verify the issuer’s authority.
HTTP request sent, awaiting response... No data received.
Retrying.

.
.
.

--2012-05-20 17:20:14--  (try:15)  
https://mentor.ieee.org/802.18/dcn/11/18-11-0085-00-0000-rr-tag-agenda-november-2011.ppt
Connecting to mentor.ieee.org|140.98.193.36|:443... connected.
WARNING: cannot verify mentor.ieee.org’s certificate, issued by 
“/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at 
https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA 
- G3”:
  Unable to locally verify the issuer’s authority.
HTTP request sent, awaiting response... 200 OK
Length: 356864 (348K) [application/vnd.ms-powerpoint]
Saving to: 
“mentor.ieee.org/802.18/dcn/11/18-11-0085-00-0000-rr-tag-agenda-november-2011.ppt”

100%[======================================>] 356,864      976K/s   in 0.4s

-------------------------------------------------------

I finally killed the command, so I don't know what would have happened 
when it got to year 10.

So why the problem?  And more importantly what can I do about it?